IceStorm - Topic creation

[ritesh]

Hi,

I have started using/evaluating IceStorm for my application and noticed an issue with Topic creation.
The icestormadmin utility takes quite a bit of time to create topics. Also, the very first publish call is really slow if everything is setup to TCP with client and server sitting on two different networks.

Another confusion which I have is about number of topics. Should I proceed with more no. of topics or fewer topics but more interfaces? I want to achieve one-to-one binding between each slice data element (i.e. my msg load shouldn't be anything else other than a BYTE).

Thanks,
Ritesh

[matthew]

It should be very fast to create topics and such. I suspect that the reason is that you have unreachable network interfaces published in the proxies. You can turn on network tracing (Ice.Trace.Network=2) to see if this is actually the reason.

I'm can't really advise you on the number of topics to have. As a general rule if all events must be received by all subscribers then you want a single topic -- otherwise it becomes burdensome to manage. However, if you have lots of different classes of events in which different classes of subscribers have interest then you will want to use different topics to avoid network congestion and wierd looking kitchen sink interfaces.

[Malin]

What is the maximum number of topics ICEStorm may handle and what is your recommendable maximum? How will a large number of topics impact performance of ICEStorm itself?

Regards,
Malin

[bernard]

Each time you create a Topic a corresponding record in the IceStorm database is created, and a servant (object implementation) is kept in the IceStorm server.

Topic creation takes some time because of this database writing / committing. If this is a concern, you can tune the database to make it faster; see Icestorm<->Berkeley DB error.

The maximum number of topics in an IceStorm instance depends on the amount of RAM available; beyond a few thousands, you will also need some database configuration change. There is however no hard limit in the IceStorm implementation.

That said, IceStorm is typically used with only a few topics. So it's possible that you're not using IceStorm properly if you envision hundreds or thousands of topics. Could you describe more the events you want to publish, and why you want to create so many topics?

Thanks,
Bernard

[ritesh]

Thanks Mathew and Bernard for shedding some light on IceStorm Topics. My problem statement also revolves around # of topics which Malin is also asking for.
I need to achieve something which is described below.

Method 1, where different methods ensure different messages/events per topic.I need to have around 2000 different topics and 16 different Event messages.

interface Event {
void onEvent1(short value);
void onEvent2(short value);
...
...
void onEventN(short value);
};

Method 2, where different interfaces ensure different message per topic.
interface Event {
void onEvent(short value);
};

interface Event1 extends Event {
};
...
...
interface EventN extends Event {
};

and finally, different topics for all different events on topics.
a topic list will look like this,
TOPIC1Event1
TOPIC1Event2
TOPIC1Event3...
TOPIC2Event1...
...
...
TOPICNEventN

please suggest me the correct approach to proceed with as I am expecting 2000-2500 different topics and 16-32 different events.

thanks,
Ritesh

[matthew]

This still doesn't really tell me enough to advise you on what to do. Why do you need so many topics? Are the events really just a short? Does every subscriber want to receive all event types, or only a subset of the broadcast events?

Another thing you should keep in mind is how many publishers and subscribers you are thinking of supporting, and what the expected event throughput is. For example, if you have 10 publishers publishing 10 events a second to 10 subscribers then you have 10 * 10 * 10 events per second flowing through the service. At some point you simply cannot support so many events and need to federate. If you have created loads of topics this becomes increasingly more difficult.

[ritesh]

Mathews,
Thanks for the quick response. You could consider my application to be a Ticker which has a Matrix which I need to update. the rows are my topics and columns are my events. I am expecting 2000-2500 rows at the most.

So, a subscriber is interested in all the column updates i.e. it should subscribe for all the events. ( the Message load is only a short)

What is the throughput I can expect with IceStorm here. (assuming I am using the best possible approach). Also, UDP access is fine for me as I am not interested in the assured delivery but it is the latency which is important for me.

please advise.

thanks,
Ritesh

[matthew]

Are all subscribers interested in all events for all tickers? If so wouldn't the better model be something like:

interface TickerUpdate
{
   void update(string symbol, short event);
};

If not then without IceStorm filtering (which isn't possible yet) then, yes, you need lots of different topics.

As to the expected throughput you can run the throughput demo and look at the various times for twoway, oneway, etc. The demo doesn't support udp so you'd have to do some hand modification if you wanted to know those numbers. The maximum sustained throughput for IceStorm will be somewhat less than than this in terms of events-per-second. You can mitigate the sustained throughput by using event batching which will allow you to have more throughput at the expense of latency.

Lets say you have 100 events per second going to 100 subscribers. In this case you will have 10 * 100 or 1000 events per second. If you move to 100 events per second then you'll have 10,000 EPS which IceStorm may not be able to keep up with. In this case you'll need to federate (that is have an IceStorm server distributed to 2 or more linked IceStorm servers to which the subscribers subscribe) which will allow you to cut the load by some factor.

[ritesh]

Thanks Matthew for your response. I just wanted to avoid string comparison on the client side. But now the above seems to be the only solution; as you said more # of topics would degrade the performance of the application.

Also, will it help to configure everything to UDP i.e. both publisher and subscriber using UDP to connect to ICEStorm. if yes then could you please guide me in setting up the configuration for the same as I tried the combination but seems to be missing something.

Thanks,
Ritesh

[matthew]

For the clock demo do the following:

- In Publisher.cpp:

Change:

if(!obj->ice_isDatagram())
{
obj = obj->ice_oneway();
}

to:

obj = obj->ice_datagram();

In Subscriber.cpp change add after the subscription:

object = object->ice_datagram();

in config.service change:

IceStorm.Publish.Endpoints=default

to:

IceStorm.Publish.Endpoints=udp

in config.sub change:

Clock.Subscriber.Endpoints=tcp

to:

Clock.Subscriber.Endpoints=udp

[marc]

Can you please update your project in your profile? "IceStorm" is our project

[ritesh]

Thanks to ZeroC staff for all the answers. My stuff seems to be working now but again I am stuck with a design issue.

As I described earlier, I have a single IceStorm interface with a msg containing key and value.

Now, in my subscriber, if I subscribe to the topic with the implementation instance of say EventI, then how am I suppose to get a one-to-one binding between topic and object implementation (callback).

Should I be creating different object instances of EventI for each topic while I am subscribing? Is there a work around to this because later on I am planning to have one communicator manage say a set of 100 topics to improve performance.

Thanks,
Ritesh

[matthew]

Should I be creating different object instances of EventI for each topic while I am subscribing? Is there a work around to this because later on I am planning to have one communicator manage say a set of 100 topics to improve performance.

You can create a single EventI and register this multiple times with the OA, each using a different id. Then correlate the id with the topic in some way (for example, by convention). In the method call you can examine current.id to find out which lets you discover the topic.

This all being said I fail to see how 100 servants could cause a performance problem.

[ritesh]

Hi,
Thanks for the reply.
My application is likely to handle 3000 odd topics and different communicator would mean different thread pool for handling the servants.

“100” is just an odd guess, but somewhere I need to distribute the load.
Please comment if this approach is good enough or not.

Thanks,
Ritesh

[matthew]

That doesn't sound like a good approach to me. There is nothing to be gained from having separate thread pools performance wise. If you don't have enough threads to take advantage of the available processors then add more threads to the pool.