Multicast Events

[jody]

Very interesting, indeed!

We currently make heavy use of TAO.

Specifically, I am interested in IceStorm (I can not find any detailed information in the book). Please indulge my neophite questioning.

IceStorm is presented as an efficient distributed event dispatcher, but I can not find any details in the book. You indicate that it can be federated for very large applications. Additionally, you refer to a stock ticker application as an example. Assume 10,000 different symbols, 200 different kinds of messages, and an average of 8,000 messages per second (with spikes of 20,000 messages per second). Also, assume 300 clients, each filtering some subset of of messages based on a combination of symbols and message types -- most clients receive a substantial subset of the messages.

How would IceStorm scale to this application?
Are there any plans for multicast support?
How well does the service tolerate network or host faults?
If communication between the consumer/supplier/eventservice is lost, how is it reestablished?
Are the any automatic recovery mechanisms?
What are the reliability/recover guarantees, especially w.r.t. messages sent by the supplier -- can the supplier assume that all registered consumers will receive the message?
What if a consumer crashes or loses network connectivity for some period, will it still get all the messages (via some sort of resend protocol) when it reestablishes communication with the event service?

Thanks!!

Jody Hagins

[matthew]

Hi,

Thanks you for your interest!

How would IceStorm scale to this application?

IceStorm could in theory handle the load with federation support. Due to batching (where multiple method invocations are sent as one physical message) the actual number of physical messages delivered can be much less than the individual events.

Are there any plans for multicast support?

With sufficient interest support for unreliable multicast could be added.

How well does the service tolerate network or host faults?
If communication between the consumer/supplier/eventservice is lost, how is it reestablished?
Are the any automatic recovery mechanisms?

The service at present is quite primitive. If a faulty consumer is detected it is booted from the system and all queued events are thrown out.

What are the reliability/recover guarantees, especially w.r.t. messages sent by the supplier -- can the supplier assume that all registered consumers will receive the message?

No, it cannot. In fact, I'd say for any notification style system this cannot be guaranteed since consumers & suppliers are decoupled.

What if a consumer crashes or loses network connectivity for some period, will it still get all the messages (via some sort of resend protocol) when it reestablishes communication with the event service?

No, any messages that are not delivered are lost. There is no queuing yet.

Note that this is a first release. We are very interested in any features that you need to make this a useful product for your application.

Regards, Matthew

[marlowa]

Originally posted by matthew
Hi,

Thanks you for your interest!

With sufficient interest support for unreliable multicast could be added.

Regards, Matthew

Yes, I am interested in reliable multicast. There are number of competing protocols in draft RFC stage and a number of actual implementations. If Ice is to offer RMC maybe it could build on existing s/w even though this would create a dependency.

[marc]

Originally posted by marlowa
Yes, I am interested in reliable multicast. There are number of competing protocols in draft RFC stage and a number of actual implementations. If Ice is to offer RMC maybe it could build on existing s/w even though this would create a dependency.

Since Ice offers dynamic protocol plug-ins, we could add this without introducing any new dependencies to the Ice core.

This is certainly an area where we want to extend Ice in the future. It's just a matter of time & priorities

[marlowa]

Since Ice offers dynamic protocol plug-ins, we could add this without introducing any new dependencies to the Ice core.

This is certainly an area where we want to extend Ice in the future. It's just a matter of time & priorities

Have there been any ICE developments in this area since my last post? I have not looked at ICE for a while a now but a key ability that I require of middleware I use is to be able to do pub-sub with reliable multicast.

As far as protocols are concerned, has anyone looked at NORM? It is supposedly the successor to PGM. Of course, PGM support would be good as a first step.

[marc]

Sorry, but so far we have not worked on reliable multicast for Ice.

[pjtait]

Any progress to report?

[marc]

We currently do not have any plans for adding a reliable multicast protocol to Ice. If you have a commercial demand for such a feature, please contact us at info@zeroc.com.