IceStorm trouble..

[wrobbie]

Hi,

I'm trying to solve an issue with clients that are apparently removed from their IceStorm topic subscriber list. It seems that sometimes a client can be kicked off the list and not receive any more updates. Clients in some network segments are more susceptible than others. For example, see the following IceStorm.err snippet:

[ IceStorm: Subscriber: 24F6DF87-12E2-4785-9C95-5718132305FD: publish failed: .\Network.
cpp:557: Ice::ConnectionRefusedException:
  connection refused: WSAECONNREFUSED ]
[ IceStorm: Subscriber: DFD854AF-5898-41B7-B2A1-6F4C035005FD: publish failed: .\Network.
cpp:557: Ice::ConnectionRefusedException:
  connection refused: WSAECONNREFUSED ]
[ IceStorm: Subscriber: C088B534-DBD4-489F-80E6-6B47AF0405FD: publish failed: .\Network.
cpp:557: Ice::ConnectionRefusedException:
  connection refused: WSAECONNREFUSED ]
[ IceStorm: Subscriber: 3440A8F1-9026-49BB-8B00-FBD3C2D005FC: publish failed: .\Network.
cpp:557: Ice::ConnectionRefusedException:
  connection refused: WSAECONNREFUSED ]
[ IceStorm: Topic: Subscribe: 374AEF35-8F36-4E7B-9985-F62D23E68622 QoS:  ]

Here, all four clients are involved in a 'publish failed', and one client is found dead and manually restarted, after which it resubscribes. The other clients survive the incident and do not need restarting. The lost client is one that always seems to have issues; more than any of the others.

I understand Storm will remove subscribers on network failure, but don't really understand how the above scenario can happpen.. I'm using 2.1.2. Does someone have a suggestion?


Another point is that it is rather difficult to find out when a client is removed from a subscriber list. I'm experimenting with keep-alive signals and such, but that is all sub-optimal. Is there a proper way to do this?

(And how about a restarting storm server?)

Cheers,

Rob

[benoit]

The Ice::ConnectionRefusedException from your traces means that the IceStorm service couldn't connect to your subscriber.

This could be because your client isn't listening anymore on the network port it was running when it subscribed. To investigate this, I would bump the network tracing (set Ice.Trace.Network to 2) on both the IceStorm service and the subscriber to see to which host/port the IceStorm service is trying to connect to and which host/port your client is listening on.

There's no easy way to know when a subscriber is removed from the subscriber list other than looking at the IceStorm traces. A subscriber is removed from a topic subscriber list if it failed or if it unsubscribed. Could you detail a little why you need to know when it was removed? Also, what would you like to know about IceStorm service restart?

Benoit.

[wrobbie]

Hi Benoit,

Originally Posted by benoit

To investigate this, I would bump the network tracing (set Ice.Trace.Network to 2) on both the IceStorm service and the subscriber to see to which host/port the IceStorm service is trying to connect to and which host/port your client is listening on.

Ah thanks, I've added that to the IcePack config for Storm.

Originally Posted by benoit

There's no easy way to know when a subscriber is removed from the subscriber list other than looking at the IceStorm traces. A subscriber is removed from a topic subscriber list if it failed or if it unsubscribed. Could you detail a little why you need to know when it was removed? Also, what would you like to know about IceStorm service restart?

Well, my client is a display controller in an information display system and is totally dependent on Storm for its updates. If its subscriber is removed from Storm's list, the client is dead in the water, i.e. never shows an update again. If it can detect this situation somehow, it can recover by syncing its data with the server (pull from server) and resubscribing to Storm to get data pushes again.

Similarly, if IceStorm restarts, all clients are in trouble since Storm's subscription lists are not persistent.

One solution to all this is to add keep-alive methods to all the interfaces that I use to Storm messages, but that is intrusive and a bit ugly..

Am I missing something?

Thanks for your help,

Rob

[benoit]

Originally Posted by wrobbie

Well, my client is a display controller in an information display system and is totally dependent on Storm for its updates. If its subscriber is removed from Storm's list, the client is dead in the water, i.e. never shows an update again. If it can detect this situation somehow, it can recover by syncing its data with the server (pull from server) and resubscribing to Storm to get data pushes again.

If the subscriber is removed from the topic, it's because of a communication problem with your client. If this happens, IceStorm would most likely not be able to notify your client that its subscription was removed.

Note that you can configure Ice to retry several times at different time intervals to ensure that it won't give up to soon in case of a network problem -- see the Ice.RetryIntervals property for more information (the default is to retry only once immediately). You could configure this property on your IceStorm service if it's possible for a network link to be down for some time.

Similarly, if IceStorm restarts, all clients are in trouble since Storm's subscription lists are not persistent.

Adding persistency for subscribers is on our TODO list, if you have any commercial interest in this feature we could certainly look into it sooner rather than later. Please contact us at info@zeroc.com if that's the case.

One solution to all this is to add keep-alive methods to all the interfaces that I use to Storm messages, but that is intrusive and a bit ugly..

Another solution would be to notify your subscribers through a topic that the IceStorm service is going to be shutdown but your clients will still need to re-subscribe to the IceStorm service when it comes up again.

Benoit.