Load Balancing with IceStorm

[zhi]

Does IceStorm support load balancing policies like those of ICEGrid's replica's? If so, how do I go about achieving this functionality?

[benoit]

Hi,

No, IceStorm doesn't support any load balancing. However, you can deploy several IceStorm instances with IceGrid and use replica groups to provide load balancing for the topic publisher objects (you'll have to manually setup the topic on each service and your subscribers will have to subscribe to the topic from each service instance).

The next Ice version will provide a demo that shows how to do this.

Cheers,
Benoit.

[zhi]

This would provide load balancing among the publisher objects, but how about load balancing between subscribers attached to a publisher?

[benoit]

Could you detail a bit more what you're trying to achieve? I don't see how load balancing for the subscribers would be useful. In general, a subscriber wants to receive all the messages from the topic so I'm not sure why you would need load balancing for the subscribers.

Cheers,
Benoit.

[zhi]

Hi Benoit, thanks for the quick reply.

I'm building a distributed computation system whereby a client issues a job to a server which splits it up into sub-jobs and consolidates the results. The server then sends the results back to the client.

Upon splitting up the job into sub-jobs, the server would use Publish/Subscribe semantics to distribute the workload out to subscribers. The reason I'm using Publish/Subscribe is to allow the "result consolidator" process to be able to be substituted.

One of the key goals of the system is performance. Using IceGrid, I was able to distribute the workload based on a replica policy such as "adaptive". The replicas are stand-alone nodes which are solely dedicated to processing sub-jobs from the server. However, using IceGrid alone would not achieve modifiability in that the "result consolidator" process would not be able to be substituted.

I would therefore like to use IceStorm to implement Publish/Subscribe, having the stand-alone nodes as subscribers and also adopt the load balancing features that IceGrid has.

[benoit]

Hi,

I'm afraid it's still not clear to me how you could use IceStorm for this. Communications with IceStorm are unidirectional and a message published on a topic is expected to be received by all the subscribers. It seems to me that it would be better if you implement your own service for splitting up the job into sub-jobs and consolidating the results.

Cheers,
Benoit.

[Vivien Delage]

I am actually trying to use Icestorm server with Icegrid but I am a little bit lost using the current documentation of how to implement icebox with icegrid.

Is this demo already available? It would be a great help for me!

[benoit]

Hi,

This demo will only be available with the upcoming Ice 3.2. Did you check out the IceStorm templates from the config/templates.xml file included with your Ice distribution?

You should be able to re-use these templates in your application by either copying the IceStorm descriptors in your application dessriptor or by importing the default templates with the default template mechanism (see section 36.6.4 of the Ice manual for more information).

Let us know if you still have problems with deploying IceStorm in your application with IceGrid!

Cheers,
Benoit.

[Vivien Delage]

Thank you for your quick answer.
I have not seen those templates before.
Now by using this it is working very fine.

Thanks a lot!