IceStorm using lots of memory

[borax00]

Hi,

I'm running icestorm, and I noticed that it's using an awful lot of memory.
When I noticed this, I shut down every process that might be talking to IceStorm to see if one of them was responsible/related.
After shutting them all down, the relevant line from 'top' is:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2313 mrsys 25 0 736m 209m 4760 S 0.0 44.3 1:45.22 icebox

This is a lot for a process doing nothing...
I'm out of ideas for how to debug this, it's difficult to see what IceStorm is doing -- any ideas?

I'm running IceStorm in non-persistent mode, on Linux.
My Ice version is 3.4.0.

Here's my icebox/icestorm config:

icebox:
Ice.Warn.Connections=1
Ice.PrintAdapterReady=1
Ice.MessageSizeMax=30000
IceBox.ServiceManager.Endpoints=tcp -p 9998
IceBox.InheritProperties=1
IceBox.LoadOrder=IceStorm
IceBox.Service.IceStorm=IceStormService,34:createI ceStorm --Ice.Config=/etc/marathon/icestorm.cfg
IceBox.UseSharedCommunicator.IceStormService=1

icestorm:
IceStorm.TopicManager.Endpoints=default -p 10000
Ice.Warn.Connections=1
Ice.Warn.Dispatch=2
Ice.Warn.Endpoints=1
Ice.Warn.AMICallback=1
Ice.Warn.UnknownProperties=1
Ice.ThreadPool.Client.Size=1
Ice.ThreadPool.Client.SizeWarn=0
Ice.ThreadPool.Client.SizeMax=2
IceStorm.Discard.Interval=60000
IceStorm.Trace.TopicManager=1
IceStorm.Trace.Topic=0
IceStorm.Trace.Subscriber=1
IceStorm.Publish.Endpoints=default
IceStorm.Transient=1


I can't see anything interesting or out-of-the-ordinary in the logs.
After I killed everything that interacted with IceStorm I see no output whatsoever in the logs (as expected).


Thanks for any help you can offer,


Alex

[borax00]

I have some more information on this -- I figured out how to reproduce the problem, it was due to a bug in my code where lots of data is sent to IceStorm extremely rapidly.

So it seems reasonable that IceStorm uses lots of memory while this is occurring, I expect that my process was sending data faster than IceStorm could deliver it, so it needs to allocate really big buffers -- does that sound right?

But should the memory usage decrease when I stop sending data? Is there any way IceStorm will reclaim that memory without being re-started?


Thanks,

Alex

[benoit]

Hi,

Yes IceStorm could be consuming quite a bit of memory if you publish many large messages at the same time. I would expect IceStorm to release the memory after shutting down all the subscribers and once it has finished trying reconnecting to them (if they were not shutdown in a cleanly manner, i.e.: if they were not explicitly unsubscribed from the topic). This might take a bit of time (by default the timeout for subscribers is set to 60s so it shouldn't take a lot more than a couple of minutes).

It's also not clear to me if RES is supposed to immediately go down after the process releases the memory... this most likely depends on the memory allocator implementation. On which platform do you see this?

Cheers,
Benoit.

[borax00]

Hi Benoit,

I wasn't sending any large messages, but a huge number of tiny ones.

I left IceStorm running with no subscribers overnight after I saw this, it was the same in the morning.

I'm running on Linux.


Alex