IceBox segfault

[n2503v]

Ice-3.3.0, Linux

IceBox running IceStorm (the only service).
Fairly loaded, total number of clients and servers: up to 180, often subscribing and unsubscribing. This is the highest number we've ever tried. This is also the first time we've seen icebox segfault.

At some point the IceBox segfaulted with the following syslog message:

Apr 29 21:47:36 <hostname> kernel: [416257.979832] icebox[19691]: segfault at 24 ip b7ad7305 sp af7fe950 error 4 in libstdc++.so.6.0.10[b7a41000+e2000]

I have backtraces from the core dump (all 19 threads) but they don't fit into this post. Let me know if you want to examine them.

icebox.cfg

IceStorm.TopicManager.Proxy=IceStorm/TopicManager:default -p 10000
Ice.Warn.Connections=1
Ice.PrintAdapterReady=1
Ice.MessageSizeMax=30000
IceBox.ServiceManager.Endpoints=tcp -p 9998
IceBox.InheritProperties=1
IceBox.LoadOrder=IceStorm
IceBox.Service.IceStorm=IceStormService,33:createIceStorm --Ice.Config=/etc/marathon/icestorm.cfg
IceBox.UseSharedCommunicator.IceStormService=1

icestorm.cfg

IceStorm.TopicManager.Endpoints=default -h localhost -p 10000: default -h <hostname> -p 10000
Ice.Warn.Connections=1
Ice.Warn.Dispatch=2
Ice.Warn.Endpoints=1
Ice.Warn.AMICallback=1
Ice.Warn.UnknownProperties=1
Ice.ThreadPool.Client.Size=1
Ice.ThreadPool.Client.SizeWarn=4
Ice.ThreadPool.Client.SizeMax=4
Ice.ThreadPool.Server.Size=1
Ice.ThreadPool.Server.SizeWarn=3
Ice.ThreadPool.Server.SizeMax=3
IceStorm.Send.Timeout=60000
IceStorm.Trace.TopicManager=1
IceStorm.Trace.Topic=2
IceStorm.Trace.Subscriber=1
IceStorm.Publish.Endpoints=default
IceStorm.Publish.ThreadPool.Size=1
IceStorm.Publish.ThreadPool.SizeWarn=5
IceStorm.Publish.ThreadPool.SizeMax=5
IceStorm.TopicManager.ThreadPool.Size=1
IceStorm.TopicManager.ThreadPool.SizeWarn=2
IceStorm.TopicManager.ThreadPool.SizeMax=3
Freeze.DbEnv.IceStorm.DbHome=/home/mrsys/sys/icestorm/stormdb

Thanks for your help,
alex

[benoit]

Hi Alex,

Yes, seeing the stack traces would be helpful. Can you attach them to this thread with a text file? Also, any reasons why you set the thread pool sizes of IceBox and IceStorm?

Cheers,
Benoit.

[n2503v]

Hi Benoit,
the traces are attached.

Regarding the thread pool sizes, I am not entire sure. The config file has evolved from the pre-Ice3, when the configs were different. Now that I look at it, it seems that the following would apply to the IceBox but would also get inherited by IceStorm (is this correct?).

Ice.ThreadPool.Client.Size=1
Ice.ThreadPool.Client.SizeWarn=4
Ice.ThreadPool.Client.SizeMax=4
Ice.ThreadPool.Server.Size=1
Ice.ThreadPool.Server.SizeWarn=3
Ice.ThreadPool.Server.SizeMax=3

cheers,
alex

[matthew]

...
Regarding the thread pool sizes, I am not entire sure. The config file has evolved from the pre-Ice3, when the configs were different. Now that I look at it, it seems that the following would apply to the IceBox but would also get inherited by IceStorm (is this correct?).
...

It depends how you've configured IceBox & IceStorm, but if you've used separate configuration files, and have not set IceBox.InheritProperties then IceStorm will not inherit the IceBox configuration properties.

In general, with IceStorm you do not need to configure the thread pools with Ice 3.3, since they default to appropriate sizes.

[matthew]

The crash you've reported is a bug in IceStorm. The simplest workaround is not to trace the subscribers (that is set IceStorm.Trace.Topic=0). If you need a fix let us know, and we'll prepare a patch.

[n2503v]

Hi Matthew,
thanks for tracking it down.

The work around will do us for now.

In general, with IceStorm you do not need to configure the thread pools with Ice 3.3, since they default to appropriate sizes.

I'll re-read your article on IceStorm and will get back to you if I still have questions on thread pool configuration.

Cheers,
Alex