Assertion failed in IceDB/FreezeTypes.cpp

[lafayej]

Hello,

We have regular unexpected failures of our IceStorm setup. Diving into the logs, we found an occurrence of the line below every time we had a failure:

-- 09/25/11 16:30:10.689 RTCSStorm-us-3-IceStorm: Topic: mainDispatchTopic: subscribeAndGetPublisher: B8B9BFC0-CD32-42C5-A7CD-19798F87B9AE
icebox: ../../../src/IceDB/FreezeTypes.cpp:118: virtual void FreezeDB::DatabaseConnection::beginTransaction(): Assertion `!_connection->currentTransaction()' failed.

We run two icestorm replica groups. Each of them have 4 icestorm instances, distributed on 2 servers (2*2). The assertion is raised only on the last instance of the group. We have had the problem for some time and we haven't had a single assertion failed entry in all the icestorm log files except the last one of the group.
The problem is only appearing when we are registering our subscribers. Those subscribers register simultaneously and we first had the problem when increasing their number.

I tried to reproduce the problem in a lab setup but I have not been able to reproduce it so far. Do you have any clue on how to find/resolve the problem ?

Our setup:
* Linux RHEL5, x86_64
* Ice 3.4.1, home-compiled with optimizations disabled
* FreezeDb linked with bdb provided with our distro (RHEL5): libdb_cxx-4.3.so

IceStorm configuration files:

# Configuration file (12/23/10 08:32:26.035)

# Server configuration
Ice.Admin.ServerId=RTCSStorm-us-0
Ice.Admin.Endpoints=tcp -h 127.0.0.1
Ice.ProgramName=RTCSStorm-us-0
IceBox.Service.IceStorm=IceStormService,34:createIceStorm --Ice.Config\='/opt/user/ice-node/node-data/servers/RTCSStorm-us-0/config/config_IceStorm'
IceBox.InstanceName=RTCSStorm-us-0
IceBox.LoadOrder=IceStorm\
# Server descriptor properties
Ice.StdOut=/opt/collector/ice-node/icestorm/RTCSStorm-us-0.log
Ice.StdErr=/opt/collector/ice-node/icestorm/RTCSStorm-us-0.log
Ice.Default.Locator=IceGrid/Locator:tcp -p 4061 -h rtcs-registry-master.data.cfm.fr

# Configuration file (12/23/10 08:32:26.035)

# Object adapter IceStorm.Node
IceStorm.Node.Endpoints=default -h usen105l29.us.cfm.fr
IceStorm.Node.AdapterId=RTCSStorm-us-0.IceStorm.IceStorm.Node
# Object adapter IceStorm.Publish
IceStorm.Publish.Endpoints=default -h usen105l29.us.cfm.fr
IceStorm.Publish.AdapterId=RTCSStorm-us-0.IceStorm.IceStorm.Publish
IceStorm.Publish.ReplicaGroupId=RTCSStorm-us-PublishReplicaGroup
# Object adapter IceStorm.TopicManager
IceStorm.TopicManager.Endpoints=default -h usen105l29.us.cfm.fr
IceStorm.TopicManager.AdapterId=RTCSStorm-us-0.IceStorm.IceStorm.TopicManager
IceStorm.TopicManager.ReplicaGroupId=RTCSStorm-us-TopicManagerReplicaGroup
# Service descriptor properties
IceStorm.InstanceName=RTCSStorm-us
IceStorm.Trace.TopicManager=2
IceStorm.Trace.Topic=1
IceStorm.Trace.Subscriber=1
IceStorm.Trace.Election=1
IceStorm.NodeId=0
Ice.StdOut=/opt/user/ice-node/icestorm/RTCSStorm-us-0.log
Ice.StdErr=/opt/collector/ice-node/icestorm/RTCSStorm-us-0.log
Freeze.DbEnv.IceStorm.DbHome=/opt/collector/ice-node/icestorm/db-RTCSStorm-us-0/
IceStorm.Trace.Network=1
Ice.Default.Locator=IceGrid/Locator:tcp -p 4061 -h rtcs-registry-master.data.cfm.fr

[benoit]

Hi,

Could you get the stack trace of the assert from the core file and post it here? You can enable core files by running "ulimit -c unlimited" in the environment where the process is running.

Cheers,
Benoit.