Crash on exit, after upgrade to Ice 3.2

[vanco]

I just upgraded from Ice 3.1.1 to Ice 3.2.1. After the upgrade, one of my servers crashes whenever I try to shut it down. The shutdown is through icegridadmin (server stop <server name>).

Here's the crash stack:

*** glibc detected *** /tmp/myexe: double free or corruption (fasttop): 0x00000000007c9400 ***
======= Backtrace: =========
/lib/libc.so.6[0x2afe7f20faad]
/lib/libc.so.6(cfree+0x76)[0x2afe7f211796]
/usr/lib/libIce.so.32(_ZNSt8_Rb_treeISsSt4pairIKSsN11IceInt ernal6HandleIN3Ice6ObjectEEEESt10_Select1stIS7_ESt 4lessISsESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E +0x59)[0x2afe7e579269]
/usr/lib/libIce.so.32(_ZNSt8_Rb_treeISsSt4pairIKSsN11IceInt ernal6HandleIN3Ice6ObjectEEEESt10_Select1stIS7_ESt 4lessISsESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E +0x30)[0x2afe7e579240]
/usr/lib/libIce.so.32(_ZNSt8_Rb_treeIN3Ice8IdentityESt4pair IKS1_St3mapISsN11IceInternal6HandleINS0_6ObjectEEE St4lessISsESaIS2_IKSsS8_EEEESt10_Select1stISF_ES9_ IS1_ESaISF_EE8_M_eraseEPSt13_Rb_tree_nodeISF_E+0x3 6)[0x2afe7e579986]
/usr/lib/libIce.so.32(_ZN11IceInternal14ServantManager7dest royEv+0x34b)[0x2afe7e57799b]
/usr/lib/libIce.so.32(_ZN3Ice14ObjectAdapterI7destroyEv+0x1 ab)[0x2afe7e51c32b]
/usr/lib/libIce.so.32(_ZSt8for_eachISt17_Rb_tree_iteratorIS t4pairIKSsN7IceUtil6HandleIN3Ice14ObjectAdapterIEE EEEN15IceUtilInternal16SecondVoidMemFunIS2_S6_S7_E EET0_T_SE_SD_+0x7d)[0x2afe7e51bbfd]
/usr/lib/libIce.so.32(_ZN11IceInternal20ObjectAdapterFactor y7destroyEv+0xbc)[0x2afe7e519cac]
/usr/lib/libIce.so.32(_ZN11IceInternal8Instance7destroyEv+0 x39f)[0x2afe7e4e431f]
/usr/lib/libIce.so.32(_ZN3Ice13CommunicatorI7destroyEv+0x28 )[0x2afe7e4a14f8]
/usr/lib/libIce.so.32(_ZN3Ice11Application4mainEiPPcRKNS_18 InitializationDataE+0x3cc)[0x2afe7e4916fc]
/usr/lib/libIce.so.32(_ZN3Ice11Application4mainEiPPc+0x43)[0x2afe7e491d03]
/tmp/myexe[0x414cb0]
/lib/libc.so.6(__libc_start_main+0xf4)[0x2afe7f1c0b74]
/tmp/myexe(_ZN12IceDelegateD3Ice6Object7ice_isAERKSsPKS t3mapISsSsSt4lessISsESaISt4pairIS2_SsEEE+0x69)[0x414609]

The only extraordinary thing about the server is that it attempts to implement two interfaces. The management of all the memory is done through smart pointers:

class MyApp : public Ice::Application
{
virtual int run(int argc, char* argv[]) {
//...initialization cut out
Ice::CommunicatorPtr ic = this->communicator();

// implementation of the two interfaces. Doesn't directly derive from anything
boost::shared_ptr<MyImpl> myImpl(new MyImpl(ic, <some args>));

// First interface, stores myImpl as a shared_ptr
MyI1Ptr p1 = new MyI1(myImpl);
// Second interface, stores myImpl as a shared_ptr
MyI2Ptr p2 = new MyI2(myImpl);

// Make the facets and activate them
Ice::ObjectPrx prx = adapter->add(p1, ic->stringToIdentity("myname"));
adapter->addFacet(p2, prx->ice_getIdentity(), "facetname");
adapter->activate();

// OK, the server is now up and running. Wait for it to shutdown.
ic->waitForShutdown();
}
};

Any help would be appreciated...

<Edit - added valgrind output>
For kicks I ran valgrind on the executable. The output is full of 'Invalid read of size 8' and 'Address is 24 bytes inside a block of size 48 free'd'. I've attached two of the pertinent reports, all caused on application exit.

[matthew]

The stack trace indicates that its crashing while releasing the servants, so I would point my finger squarely at the memory management of that shared object. Without more details, or a code example, its hard to tell you exactly what is going wrong however. You could try changing the use of boost:shared_ptr to IceUtil::Handle, and derive MyImpl for IceUtil::Shared and see if that fixes your problem.

[vanco]

Sorry - changing boost::shared_ptr to IceUtil::Handle and deriving MyImpl from IceUtil::Shared didn't work. That makes sense - since both just do reference counting.

I'll try to cook up a simple example that replicates my problem. In the meantime, any suggestions are welcome.

[vanco]

Forgot to mention this: the crash on exit only happens once at least one of the servants have been invoked at least once. When I stop the server immediately after it starts up, without executing any of its functions, I see no crash on exit.

[vanco]

No luck reproducing the problem on a small testcase - the small testcase works like a charm.
Instead, I tried going top-down - started removing code from my server, until it started working. Interestingly, it started working when I removed some completely innocuous line of code (a=b*c, where all three are floating points). It almost looks like either an uninitialized variable, or a race condition, most likely in the guts of Ice (since valgrind cannot find it in my code).
Building the debug version of Ice to see if that will show anything...

[vanco]

Managed to track the bug to user code. There was a bug in the destruction sequence of the MyImpl object.

Curiously, though - this bug was never triggered in 3.1.1. It's almost like the MyImpl object was never destroyed in 3.1.1... If it were destroyed, this bug would have shown up immediately.