IeStorm core on stopAll

[Oleh Babyak]

Hi,

We encounter IceStorm crashing time to time with a stack listed below. Debian GNU/Linux 4.0, Ice 3.1.1.

#4 0xb7b72994 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#5 0xb7b703b5 in std::set_unexpected () from /usr/lib/libstdc++.so.6
#6 0xb7b703f2 in std::terminate () from /usr/lib/libstdc++.so.6
#7 0xb7b7052a in __cxa_throw () from /usr/lib/libstdc++.so.6
#8 0xb66dbfd5 in DbEnv::runtime_error () from /usr/lib/libdb_cxx-4.3.so
#9 0xb66dc218 in DbEnv::_stream_error_function () from /usr/lib/libdb_cxx-4.3.so
#10 0xb66dc24b in _stream_error_function_c () from /usr/lib/libdb_cxx-4.3.so
#11 0xb674e4be in __db_errcall () from /usr/lib/libdb_cxx-4.3.so
#12 0xb674e52c in __db_err () from /usr/lib/libdb_cxx-4.3.so
#13 0xb678dcab in __memp_sync_int () from /usr/lib/libdb_cxx-4.3.so
#14 0xb678df87 in __memp_sync () from /usr/lib/libdb_cxx-4.3.so
#15 0xb6767c75 in __dbenv_get_open_flags () from /usr/lib/libdb_cxx-4.3.so
#16 0xb6767d0c in __dbenv_close () from /usr/lib/libdb_cxx-4.3.so
#17 0xb6767ed2 in __dbenv_close_pp () from /usr/lib/libdb_cxx-4.3.so
#18 0xb66df378 in DbEnv::close () from /usr/lib/libdb_cxx-4.3.so
#19 0xb684fc5e in ~SharedDbEnv (this=0x806b108) at SharedDbEnv.cpp:157
#20 0xb685053e in Freeze::SharedDbEnv::__decRef (this=0x806b108) at SharedDbEnv.cpp:215
#21 0xb680d54e in IceUtil::Handle<Freeze::SharedDbEnv>:perator= (this=0x80d21d8, p=0x0) at ../../include/IceUtil/Handle.h:195
#22 0xb680b65f in Freeze::ConnectionI::close (this=0x80d21d0) at ConnectionI.cpp:59
#23 0xb680b6ba in ~ConnectionI (this=0x80d21d0) at ConnectionI.cpp:78
#24 0x0805a38c in IceUtil::Shared::__decRef (this=0x80d21f8) at ../../include/IceUtil/Shared.h:230
#25 0xb680dd04 in IceInternal::decRef (p=0x80d21d0) at Connection.cpp:45
#26 0xb6906d90 in ~Handle (this=0x80d219c) at ../../include/Ice/Handle.h:91
#27 0xb68fbeba in ~TopicI (this=0x80d2150) at TopicI.cpp:360
#28 0xb7d6f7cb in IceInternal::GCShared::__decRef (this=0x80d21c0) at GCShared.cpp:56
#29 0xb690a545 in ~Handle (this=0x80d2574) at ../../include/IceUtil/Handle.h:177
#30 0xb690b405 in ~pair (this=0x80d2570) at /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../include/c++/4.1.2/bits/stl_pair.h:69
#31 0xb690b461 in __gnu_cxx::new_allocator<std:air<std::string const, IceUtil::Handle<IceStorm::TopicI> > >::destroy (this=0xbfa1113f,
__p=0x80d2570) at /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../include/c++/4.1.2/ext/new_allocator.h:107
#32 0xb690b4a5 in std::_Rb_tree<std::string, std:air<std::string const, IceUtil::Handle<IceStorm::TopicI> >, std::_Select1st<std:air<std::string const, IceUtil::Handle<IceStorm::TopicI> > >, std::less<std::string>, std::allocator<std:air<std::string const, IceUtil::Handle<IceStorm::TopicI> > > >::destroy_node (this=0x806b0ac, __p=0x80d2560)
at /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:391
#33 0xb690b538 in std::_Rb_tree<std::string, std:air<std::string const, IceUtil::Handle<IceStorm::TopicI> >, std::_Select1st<std:air<std::string const, IceUtil::Handle<IceStorm::TopicI> > >, std::less<std::string>, std::allocator<std:air<std::string const, IceUtil::Handle<IceStorm::TopicI> > > >::_M_erase (this=0x806b0ac, __x=0x80d2560)
at /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:1266
#34 0xb690b577 in ~_Rb_tree (this=0x806b0ac) at /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:578
#35 0xb690b5d3 in ~map (this=0x806b0ac) at /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../include/c++/4.1.2/bits/stl_map.h:93
#36 0xb690912d in ~TopicManagerI (this=0x806b080) at TopicManagerI.cpp:51
#37 0xb7d6f7cb in IceInternal::GCShared::__decRef (this=0x806b0e0) at GCShared.cpp:56
#38 0xb691a11d in ~Handle (this=0x8066644) at ../../include/IceUtil/Handle.h:177
#39 0xb691950b in ~ServiceI (this=0x8066640) at Service.cpp:88
#40 0x0805a38c in IceUtil::Shared::__decRef (this=0x8066654) at ../../include/IceUtil/Shared.h:230
#41 0xb7ed6c50 in IceInternal::decRef (p=0x8066640) at IceBox.cpp:48
#42 0x0805852c in IceInternal::Handle<IceBox::Service>:perator= (this=0x80c5604, p=0x0) at ../../include/Ice/Handle.h:109
#43 0x080516c1 in IceBox::ServiceManagerI::stopAll (this=0x8066430) at ServiceManagerI.cpp:484
#44 0x08051c45 in IceBox::ServiceManagerI::stop (this=0x8066430) at ServiceManagerI.cpp:184
#45 0x0805b82e in IceBox::IceBoxService::stop (this=0xbfa116c0) at Service.cpp:90
#46 0xb7e13588 in Ice::Service::run (this=0xbfa116c0, argc=@0xbfa11730, argv=0xbfa117a4, initData=@0xbfa116dc) at Service.cpp:555
#47 0xb7e13e78 in Ice::Service::main (this=0xbfa116c0, argc=@0xbfa11730, argv=0xbfa117a4, initData=@0xbfa116dc) at Service.cpp:458
#48 0x0805c5e8 in main (argc=1, argv=0xbfa117a4) at Service.cpp:142

[matthew]

Thanks for the bug report. There are two problems here:

The crash itself is ultimately caused by a destructor throwing an exception when unwinding the stack. The destructor should catch and consume the exception (generating a warning since this is not expected). This is easily fixed and we will do so in the next release of Ice.

However, the ultimate cause of the exception is that somehow the close of the database environment is failing and this is not expected to occur. Are you doing any manual removal or other manipulation of the IceStorm database files?

[Oleh Babyak]

Yes, after brief investigation it looks like DB files had been removed during deinstallation procedure before icestrom was successfully stopped. Thank you for the quick response.