Multiple threads smashing the stack

[Ric]

I'm currently hunting an elusive bug and have cornered it in a small portion of code where it looks like one Ice thread is smashing the stack of another Ice thread.


Until "adapter->activate();" is called (see the code below) the process is single-threaded.
After "adapter->activate();" is called a number of Ice threads are started, with one working and the rest waiting in a semaphor. This is still as I expected.

When GameClientConnPrx::uncheckedCast() is called however, an extra active Ice-thread shows up shortly, giving me a total of two active Ice-threads for a short while.
I would have expected that only one Ice-thread would be active at any given moment, with the rest waiting in a semaphor.

When the last line (::GameClient::MasterServerPrx::checkedCast(base)) is executed I again have two active Ice-threads, but this time the stack looks broken (I can reproduce it most of the time, but it does seem dependent on timing between threads).

In the stack (see stackdump below), a variable "void * pUserData", which is passed unmodified from "free()" to "_free_dbg()", has changed value from one stackframe to the next.
Also, if I read the code correctly the next stackframe after "msvcr71d.dll!_free_dbg()" should be "msvcr71d.dll!_free_dbg_lk()", not somewhere in "mswsock.dll".
This looks like something smashed the stack.


Am i misusing the Ice library by calling Ice functions (checkedCast() and such) from the main thread after starting "adapter->activate()"?

If so, then how do I interact with the Ice from my main thread?
(The main thread run a single-threaded graphics engine).


The relevant portion of code looks like this:
main() {
<< Initializing qube 3D system and loading Q-databases >>
<< calling initIceLocal()>>
[...]
}
void GameContext::initIceLocal() {
properties = Ice::Application::communicator()->getProperties();
initIceFactories();
Ice::Identity ident = Ice::stringToIdentity("GameClient_"+IceUtil::gener ateUUID());
connection = new GameClientConnImpl();
adapter = Ice::Application::communicator()->createObjectAdapter("GameClient");
adapter->add(connection, ident);
adapter->activate();
thisProxy = GameClientConnPrx::uncheckedCast(adapter->createProxy(ident));
string masterServerProxy = properties->getProperty("GameClient.MasterServer");
Ice::ObjectPrx base = Ice::Application::communicator()->stringToProxy(masterServerProxy);
masterServer = ::GameClient::MasterServerPrx::checkedCast(base);
}



***** Begin Threads at the end of initIceLocal() ******
3752 main GameContext::initIceLocal Normal 0
2336 _threadstartex std::_Tree<std::_Tmap_traits<unsigned int,IceInternal::Handle<IceInternal::EventHandler> ,std::less<unsigned int>,std::allocator<std::pair<unsigned int const ,IceInternal::Handle<IceInternal::EventHandler> > >,0> >::_Buynode Normal 0
3880 _threadstartex IceUtil::Semaphore::wait Normal 0
2148 _threadstartex IceUtil::Semaphore::wait Normal 0
232 _threadstartex IceUtil::Semaphore::wait Normal 0
2372 _threadstartex IceUtil::Semaphore::wait Normal 0
2540 Win32 Thread 7c90eb94 Above Normal 0
3744 _threadstartex _free_dbg Normal 0
***** End Threads at the end of initIceLocal() ******

***** Begin stack of thread 3744 (_free_dbg) ******
ntdll.dll!7c90eb94()
ntdll.dll!7c90e9c0()
mswsock.dll!71a53ca5()
ntdll.dll!7c90d8ef()
mswsock.dll!71a55fa7()
msvcr71d.dll!_free_dbg(void * pUserData=0x00000000, int nBlockUse=0x00000001) Line 1070 + 0x1c C
msvcr71d.dll!free(void * pUserData=0x71aa150c) Line 1025 + 0xb C
05f2e5f0()
ws2_32.dll!71ab2e67()
ice21d.dll!IceInternal::ThreadPool::run() Line 342 + 0x1c C++
ice21d.dll!IceInternal::ThreadPool::EventHandlerTh read::run() Line 813 + 0x1c C++
iceutil21d.dll!startHook(void * arg=0x053fb718) Line 203 + 0x1d C++
msvcr71d.dll!_threadstartex(void * ptd=0x053fb8a8) Line 241 + 0xd C
kernel32.dll!7c80b50b()
kernel32.dll!7c8399f3()
***** End stack of thread 3744 (_free_dbg) ******


Regards,

Richard Jørgensen
(Programmer at Runestone Game Development).

[bernard]

Hi Richard,

I don't see anything incorrect in your code; naturally you can make remote calls after activating an adapter (in the same thread or another thread).

If you look at the Ice source code, the second thread is waiting on a select() call, which is expected. Also, I am not convinced this stack trace shows any problem!

Could you maybe give more details on the bug you're chasing? If there is some memory corruption, I'd recommend to use Purify to get more information.

Best regards,
Bernard

[Nis Baggesen]

If you look at the Ice source code, the second thread is waiting on a select() call, which is expected. Also, I am not convinced this stack trace shows any problem!

Just to clarify I work with Ric.

While there may not be other issues I would say the the three lines Ric refers to do look at bit strange to me as well.

mswsock.dll!71a55fa7()
msvcr71d.dll!_free_dbg(void * pUserData=0x00000000, int nBlockUse=0x00000001) Line 1070 + 0x1c C
msvcr71d.dll!free(void * pUserData=0x71aa150c) Line 1025 + 0xb C

And if nothing else it is obviously an error (of some kind) that free is called on 0x00000000

Ofcourse the strange look of the stack may be caused by a confused debugger, but I don't understand why you think it looks fine.

mvh

Nis

[bernard]

I don't know enough of the C library internals to say that this stack looks suspicious or not.
free(NULL), like delete 0, is correct: it does nothing. But clearly in this stack it's more than just a no-op free(NULL).

We'd really need more information to provide a more useful answer.

Cheers,
Bernard

[Nis Baggesen]

I don't know enough of the C library internals to say that this stack looks suspicious or not.
free(NULL), like delete 0, is correct: it does nothing. But clearly in this stack it's more than just a no-op free(NULL).

We'd really need more information to provide a more useful answer.

Cheers,
Bernard

Fair enough. We'll provide more information as soon as we can. And I'm sorry if I implied I was holding you responcible for what might very well be our own mess. We where mostly going by the fact that something obvoiusly goes wrong somewhere, and that the stack looked if not wrong, then slightly weird going by what the debugger shows of the C internal libraries.

Anyway, as stated we will get back to you if and when we can get something more useful than these stacktrace. If possible we'll try to create a small example and send that, if that would be alright.

[Nis Baggesen]

We have come to the conclusion the the strange looking thread was propably a finished thread. But because the thead is kept in the threadpool Visual C++' debugger didn't detect that the thread had actually finished.

Does this sound remotely reasonable ?

mvh

Nis

[marc]

When the Ice runtime shuts down, all threads in all thread pools are terminated (i.e., the destroy() function uses join() to finish all threads). So there should be no threads sticking around after destroy() had been called.

If you use dynamic thread pools (i.e., thread pools where dynamically threads are added and removed), then threads are also join()ed with, once they are not needed anymore.

[bernard]

Coming back to Ric's first post, I don't see how the uncheckedCast or the checkedCast could have any effect on threads in the Ice thread pools. The stack-trace of these threads would be interesting!

Cheers,
Bernard

[Nis Baggesen]

When the Ice runtime shuts down, all threads in all thread pools are terminated (i.e., the destroy() function uses join() to finish all threads). So there should be no threads sticking around after destroy() had been called.

Well the stack trace is taking with from the debugger at a time when the Ice runtime is still running. Once the runtime is shutdown everything does get cleaned up just as you say.