Response thread

[ngambek2003]

I set it for the service..

Here's a screenshot of IceGrid GUI (attached).

Interestingly enough for the cases when it didn't hang, I do see multiple "JavaGameServer-lucketicket-1-GameService-lucketicket-Ice.ThreadPool.Server" threads in the dump. So it seems like sometimes the Communicator doesn't get setup with the right pool?, even though the IceGrid GUI shows that it does?

Could this be because it is running on a VMware virtual machine??

[ngambek2003]

I have another server dump that shows that two threads are in the server pool (attached). In this case the system still hangs.

What I'm seeing is that even though I set the size to, say 5, I don't consistently get 5 threads in the dump.

[benoit]

This latest thread dump is definitely suspicious. Your configuration looks fine. The thread dump shows 2 threads for your service server thread pool: one is waiting on the serverToClient remote call and the other is idle waiting on the thread pool monitor.

I don't see any threads waiting on the thread pool selector (which monitors Ice connections and eventually read data from connections). This is wrong -- there should always be such a thread unless there are no more threads available in the thread pool (which isn't the case here).

So something must have happened to this thread.

Do you see any errors in the stderr output of the server? You should see thread dumps of the JVM when sending the SIGQUIT signal, is perhaps something not working with the standard error output? Do you use any logging facilities such as log4j which could interfere with Ice stderr/stdout redirection (there used to be some issues with log4j, see this thread for example, I'm not sure this is still relevant though)?

At a first step, you could try running the icegridnode from the command line and disable the stderr/stdout file redirection. The stderr/stdout output of servers should then go to the console and you should see thread dumps of Java servers when you send them the SIGQUIT signal. If this works, you could try to reproduce the deadlock with this configuration and see if any errors show up.

Cheers,
Benoit.

[ngambek2003]

I am using log4j. Once I turn off redirection, I can get the dump by sending the pid kill -3.

I have full output from the node now. I don't really see any error but I do see these. So the pool size keeps shrinking until it's at 1.

-- 5/3/10 16:59:23:246 JavaGameServer-lucketicket-1-GameService-lucketicket: ThreadPool: shrinking Ice.ThreadPool.Server: Size=4

I can send you the full log via email. Is there an email I can send it to?

Thanks
Budyanto

[benoit]

You can send it to support@zeroc.com.

Cheers,
Benoit.