Server, running on IceGridNode, doesn't stop when inheriting handles

[andy_84]

Hi all.

I have the following IceGrid: 2 nodes, one running server A, exchanging data with server B, running on the other node (doesn't matter whether both nodes are running on the same computer or not). Server B calls CreateProcess to run a separate App (for example, notepad.exe).

The problem is that if I stop server A through IceGridGUI or icegridadmin.exe, and App was started inheriting handles of server B - A hangs for a minute, then stops. If I kill App - server A stops instantly. If App doesn't inherit B's handles - A stops instantly.

ProcessExplorer shows that App inherits from server B several objects of type 'File' with name '\Device\Afd'.
TCPView shows a single established connection between A and B, which disappeares only in a minute after I stop server A.

Trace:

-- 05/04/11 10:54:48.875 A.server: Network: sent 14 of 14 bytes via tcp
local address = 172.16.130.31:34454
remote address = 172.16.130.31:34453

but there is no corresponding 'Network: closing tcp connection' message for this ports, like this:

-- 05/04/11 10:54:48.875 A.server: Network: sent 14 of 14 bytes via tcp
local address = 172.16.130.31:34460
remote address = 172.16.130.31:32412
-- 05/04/11 10:54:48.875 A.server: Network: closing tcp connection
local address = 172.16.130.31:34460
remote address = 172.16.130.31:32412

In my scenario server A might not send any stop signal to server B, causing App to stop, and I need to stop A instantly. How can I achieve this?

Windows XP x64, ICE 3.4.1.

Thanks.

[bernard]

Hi Andy,

Could you clarify the interactions between server A and server B: is server A sending requests to server B (and waiting for responses from B), or vice versa?

Did you try to attach to server A while it's stopping slowly, to see what it's waiting for?

Best regards,
Bernard

[andy_84]

Hi! Thanks for paying attention.

Server A sends async requests to B, while B is making sync requests to A through a oneway proxy.

Yes, I've tried, A is waiting here:

void
Ice::ConnectionI::waitUntilFinished()
{
IceUtil::Monitor<IceUtil::Mutex>::Lock sync(*this);

//
// We wait indefinitely until the connection is finished and all
// outstanding requests are completed. Otherwise we couldn't
// guarantee that there are no outstanding calls when deactivate()
// is called on the servant locators.
//
while(_state < StateFinished || _dispatchCount > 0)
{
wait(); <--- is waiting here
}

...

This is the same connection TCPView shows me. I'm worrying that inheriting handles is the real cause - App must inherit B's handles...

Regards, Andy.

[benoit]

Hi,

Yes, this is expected when inheriting the handles. The forked App process gets a handle on the TCP/IP connection established between A and B. A TCP/IP socket connection gets closed only once all the handles are closed (so here both B and App should close the socket handle for the TCP/IP connection to close).

Ice graceful connection closure waits for the peer to close the connection before considering the connection to be closed (see 36.6.1 here) and this is what is happening here: A waits for B to close its side of the TCP/IP connection to shutdown.

You should either not inherit handles or use a smaller close timeout (you can override the close timeout with Ice.Override.CloseTimeout in Ice >= 3.4.x).

Cheers,
Benoit.

[andy_84]

Benoit, great thanks!

By the way, Ice.Override.CloseTimeout is not mentioned in ICE 3.4.1 manual, only here http://download.zeroc.com/Ice/3.4/Ice-3.4.1-CHANGES

Regards, Andy