Connection between Linux Server and Windows Client

[jaleira]

Hello everybody,

I am facing a weird problem:
After creating a Linux server for registration, I now created an application on Windows (XP, VS2005). This application has to register on the linux server after starting. That is working. Now the Linux server pings the connected clients every second to see, if everything is okay. If an exception occurs during pinging a client, the client is removed from the list of registered clients from the server. This pinging is working until i shut down the windows application. Sometimes the linux server receives an exception and removing the client from the list works fine. But more often the ping simply seems to run into some kind of nirvana and no exception is thrown... The server application than hangs up and has to be hardly killed.
Amazingly the same mechanisms are running quite well when only using clients running under linux...

Any ideas?

Regards,
Jaleira

[benoit]

Hi Jaleira,

There can be many reasons why the ping hangs and why it takes a long time to fail. Is the Windows firewall enabled perhaps? You could try to enable network tracing on the server (with --Ice.Trace.Network=2) to see when it hangs.

What about having the client ping the server instead? If the server, doesn't receive a ping from a client within a given time interval, it would consider the client dead and remove it from its list of registered clients. The advantage here is that the server doesn't have to establish a connection to the client and it won't have to deal with clients being unreachable. See the demo/Ice/session example in the Ice distribution for an example.

Cheers,
Benoit.

[jaleira]

Hello Benoit,

the windows firewall is enabled, but I configured the firewall not to block the program.

Network tracing gives the following exception on linux side after terminating the windows client:
./registryserver: warning: connection exception:
TcpTransceiver.cpp:285: Ice::ConnectionLostException:
connection lost: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
local address = 141.*.*.118:53212
remote address = 141.*.*.149:3521
./registryserver: warning: connection exception:
TcpTransceiver.cpp:285: Ice::ConnectionLostException:
connection lost: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
local address = 141.*.*.118:10001
remote address = 141.*.*.149:3518

After that sending a ping never terminates.

Cheers,
Jaleira

[marc]

My guess is still that a firewall somehow prevents proper notification about your client not running anymore.

In any case, I recommend to set a timeout for the ping. This is a good idea in general. For example, if somebody would just "cut the cable" you wouldn't get an exception either, because the client host wouldn't be reachable at all, and thus couldn't send a notification to the server, meaning that the connection attempt (initiated by the ice_ping()) would just hang.

I also recommend to run the server (which sends the pings to the client) with Ice.Trace.Network=2 to see what's going on.

[jaleira]

I see...
How can i set a timeout for the ping?

[marc]

Originally Posted by jaleira

I see...
How can i set a timeout for the ping?

Please see the manual for details on how to use timeouts.

[jaleira]

Okay. I set a timeout for the ping. See the source code below:

try 
{
	cout << "---> sending ping"<<endl;
	(*theIterator).servent->ice_timeout(100);
	(*theIterator).servent->ice_ping();
	cout << "received something on ping" <<endl;
}//try
catch(const Exception& ex)
{
	cerr<<"--> (ping) Cought Exception: "<< ex << endl;
	(*theIterator).serviceList.clear();
	cout << "--> (ping)" << (*theIterator).servent->ice_getIdentity().name << " is down!"<<endl;
	emit serventDown(*theIterator);
}

During tracing the network and after shutting down the windows client I see, that Ice recognizes a Ice::ConnectionLostException, but my server does not catch this exception. So my question is: Why not?
The server is still hanging up when trying to ping the client... (But not always, sometimes it's working and I catch some exception during pinging, so removing the client is working)

Regards,
Jaleira

[marc]

It's not entirely clear to me what you mean. A server doesn't catch exceptions, all it can do is to log them.

I don't know why your ping hangs. If there is a timeout, it should return eventually when the timeout expires.

It would really help if you would trace the client and the server with Ice.Trace.Network=2, as suggested.

[marc]

BTW, what might confuse you is that there are two connections, one from the client to the server, and another from the server to the client for the ice_ping(). If you use Ice.Warn.Connections=1, then you might get a ConnectionLostException warning for the connection from the client to the server. (The server can only warn here--there is no user code that could intercept this exception.) However, this has nothing to do with the connection from the server to the client, which is used for ice_ping().

Again, trace both your client and server with Ice.Trace.Network=2, and show us the logs for both the client and the server for the time period when you get the hang. You can also add Ice.Trace.Protocol=1 for even more debug information. The warnings output from Ice.Warn.Connections are not sufficient to debug the problem.

(Of course, all the above assumes that you do not use bi-directional connections, which you don't if you do not explicitly request them in your code.)

[jaleira]

Hello again,

first of all: thanks for the quick answers!
Here are the Tracing messages on the client side:

[ PolarisServent.exe: Protocol: received request
  message type = 0 (request)
  compression status = 0 (not compressed; do not compress response, if any)
  message size = 53
  request id = 7
  identity = Polaris-Servent
  facet =
  operation = ice_ping
  mode = 1 (nonmutating)
  context =  ]
[ PolarisServent.exe: Protocol: sending reply
  message type = 2 (reply)
  compression status = 0 (not compressed; do not compress response, if any)
  message size = 25
  request id = 7
  reply status = 0 (ok) ]

And here on the server side:

---> sending ping
[ ./registryserver: Protocol: sending request
  message type = 0 (request)
  compression status = 0 (not compressed; do not compress response, if any)
  message size = 53
  request id = 8
  identity = Polaris-Servent
  facet =
  operation = ice_ping
  mode = 1 (nonmutating)
  context =  ]
[ ./registryserver: Network: shutting down tcp connection for writing
  local address = 141.*.*.118:53536
  remote address = 141.*.*.221:1574 ]
[ ./registryserver: Network: closing tcp connection
  local address = 141.3*.*.118:53536
  remote address = 141.*.*.221:1574 ]
[ ./registryserver: Network: shutting down tcp connection for writing
  local address = 141.*.*.118:10001
  remote address = 141.*.*.221:1571 ]
[ ./registryserver: Network: closing tcp connection
  local address = 141.*.*.118:10001
  remote address = 141.*.*.221:1571 ]

Than it hangs up!

By

my server does not catch this exception

i mean, that my application doesn't catch the exception although I have the try/catch stuff around pinging. Sorry for the misunderstanding.

Regards,
Jaleira

[marc]

What is the trace for the client at the time when the hang occurs?

Also, it's not clear to me that you don't get any exception. You should at least get a timeout exception.

For even more debug information, you could use Ice.Trace.Protocol=1 together with Ice.Trace.Network=3. With the given information, I really have no clue what the problem is. Again, please provide complete traces for both the client and the server.

Also, I still do not understand what exception you are seeing that the client doesn't catch? How do you see it?

[jaleira]

Good morning!

Here again the tracing protocols, this time with Ice.Trace.Protocol=1 and Ice.Trace.Network=3. First the protocol from the Linux-Server: At first you see a ping during normal application function. At last you see that I am sending a ping and simply nothing happens - the server hangs up. This is the case when shutting down the windows client.

---> sending ping
[ ./registryserver: Protocol: sending request
  message type = 0 (request)
  compression status = 0 (not compressed; do not compress response, if any)
  message size = 53
  request id = 14
  identity = Polaris-Servent
  facet =
  operation = ice_ping
  mode = 1 (nonmutating)
  context =  ]
[ ./registryserver: Network: sent 53 of 53 bytes via tcp
  local address = 141.*.*.118:53871
  remote address = 141.*.*.149:4417 ]
[ ./registryserver: Network: received 14 of 14 bytes via tcp
  local address = 141.*.*.118:53871
  remote address = 141.*.*.149:4417 ]
[ ./registryserver: Network: received 11 of 11 bytes via tcp
  local address = 141.*.*.118:53871
  remote address = 141.*.*.149:4417 ]
[ ./registryserver: Protocol: received reply
  message type = 2 (reply)
  compression status = 0 (not compressed; do not compress response, if any)
  message size = 25
  request id = 14
  reply status = 0 (ok) ]
received something on ping
[ ./registryserver: Network: shutting down tcp connection for writing
  local address = 141.*.*.118:10001
  remote address = 141.*.*.149:4414 ]
[ ./registryserver: Network: closing tcp connection
  local address = 141.*.*.118:10001
  remote address = 141.*.*.149:4414 ]
[ ./registryserver: Network: shutting down tcp connection for writing
  local address = 141.*.*.118:53871
  remote address = 141.*.*.149:4417 ]
[ ./registryserver: Network: closing tcp connection
  local address = 141.*.*.118:53871
  remote address = 141.*.*.149:4417 ]
---> sending ping
[ ./registryserver: Network: trying to establish tcp connection to 141.*.*.149:4417 ]

Protocol on Windows application side:

[ PolarisServent.exe: Protocol: received request
  message type = 0 (request)
  compression status = 0 (not compressed; do not compress response, if any)
  message size = 53
  request id = 14
  identity = Polaris-Servent
  facet =
  operation = ice_ping
  mode = 1 (nonmutating)
  context =  ]
[ PolarisServent.exe: Protocol: sending reply
  message type = 2 (reply)
  compression status = 0 (not compressed; do not compress response, if any)
  message size = 25
  request id = 14
  reply status = 0 (ok) ]
[ PolarisServent.exe: Network: sent 25 of 25 bytes via tcp
  local address = 141.*.*.149:4417
  remote address = 141.*.*.118:53871 ]

Refering to your question:

Also, I still do not understand what exception you are seeing that the client doesn't catch? How do you see it?

I saw it when running the server with Ice.Warn.Connections=1. The protocol shows a "ConnectionLostException" - i thought that this exception must be thrown and wondered why my application doesn't catch it(see again the source-code in my fourth message in this thread).

Awaiting your advices,
Jaleira

[benoit]

Hi,

Originally Posted by jaleira

---> sending ping
[ ./registryserver: Network: trying to establish tcp connection to 141.*.*.149:4417 ]

These traces are different from the ones above, here it's clear that the server is hanging trying to establish the connection to your Windows machine. Most likely, this is caused by the server not listening anymore on the TCP/IP port and your Windows machine ignoring the connection attempt because of the firewall. Did you try disabling the firewall to see if this makes a difference?

If you have just "unblocked" the client executable in the Windows firewall (or added it to the list of "exceptions"), when the executable isn't running the firewall will discard connection attempts on the port. You can easily try it out by trying to connect to the IP port number with telnet ("telnet 141.*.*.149 4417"): it will also hang. However, if instead you add a port to the exceptions list of your Windows firewall configuration, Windows will reject connection attempts on the port if no programs are listening on the port.

So if you need to use the Windows firewall, I would recommend to configure the firewall to allow the port number of your client -- this way the connection attempts will be rejected by Windows if your client isn't running anymore.

Originally Posted by jaleira

Refering to your question:

I saw it when running the server with Ice.Warn.Connections=1. The protocol shows a "ConnectionLostException" - i thought that this exception must be thrown and wondered why my application doesn't catch it(see again the source-code in my fourth message in this thread).

No, you won't necessarily get the Ice::ConnectionLostException. Whether or not this exception is propagated to the client depends on many things.

For example, the Ice runtime has a retry mechanism and if it's safe to retry an operation, Ice will automatically try to re-establish the connection (you can see when Ice retries or not by enabling retry tracing with --Ice.Trace.Retry=2).

This connection warning message might also be for a different connection than the connection from your server to your client: it could be for the connection from your client to the server for instance.

In any case, if you set a timeout on the proxy (or use Ice.Override.Timeout), the ping call which is hanging should eventually throw an Ice::TimeoutException.

Cheers,
Benoit.

[jaleira]

Originally Posted by benoit

In any case, if you set a timeout on the proxy (or use Ice.Override.Timeout), the ping call which is hanging should eventually throw an Ice::TimeoutException.

What do you mean with "eventually throw"? Shouldn't it throw something?

Shutting down the firewall, alternatively opening the port works, but an exception thrown would be the best solution! What if somebody cut the cable and this is why the connection is lost? The ping call will never throw an exception... Now i only cover the situation that the clilent shuts down and is behind a firewall.

Regards,
Jaleira

[benoit]

Hi,

Sorry, I meant that it should definitely throw after the timeout expires. If you set a timeout and your server doesn't throw an exception after the timeout expires then something is wrong. How do you set the timeout?

I just tried with the hello world demo (located in the demo/Ice/hello directory of your Ice distribution) in a similar configuration (hello client running on a Mac OS X machine and the hello server running on Windows XP with the Windows firewall enabled and the server "unblocked") and it works as expected for me. If I shutdown the server and don't use the timeout in the client, the client hangs trying to re-establish the connection. If use the timeout (the "T" option in the hello client), I get an Ice::ConnectionRefusedException after the timeout expires.

Could your perhaps try the demo to see if you can reproduce the issue you're getting with your client and server? You should enable the network and retry tracing on the client to see what Ice is doing (--Ice.Trace.Network=2 --Ice.Trace.Retry=2).

Cheers,
Benoit.