Ice & Mono - exepath and configuration

[JensAernouts]

Hello

First, I will try to explain the current situation:
I have one computer with Windows 7, which runs my ICE Client, IceGridRegistry, IceGridAdmin, etc.
I have one computer with windows XP, which runs one of my ICE servers, what's working without any problem
I have one computer with ProxMox installed and a Virtual Linux (Cent OS) running on it. Here, I use mono to run my built exe's. The node I run here isn't working like it should...
(!: To make it possible to run the exe I built in Windows, on linux using Mono, I wrote a script 'mono my.exe "$1" '. This start the exe with mono, using the first argument when starting the script; The 'exepath' of my node is thus 'linux.script'... I don't know if this is completely right...)

The problem is that I can't contact the node, when I use 'checkedcast', it just hangs and keeps hanging... I don't get any error, it doesn't crash, it just hangs...
To check what was wrong, I tested some command via IceGridAdmin and I found out that 'server properties xxxxx' doesn't work either; it gives the same problem as the checkedcast command. First, I used Wireshark to check whether it was a network problem. But that wasn't the problem; On the other hand, what I saw was that the reply on 'Properties.getPropertiesForPrefix' was empty... I get an 'ACK', but the linux node doesn't send anything back...
Next, I set Ice.Trace.Network, Ice.Trace.Protocol and Ice.Trace.ThreadPool to 'visible' (well, I used the right numbers) in both the icegridregistry and the icegridnode. This is were I get stuck now... The icegridnode just doesn't reply...
When I compare both logs, I saw that on the windows client, the ThreadPool size grows when 'getPropertiesForPrefix' was raised; this doesn't happen on the linux node...
I added the screenshots from my CMD windows, maybe this helps to make the problem more clear.

I hope you understand my problem and that you can help me!

Kind Regards

Jens Aernouts

[bernard]

Hi Jens,

While you can communicate directly with the IceGrid registry (for example with icegridadmin or the IceGrid::Admin interface), you never interact directly with an IceGrid node. There is no public API for an IceGrid node, and icegridadmin actually never opens any direct connection to a node (only to the registry).

The command 'server properties serverID' gets the runtime properties of a server by sending a request to the IceGrid registry, which forwards this request to its IceGrid node, which in turns forwards the request to the server. (The response takes the reverse path). In your situation, the node looks fine, but it's unclear why your mono server does not return the properties or responds to other requests such as checkedCast/ice_isA (presumably on an object hosted by this server).

The first think I would look into is the activation of this mono server. Can you start the server with 'server start serverID'? Does 'server state serverID' return active? See also http://www.zeroc.com/doc/Ice-3.4.1-I...rid.39.26.html

I would also suggest to get rid of your script, and simply use 'mono' to start the server. The README of the C# IceGrid demo describes how to write your IceGrid server descriptor.

Best regards,
Bernard

[JensAernouts]

I got rid of the script I was using, but this doesn't solve the problem...
When I check the state, it says: 'active (pid = 548, enabled)', so that's kind a weird... Another possible solution?

Kind Regards

Jens Aernouts

EDIT: I used the '<option>exe</option>' way to get rid of the script, it wasn't possible to use the binfmt module on Cent OS, so I couldn't change my kernel so that the binaries are automatically started with mono...

EDIT²: When I check the running processes on my Linux computer, I see that the user of my server.exe is 'nobody'... Can this be a problem? I don't think so, but I'm not sure...

[bernard]

Your mono server is expected to run as nobody if the IceGrid node is running as root. It's probably not an issue, provided the server can read its config file and any other file it may need.

The next step is to figure out if a client can communicate with this server.

Presumably your client has an indirect proxy to an object in this server. What happens if the client calls ice_ping on this proxy? If this does not work, you may want to enable network tracing on both the client and server to see what's happening. You should also double-check no firewall is running on your Linux virtual machine.

Best regards,
Bernard

[JensAernouts]

The firewall is off... the problem is that I even can't call 'server properties xxxxx'... When I use my client, it hangs on '.....checkedcast(obj)'...
What should I see on the network trace? That trace is on in the screenshots I provided in the first post, maybe you can have a look at that if something is wrong?

Kind regards

[bernard]

'server properties serverID' involves many processes and is therefore more difficult to troubleshoot than a plain ping or checkedCast. (checkedCast is a wrapper for the ice_isA operation).

So let's focus on checkedCast. What is obj? Can you post the network traces for the client? For the mono server?

The network traces for the IceGrid registry and IceGrid node are not that interesting, since you're trying to figure out why your client can't talk to your server.

Best regards,
Bernard

[JensAernouts]

This is the code I use: It just searches for all active replica's...

{
ObjectPrx proxyC = this.distributingCommunicator.ic.stringToProxy("Ic eGrid/Query");
QueryPrx queryC = QueryPrxHelper.checkedCast(proxyC);
string type = CalculatorDisp_.ice_staticId();
ObjectPrx[] objType = queryC.findAllObjectsByType(type);
foreach (ObjectPrx objectPrx in objType)
{
ObjectPrx[] allReplicas = queryC.findAllReplicas(objectPrx);
foreach (ObjectPrx replica in allReplicas)
{
try
{
this.calculatorPrx = CalculatorPrxHelper.checkedCast(replica);
this.activeReplicas.Add(new ActiveReplica(replica));
}
catch (Exception ex)
{
}
}
}
}

Screenshots of both the node and the client with network trace on are provided... I can't see anything on it... I don't think anything responds on the client, but I'm not sure

[JensAernouts]

Since I can't edit my post above, I'll reply with a new one:

When I use 'replica.ice_ping()' in the code above, it gives the same problem as the checkedcast... it just hangs...

Kind Regards

[benoit]

Hi,

Is the client the OSLServer.exe process? Instead of using --Ice.Trace.Network=3, can you use --Ice.Trace.Network=2 and --Ice.Trace.Protocol=2?

Can you also enable this tracing on the server which is supposed to receive the client's requests and see if the client correctly establishes the connection to the server and receive the requests?

Cheers,
Benoit.

[JensAernouts]

New screenshots:

EDIT: Indeed, the OSLServer is my Client process. OSLServerIce is my server process.

[bernard]

Hi Jens,

Unfortunately these screenshots do not show any trace from your Mono server. Did you enable network tracing in this server? Can you post the IceGrid descriptor you use for this server?

I would also recommend to redirect the trace output of your client and server to files, and then post attached these files ... text files are much easier to read and search.

C:\Users> OSLServer.exe 2>trace.txt

For the server started by IceGrid, see IceGrid.Node.Output at http://www.zeroc.com/doc/Ice-3.4.1-I...Ref.50.15.html

Best regards,
Bernard

[bernard]

It would also be helpful to provide a protocol trace that includes the ice_isA call from the client to the server ... since we're trying to find out why this call hangs.

checkedCast() is a wrapper for this ice_isA operation.

Best regards,
Bernard

[JensAernouts]

Yeah, indeed like you said I forgot to output the Network Trace at the server (I was only doing the node network trace...). The attachments are both the Client and the Server's log file. I putted "bool test = replica.ice_isA(type);" as line before the checkedCast, and now it keeps on hanging on that line.
I hope you can help me!

Kind Regards

[bernard]

Ok, so the server is listening on port 10000:

-- 3/15/2011 09:01:56:638 CalculatorServer090155217: Network: attempting to bind to tcp socket 192.168.20.206:10000
-- 3/15/2011 09:01:56:638 CalculatorServer090155217: Network: accepting tcp connections at 192.168.20.206:10000
-- 3/15/2011 09:01:56:638 CalculatorServer090155217: Network: published endpoints for object adapter `CalculatorAdapter':
   tcp -h 192.168.20.206 -p 10000

and the client establishes a connection to this server (as expected):

-- 15/03/2011 09:03:10:234 OSLServer.exe: Network: trying to establish tcp connection to 192.168.20.206:10000

-- 15/03/2011 09:03:10:239 OSLServer.exe: Network: tcp connection established
   local address = 192.168.20.212:3461
   remote address = 192.168.20.206:10000

and then waits for the "validate connection" message from the server ... and waits and waits.

You don't get any trace for the server around the time this connection is established (09:03)?

Did you try to use an OS-assigned port for your server, in case there is some conflict with port 10000? (To do so, just don't specify -p 10000 in your server XML configuration).

Can you run the client on the same host as the server to see if it makes a difference?

Cheers,
Bernard

[mes]

Jens,

One other thing to check: please make sure that you've called activate on the object adapter in your Mono server. If you neglect to call activate, your server will appear to be available but won't actually respond to requests because the adapter is still in its initial "holding" state.

Regards,
Mark