indirect proxy and ObjectNotExistException

[rc_hz]

I have noticed that when the Ice core in the client gets an ConnectionRefusedException, it will contact IceGrid again to refresh the locator cache and then use these new endpoints to make another try. However, when it gets an ObjectNotExistException, it will simply throw the exception. I think the latter will be misleading in some cases and sometimes will never recover. For example:

1.Suppose the OA1server's ObjectAdapter selects tcp port randomly:
OA1.Endpoints=tcp -h 10.10.10.1

Suppose OA1 listens on -p 10001 now.
OA1 also register dynamically to IceGrid.

2.Suppose the client is using indirect proxy(myobj@OA1).  The client make an invocation and it gets an item from IceGrid:obj@OA1 --> obj@tcp -h 10.10.10.1 -p 10001. Of course, the client's invocation succeeds.

3.Now the following events:
1)time1: The OA1's server restarts and listens on a new port: -p 10002
2)time2: Another server starts and it listens on -p 10001
3)time3: The previous client makes an invocation on myobj@OA1 and it will get an ObjectNotException.

[rc_hz]

Another question about random ports. Suppose we have the following configuration:
1.IceGrid

IceGrid.Registry.Client.Endpoints=tcp -p 10000

2.Glacier2

Glacier2.Client.Endpoints=tcp -p 20000

3.non-Ice applications

TcpListenPort=30000

4.Normal Ice applications

OA1.Endpoints=tcp -h 10.10.10.1
OA2.Endpoints=tcp -h 10.10.10.1

It means that OA1 and OA2 will select tcp port randomly. However, if we start OA1/OA2's server first, it will has the danger of using the specified ports: 10000/20000/30000.

So it would be better if such an Ice property can be added:

Ice.TcpPortExclude=10000,20000,20050-20099,30000

[benoit]

I have noticed that when the Ice core in the client gets an ConnectionRefusedException, it will contact IceGrid again to refresh the locator cache and then use these new endpoints to make another try. However, when it gets an ObjectNotExistException, it will simply throw the exception. I think the latter will be misleading in some cases and sometimes will never recover.

The Ice::ObjectNotExistException is actually not thrown right away, the invocation is first retried by the Ice core (try again your scenario with enabling Ice.Trace.Retry). However, since the endpoints for the object adapter are still cached in the Ice core, the invocation will be tried again with the same server. You should disable the locator cache if you want this scenario to work.

Invalidating the object adapter endpoints when an invocation fails with ObjectNotExistException could be another solution. However, this means that in the most common cases we might be invalidating the endpoints too often defeating the usefulness of the cache so I'm not sure it's a good idea.

Cheers,
Benoit.

[benoit]

As for your second comment, most operating systems are using a specific range for "ephemeral ports" (ports allocated by the system, see here for more information) and sometime even allow you to configure this range. I think the best is to not use ports in the ephemeral port range to avoid any conflicts.

Cheers,
Benoit.

[rc_hz]

The Ice::ObjectNotExistException is actually not thrown right away, the invocation is first retried by the Ice core (try again your scenario with enabling Ice.Trace.Retry). However, since the endpoints for the object adapter are still cached in the Ice core, the invocation will be tried again with the same server. You should disable the locator cache if you want this scenario to work.

Invalidating the object adapter endpoints when an invocation fails with ObjectNotExistException could be another solution. However, this means that in the most common cases we might be invalidating the endpoints too often defeating the usefulness of the cache so I'm not sure it's a good idea.

Cheers,
Benoit.

Thank you. I mean: when the client's Ice core gets an ConnectionRefusedException, it will contact IceGrid again, so I think the client's Ice core can also contact IceGrid again if it gets an ObjectNotExistException.

As for your second comment, most operating systems are using a specific range for "ephemeral ports" (ports allocated by the system, see here (http://www.ncftp.com/ncftpd/doc/misc...ral_ports.html) for more information) and sometime even allow you to configure this range. I think the best is to not use ports in the ephemeral port range to avoid any conflicts.

This seems to work. Thanks.

[benoit]

Thank you. I mean: when the client's Ice core gets an ConnectionRefusedException, it will contact IceGrid again, so I think the client's Ice core can also contact IceGrid again if it gets an ObjectNotExistException.

Yes, I understand what you mean. But note that IceGrid will contact the locator again if the invocation fails with an ObjectNotExistException AND if you disable the locator cache.

The scenario you describe might happen when using the locator cache however. We could invalidate the adapter endpoints from the cache on ObjectNotExistException but it might have a performance impact on clients which are often expecting ObjectNotExistException from a server.

Also, the scenario you describe is unlikely to happen if your servers are using ephemeral ports (assigned by the system). I believe most operating systems iterate over the ephemeral port range to assign ports for sockets, so to re-use a previously used ephemeral port you first have to use all the ports available in the ephemeral port range...

Cheers,
Benoit.

[rc_hz]

I understand now. Thank you very much.