[3.3.1] NoSuchMethodError hangs the client

[joshmoore]

In our session facade, a NoSuchMethodError was thrown, which was printed to master.err:

Exception in thread "Blitz-0-Ice.ThreadPool.Server-5" java.lang.NoSuchMethodError: ome.services.sessions.events.ChangeSecurityContextEvent.<init>(Ljava/lang/Object;Lom
e/model/IObject;Lome/model/IObject;)V
        at ome.services.sessions.SessionManagerImpl.setSecurityContext(SessionManagerImpl.java:1088)
        at ome.services.blitz.impl.ServiceFactoryI.setSecurityContext(ServiceFactoryI.java:313)
        at omero.api._ServiceFactoryDisp.___setSecurityContext(_ServiceFactoryDisp.java:385)
        at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1347)
        at IceInternal.Incoming.invoke(Incoming.java:159)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
        at Ice.ConnectionI.message(ConnectionI.java:972)
        at IceInternal.ThreadPool.run(ThreadPool.java:577)
        at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)

but left the client blocking, forcing a "kill -9". Is this a known issue, and if so, is it fixed in 3.4?

~Josh.

[mes]

Hi Josh,

That's the expected behavior. Ice considers subclasses of java.lang.Error to be unrecoverable and therefore does not trap them. These errors generally indicate that something is seriously wrong with the application, as appears to be true in your example.

In this situation, you can expect a client that makes a twoway synchronous invocation (without a timeout) to hang indefinitely because the server's thread was terminated by the error.

Regards,
Mark

[joshmoore]

Hi Mark,

Thanks for the response. In the case of this particular NoSuchMethodError, the error was the type of exception that should essentially never occur (a build error). We had, however, run into similar hangs with 3.3.0 caused by java.lang.OutOfMemoryErrors (unfortunately more common in our environment). My feeling was that these had gone away with 3.3.1, and so most of all I wanted to point out possibly another code path which could/should catch Throwable.

More generally, though, the implication of terminating threads regardless of Error-type thrown is that every application method must be surrounded by a try/catch(Throwable) block. This seems fairly invasive, but if that's the policy we'll code to handle it. Though I'm probably just overlooking something, it'd be nice if this were clear in the Java section. Javas.13.6.html details the propagation of runtime exceptions, but doesn't point out the dangers of java.lang.Errors.

Cheers,
~Josh

P.S. an alternative would be registering a handler somewhat similar to the one mentioned in this thread: registering-objectfactory-exceptions.html

[mes]

Actually there are two errors that Ice will trap if raised by a servant: AssertionError and OutOfMemoryError. An occurrence of either of these will gracefully terminate the request and return Ice.UnknownException to the client. All others are allowed to propagate to the JVM.

Even if you modify every servant method to add try/catch blocks in an attempt to trap errors, you still won't catch every problem. For example, a build error similar to yours could raise NoSuchMethodError during request dispatching before the servant's code is ever entered.

Here is what Sun's documentation has to say about java.lang.Error:

An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch.

Regards,
Mark

[joshmoore]

Actually there are two errors that Ice will trap if raised by a servant: AssertionError and OutOfMemoryError. An occurrence of either of these will gracefully terminate the request and return Ice.UnknownException to the client.

Ah, that's great to hear. Thanks.

Even if you modify every servant method to add try/catch blocks in an attempt to trap errors, you still won't catch every problem. For example, a build error similar to yours could raise NoSuchMethodError during request dispatching before the servant's code is ever entered.

Of course. Understood. This is also the reason why having as much handling as possible inside of Ice is critical.

An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch.

That assumes 1) that our application is reasonable and 2) that that documentation is correct. In fact, as seen with the case of assertions and OOMs, it's not always the case.

Thanks again for the explanation.
Cheers,
~Josh.

[mes]

We catch OutOfMemoryError with the hope that a subsequent retry has a (slim) chance of actually succeeding. On the other hand, catching AssertionError is not an attempt to make the server more robust but simply a convenience for developers (including ourselves ) to help track down an assertion failure more quickly, since a retry of the same request will almost certainly trigger the same assertion again.

Essentially, our reasoning is "Can Ice do anything productive by trapping errors?" And, by extension, "How likely is it that a broken application will continue to function normally if Ice does intervene?" In general, we think it's better for the program to fail immediately.

Regards,
Mark

[joshmoore]

We catch OutOfMemoryError with the hope that a subsequent retry has a (slim) chance of actually succeeding.

Understood. There's, of course, a trade-off. If Ice were to try to do too much during an OOM, then it could possibly make the situation much worse. In our case, we have fairly good chances of surviving the single exception.

On the other hand, catching AssertionError is not an attempt to make the server more robust but simply a convenience for developers (including ourselves ) to help track down an assertion failure more quickly, since a retry of the same request will almost certainly trigger the same assertion again.

Very much appreciated. I guess I was expecting exceptions which also fall into the coding/deployment category like (most) StackOverflowErrors and the various LinkageErrors to be treated in roughly that manner.

Looking at the Error hierarchy on my CLASSPATH, there are some suprises: org.apache.commons.httpclient.HttpClientError is one example. Various XML configurators another. These are possibly poor library choices that consumers have to be aware of, but as much handling by Ice as possible would be appreciated.

Essentially, our reasoning is "Can Ice do anything productive by trapping errors?" And, by extension, "How likely is it that a broken application will continue to function normally if Ice does intervene?" In general, we think it's better for the program to fail immediately.

I agree and would even be fine with initiating a full server stop or similar on such exceptions. Freeing up client resources, though, would seem to be paramount before doing so.

Best wishes,
~Josh.