Bug in ThreadPool in Ice 3.2.1

[dhogan]

A java servant in an IceBox instance in our IceGrid deployment encountered a java.lang.OutOfMemoryError. Thereafter, our services were not available because an assert in IceInternal.ThreadPool kept popping. Here is the assert:

Exception in thread "HaloCentralGrid-Services-01-DatabaseServices-Ice.ThreadPool.Server-106" java.lang.AssertionError
at IceInternal.ThreadPool.run(ThreadPool.java:820)
at IceInternal.ThreadPool.access$100(ThreadPool.java: 12)
at IceInternal.ThreadPool$EventHandlerThread.run(Thre adPool.java:1242)

It appears that this assertion pops when the ThreadPool believes that the number of running Threads exceeds the total number of threads hosted by the ThreadPool instance. I believe that the root-cause is in the run method of the EventHandlerThread class: it appears that when it runs a servant invocation, it catches only java.lang.Exception and Ice.LocalException - it does NOT catch java.lang.Error, which is the superclass of a java.lang.OutOfMemoryError. It needs a catch block for java.lang.Error, or any java.lang.Error subclass encountered in the servant invocation will render the entire ThreadPool useless thereafter.

Here is the trace for the OutOfMemoryError:

Exception in thread "HaloCentralGrid-Services-01-DatabaseServices-Ice.ThreadPool.Server-82" java.lang.OutOfMemoryError: Java heap space
at com.microsoft.sqlserver.jdbc.DTV.setFromTDS(Unknow n Source)
at com.microsoft.sqlserver.jdbc.Column.setFromTDS(Unk nown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.in dexRow(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.co untRowsInFetchBuffer(Unknown Source)
at com.microsoft.sqlserver.jdbc.ClientFetchBuffer.get RowCount(Unknown Source)
at com.microsoft.sqlserver.jdbc.ClientFetchBuffer.rel ative(Unknown Source)
at com.microsoft.sqlserver.jdbc.PositionedCursor.posi tionedRelative(Unknown Source)
at com.microsoft.sqlserver.jdbc.PositionedCursor.next (Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet.ne xt(Unknown Source)
at com.mchange.v2.c3p0.impl.NewProxyResultSet.next(Ne wProxyResultSet.java:2859)
at org.hibernate.loader.Loader.doQuery(Loader.java:68 5)
at org.hibernate.loader.Loader.doQueryAndInitializeNo nLazyCollections(Loader.java:224)
at org.hibernate.loader.Loader.loadCollection(Loader. java:1919)
at org.hibernate.loader.collection.CollectionLoader.i nitialize(CollectionLoader.java:36)
at org.hibernate.persister.collection.AbstractCollect ionPersister.initialize(AbstractCollectionPersiste r.java:520)
at org.hibernate.event.def.DefaultInitializeCollectio nEventListener.onInitializeCollection(DefaultIniti alizeCollectionEventListener.java:60)
at org.hibernate.impl.SessionImpl.initializeCollectio n(SessionImpl.java:1676)
at org.hibernate.collection.AbstractPersistentCollect ion.initialize(AbstractPersistentCollection.java:3 44)
at org.hibernate.collection.AbstractPersistentCollect ion.read(AbstractPersistentCollection.java:86)
at org.hibernate.collection.PersistentSet.iterator(Pe rsistentSet.java:138)
at com.hp.halo.databaseservice.dtmUtils.DtmUtils.DTMM akeAudioListFromDbAudioListCapability(DtmUtils.jav a:564)
at com.hp.halo.databaseservice.dtmUtils.DtmUtils.DTMM akeNodeBaseFromDbNode(DtmUtils.java:393)
at com.hp.halo.databaseservice.node.Node.getAllNodes( Node.java:1689)
at com.hp.halo.databaseservice.DatabaseServicesImpl.g etAllNodes(DatabaseServicesImpl.java:1366)
at com.hp.halo.HaloICE._DatabaseServicesDisp.___getAl lNodes(Unknown Source)
at com.hp.halo.HaloICE._DatabaseServicesDisp.__dispat ch(Unknown Source)
at IceInternal.Incoming.invoke(Incoming.java:147)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2249)
at Ice.ConnectionI.message(ConnectionI.java:1362)
at IceInternal.ThreadPool.run(ThreadPool.java:782)
at IceInternal.ThreadPool.access$100(ThreadPool.java: 12)
at IceInternal.ThreadPool$EventHandlerThread.run(Thre adPool.java:1242)


Dirk

[mes]

Thanks Dirk, we'll take a look at this.

Regards,
Mark