Request ordering issue

[lcolborn]

We have an issue where a server under heavy load starves some connections. The ice ThreadPool does not seem to process requests in the order they were recieved.

To illustrate the issue, I have a single threaded server and a simple client. To reproduce the problem, run the server and 5+ instances of the client. You'll notice that some clients get starved of processing threads for quite a while. The connection is 'accepted' but is not alloted a thread till *all* requests, even if the requests came in later, by other connections is complete. This is reproducible with a multi-threaded server too, but needs more client instances running. We are using Ice 3.2.1 and Java.

Ice definition:
module test
{
class TestService
{
void helloworld(string name);
};
};

Server:

package ice;

import test.TestService;
import Ice.Current;
import Ice.Identity;
import Ice.InitializationData;
import Ice.Properties;


public class TestIce extends TestService {
static final int MINUTE = 1000*60;
public TestIce() {
initIce(this);
}

@Override
public void helloworld(String name, Current __current) {
System.out.println(__current.con);
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
}
}

private void initIce(TestIce facet) {
try {
Properties properties = Ice.Util.createProperties();
properties.setProperty("Ice.ThreadPool.Server.Size ", "3");
properties.setProperty("Ice.ProgramName", "Ice.Test");
properties.setProperty("Ice.Trace.Network", "2");

InitializationData initData = new InitializationData();
initData.properties = properties;
String id = "test";

Ice.Communicator communicator = Ice.Util.initialize(new String[] {}, initData);
Ice.ObjectAdapter adapter = communicator.createObjectAdapterWithEndpoints(id, "tcp -p " + 4000);
Identity identity = Ice.Util.stringToIdentity(id);
adapter.addFacet(facet, identity, "test");
adapter.activate();
} catch (Exception e) {
e.printStackTrace();
}
}

public static void main (String [] args) {
new TestIce();
}
}


The test client is :
package ice;

import test.TestServicePrx;
import test.TestServicePrxHelper;
import Ice.InitializationData;

public class TestIceClient {
TestServicePrx proxy;
public TestIceClient(TestServicePrx proxy) {
this.proxy = proxy;
}

public static void main( String [] args) throws InterruptedException {
TestServicePrx proxy = initServerProxy(initCommunicator(), "test: tcp -h localhost -p 4000");
new TestIceClient(proxy).run();
}

public void run() {
for (int i = 0; i < 100; i++) {
System.out.println("start [" + i + "]" );
proxy.helloworld(i + "");
}
}

private static Ice.Communicator initCommunicator() {

Ice.Properties iceProperties = Ice.Util.createProperties();
iceProperties.setProperty("Ice.ThreadPool.Server.S ize", "2");
iceProperties.setProperty("TestClient.Endpoints", "default -p 0");
iceProperties.setProperty("Ice.Trace.Network", "2");

InitializationData initData = new InitializationData();
initData.properties = iceProperties;

return Ice.Util.initialize(new String[0], initData);
}

private static TestServicePrx initServerProxy(Ice.Communicator _communicator, String connection) {
Ice.ObjectPrx base = _communicator.stringToProxy(connection);
return TestServicePrxHelper.checkedCast(base, "test");
}
}

[benoit]

Hi Lauren,

Note that we only provide free support on the forums for the latest Ice version. Your sample code works fine for me with 3.3.0. There were significant changes in 3.3.0 in this area so could you try with Ice 3.3.0 instead?

Cheers,
Benoit.

[benoit]

The ice ThreadPool does not seem to process requests in the order they were recieved.

Btw, note that there are in general no guarantees that requests will be processed in the order they were received when using a thread pool with multiple threads. See the "Serialization" section in the Ice manual here for more information.

Cheers,
Benoit.

[lcolborn]

Thanks benoit. We'll look into upgrading in the next couple of months.
We worked around this temporarily by increasing the threadpool size to be > the number of connections and implementing a semaphore on the object adapter side. And yes, we have been careful to stay within the ulimits :-)

cheers