Bidir Sample deadlock

[vincei]

Have a very simple use case that exhibits a deadlock in the bidir sample ice application. This occurs on WindowsXP32 using VS2005 no service pack. Hardware is a Sun Ultra20 box.

In the server, bidirS, change the line in CallbackI.cpp from:

timedWait(IceUtil::Time::seconds(2));

to

timedWait(IceUtil::Time::milliSeconds(20));

Rebuild and run the server. As I start to add clients the server will eventually deadlock and all processing ceases. What am I doing wrong?

-- Vince

[vincei]

I tried

Ice.ThreadPool.Client.Size=10
Ice.ThreadPool.Client.SizeMax=20

in config.server but this made no difference. The deadlock occurs sometime on the second client connection.

-- Vince

[vincei]

Ice3.2.0 fyi.

[benoit]

Hi,

Yes, I'm able to reproduce this. Try increasing the server thread pool size instead, with the following property:

Ice.ThreadPool.Server.Size=10

In the server, the response for invocations on the bidirectional connection is received by the server thread pool (not by the client thread pool). If all the server thread pool threads are busy dispatching calls, the response can't be received and therefore the invocation on the bidirectional connection hangs until its response can be received (or until it times out).

This is what is happening here. There's only one thread by default in the server thread pool. When the deadlock occurs, the server thread is dispatching the CallbackSenderI::addClient() method and it blocks waiting to acquire the mutex. However, the mutex can't be acquired because it's held by the sender thread in the run() method which is itself blocked waiting for the response of the callbacak() invocation...

Increasing the number of threads in the server thread pool will fix the issue. A better solution would be to modify the CallbackSenderI::run() method implementation to not lock the mutex while invoking on the callback receiver proxy.

Thanks for pointing this out, we'll look into changing this demo for the upcoming releases to fix this deadlock!

Cheers,
Benoit.

[vincei]

Hi Benoit,

That worked. I had a feeling it might be due to a deadlock while invoking
while owning the lock. I'll restructure my app to not do the same. The
thread pool thing is good to know.

Thanks!

-- Vince

[benoit]

Btw, out of curiosity, why do you need bidirectional connections?

If this for firewall traversal, you should consider using Glacier2 instead. Your servers won't have to deal with the specificities of bidirectional connections and will be better protected from potentially misbehaving clients.

Cheers,
Benoit.

[vincei]

>> Btw, out of curiosity, why do you need bidirectional connections?

>>If this for firewall traversal, you should consider using Glacier2 instead.
>>Your servers won't have to deal with the specificities of bidirectional
>>connections and will be better protected from potentially misbehaving
>>clients.

Hi Benoit,

No this is not for firewall traversal at this time. At this stage I'm working on
a proof of concept and did not want to introduce another service into the
mix. At some point in the near future I will take a detailed look into Glacier2
with the understanding that I will have to refactor some of the code.

In the meantime I'll take a quick look over the Glacier2 documentation so that
I'm aware of the potential issues going forward.

Thanks, Vince

[marc]

So if firewall traversal is not an issue, why use bi-directional connections? Why not simply use regular callbacks?

[vincei]

Hi Marc,

The main reasons is I want the server to be able to make callbacks
on the client _without_ creating a server instance on the client side.
My understanding is that bidir is the only way to do this without using
Glacier2.

I'll be the first to admit that the need to use a single socket is a
philosophical thing because my underlying transport is full duplex.
That's not to say that I can't be convinced otherwise

-- Vince

[marc]

If your Ice client receives callbacks, regardless whether over bi-directional connections, regular connections, or using Glacier2, your Ice client will also be an Ice server, and you have to create an object adapter in order to receive callbacks. From an Ice point of view, a client sends requests, and a server receives requests. Most non-trivial applications are both an Ice server and client at the same time. Have a look at "The Ice Architecture" in the Ice manual for more details.