How is Ice threading impacted by UdpClient?

[jharriot]

Hi,

My configuration: Ice 3.3.1, Windows XP.

Scenario 1:
I have an application that contains multiple threads (i.e. .NET Thread class) making periodic calls on proxies to servants located on a remote machine. Each of the proxy method calls were timed using .NET Stopwatch, and reported expected timing results (<20ms).

Scenario 2:
The application was then configured to activate some code that listens for UDP packets asynchronously using UdpClient.BeginReceive(). When a message is received the callback handler ultimately calls a method on the proxy. N.B. the thread described in scenario 1 is also still periodically calling the proxy as well. However, for this scenario the proxy methods were taking an enternity to complete, e.g. (~1000ms).
FYI. I played around with using thread pools in the Ice config file, but this made no difference.

After much head scratching I modified the code to use a worker thread to process the UDP messages. This change resulted in a normal and responsive behaviour.

So here is my question: What is going on under the covers that impacts Ice network performance when concurrently using UdpClient? On the surface it appears to be some kind of thread starvation down at the low level.


Cheers John

[mes]

Hi John,

Asynchronous I/O callbacks for methods such as UdpClient.BeginReceive are dispatched by .NET's built-in thread pool (this thread pool is unrelated to Ice's thread pools). Ice for .NET also uses asynchronous I/O. If you make a synchronous Ice invocation from within an async I/O callback, you are essentially tying up a .NET thread pool thread until the Ice invocation completes.

To add more complexity, if there are no idle threads available in the .NET thread pool, the reply for the synchronous Ice invocation cannot be dispatched. In effect, this creates a temporary deadlock situation, at least until .NET adds another thread to its pool. .NET does this automatically, but only after a delay. Currently I believe this delay is 500ms.

I'm pretty sure this is what's happening in your situation. You may be able to confirm this by closely examining the timestamps of Ice's protocol and network trace messages for both the client and the server. Specifically, if the server reports that it has sent the reply for the Ice invocation, but the client does not report that it has received the reply until later, then that is a good indication that thread starvation is occurring, as you suspected. However, adding more threads to the Ice thread pools will not make any difference in this case.

Here are some possible work arounds:

• Increase the minimum number of idle threads in the .NET thread pool. You can use ThreadPool.GetMinThreads and ThreadPool.SetMinThreads to discover and modify the number of idle threads, respectively.
• Use asynchronous Ice invocations in your async I/O callback. Doing so guarantees that you won't block the async I/O thread.
• Perform synchronous Ice invocations from a different thread, which is the solution you discovered.

Hope that helps.
Mark