Ice with C# not multi-threading properly

[amrufon]

First off, I would like to apologize to the ZeroC for giving you guys a bad rep in my small circle of techie friends.

Ok, this what happened, I had a friend who's working on a "Smart Client" project who's having some trouble with his data transfer. Basically, it times-out when he has more than 3 users. So I suggested that he uses Ice, we were discussing this and another one joins the discussion and her solution is to use .NET's WebClient object.

There was some ego thrown and some insults sneaked in so we ended up getting a copy of the errant project and should show a result in 2 hours!

So I took the c# code that I currently have for my JDB project and modified it. The object is to send a byte array streamed object and send it to the server where it would reply depending on the operation embedded on the streamed object.

SLICE Definition:

module SmartNetwork {
	sequence<byte> ByteSeq;
	interface Crawler {
                ByteSeq UploadData(ByteSeq data, string compression);
	};
};

SERVER Code:

namespace SmartEOE.Network.Server
{
    class crawlerI : SmartNetwork.CrawlerDisp_
    {
        public override byte[] UploadData(byte[] data, string compression, Ice.Current current__)
        {
            Console.WriteLine(current__.id.name);
            WebDataAccess access = new WebDataAccess(data, compression);
            return access.GetRequest();
        }
    }

    public class Server
    {
        public static void Main(string[] args)
        {
            int status = 0;
            Ice.Communicator ic = null;

            try {
                ic = Ice.Util.initialize(ref args);
                Ice.ObjectAdapter adapter
                    = ic.createObjectAdapterWithEndpoints(
                        "CrawlerAdapter", "default -p 7015 -t 2880000 -z");
                Ice.Object obj = new crawlerI();
                adapter.add(
                    obj,
                    Ice.Util.stringToIdentity("WebCrawler"));
                adapter.activate();
                ic.waitForShutdown();
            } catch (Exception e) {
                Console.Error.WriteLine(e);
                status = 1;
            }

            if (ic != null) {
                try {
                    ic.destroy();
                } catch (Exception e) {
                    Console.Error.WriteLine(e);
                    status = 1;
                }
            }
            Environment.Exit(status);
        }
    }
}

CLIENT Code:

namespace SmartEOE.Network
{
    public class Client
    {
        CrawlerPrx webCrawler;
        Ice.Communicator ic = null;
        public Client(string server)
        {
            string[] args = new string[1];
            args[0] = "--Ice.Config=" + Application.StartupPath + "\\SmartEOE.CFG";

            ic = Ice.Util.initialize(ref args);
            Ice.ObjectPrx obj = ic.stringToProxy(
                "WebCrawler:default -h " + server + " -p 7015 -t 2880000");
            webCrawler = CrawlerPrxHelper.checkedCast(obj);
        }

        ~Client()
        {
            if (ic != null)
                ic.destroy();
        }

        public byte[] UploadData(byte[] crawler, string compression)
        {
            byte[] returnData;
            returnData = webCrawler.UploadData(crawler,compression);
            return returnData;
        }

    }

SmartEOE.CFG

// This property controls the maximum size (in kilobytes) of a protocol message that
// will be accepted or sent by the Ice run time. The size includes the size of the Ice
// protocol header. Messages larger than this size cause a [MemoryLimitException].
// The default size is 1024 (1 Megabyte). Settings with a value less than 1 are
// ignored.
// This property adjusts the value of Ice.UDP.RcvSize and Ice.UDP.SndSize,
// that is, if Ice.UDP.RcvSize or Ice.UDP.SndSize are larger than Ice.Message-
// SizeMax * 1024 + 28, they are adjusted to Ice.MessageSizeMax * 1024 + 28.
Ice.MessageSizeMax=20480

So after 2 hours, the results are underwhelming. If you would view the attached JPEG file, you would see that Rea's code, which uses the WebClient object, is a little bit faster. Another thing to point out is that supposedly, the UploadData() function is given its own thread per instance ... but it seems that the method is queuing ...

Hence, Bacz will be using the WebClient object over ICE on this one. Although it is a moot question for my friends project, my project JDatabase is using the same structure ... so I am a bit concerned about this.

What did I do wrong or what should I change to make this work faster when I have more users accessing the server?

[amrufon]

Um. I thought about it some more ... and I guess my post didn't came out as expected. Was a bit distraught today because besides the burst ego ... some person (I had to delete what I originally written here) hit me with his tricycle while I was riding my motorcycle to work and rode away leaving me in the ditch ... literally. The people who helped me didn't get his plate number though. I didn't suffer any injury (my riding gear saved me) but he horribly disfigured my bike ... but thats another story.

So I would like to request the site admin to move this post to the Help forum and change the title to "Ice with C# not multi-threading properly" if its Ok.

[matthew]

Its not really very clear what you are doing. Do you actually have simultaneous clients accessing the server? If so, the default configuration is only one thread which may or may not saturate your link depending how fast the link is and how fast the clients & servers send and receive. I would try boosting the number of threads in the thread pool.

I would also verify that you are comparing apples & apples. What are you actually comparing against in your tests? Another native C# app? It is not using any native code? You are also using compression in your test. Are the compression algorithms the same? You should also, to be fair, ensure that the server-side processing in each case is identical (ie: same data, same server load, etc).

Did you try the same Ice application in C++ (especially since it is very simple). It might be interesting to try with Ice for C++ and perhaps even IceE (which is only slightly slower than the maximum possible throughput.

[amrufon]

Originally Posted by matthew

Its not really very clear what you are doing. Do you actually have simultaneous clients accessing the server? If so, the default configuration is only one thread which may or may not saturate your link depending how fast the link is and how fast the clients & servers send and receive. I would try boosting the number of threads in the thread pool.

Basically, there is already an existing application called iPLEX and it was having some trouble with simultaneous user access. What we basically did was replaced the data transfer middleware with ICE and .Net's WebClient object and both will be using C#. If you would review the attached image in the first post, there is V1 (version 1 ... not using ICE or the WebClient) and V2 which iether uses ICE or WebClient. The comparison was done using 3 simultaneous users. The problem with the V1 code is that it works well with 1 user but goes haywire with 2 or more users. To reiterate, no code changes was done on the client and server except the data transfer code.

I have always thought that ICE was by default using more than 1 thread ... so I'm a bit surprised that its using only one. Which property should I change to make ICE use more than 1 thread?

Originally Posted by matthew

I would also verify that you are comparing apples & apples. What are you actually comparing against in your tests? Another native C# app? It is not using any native code? You are also using compression in your test. Are the compression algorithms the same? You should also, to be fair, ensure that the server-side processing in each case is identical (ie: same data, same server load, etc).

I can assure you that no other changes were done on the core functionality of the application besides the data transfer. On the attached image, there were 5 specific steps and have to be done twice.

1. Login - Verifies user credentials
2. SP07 Style - Retrieves all data (around 20 tables) about Spring '07 Shirts styles
3. LCO - Retrieve all data regarding the SP07 order information
4. MRP - Runs the Material Resource Planning code on the server which calculates what and how much materials need to be ordered to produce the SP07 Shirts and returns the result
5. IO File - Retrieves data on the server for printing reports

So as you can see, all server side and client side processing are exactly the same since were using the same database and process operations.

About compression, although in the client code I pasted on this page doesn't have the -z parameter ... it was enabled during the test. It was changed when I was mucking around with it after the test. Technically, Rhea's code uses the built-in compression that comes with .NET.

Originally Posted by matthew

Did you try the same Ice application in C++ (especially since it is very simple). It might be interesting to try with Ice for C++ and perhaps even IceE (which is only slightly slower than the maximum possible throughput.

I just got swamped again with work but I am going to build the C++ equivalent. I'm just having some trouble doing the C++ .NET equivalent since the server side must call a C# class. I should look into IceE too since its been a while since I actually read the documentation.

[matthew]

You can set the size of the server side thread pool with the Ice.ThreadPool.Server.Size property. If you have concurrent clients and only the default thread pool configuration then this will likely cause a slow-down in your server as you are seeing since you will get no concurrency.

[amrufon]

Hi Matthew.

I couldn't get the same people to test the code but I was able to get some to help me test.

So with 3 more people and setting the Ice.ThreadPool.Server.Size to 256, the system seems to run more slowly. around 3 to 10 seconds more slower than the original timing I posted. Unfortunately, I couldn't get the same machines as the test system so its inconclusive.

One thing though, I don't think my implementation is compressing the data. You see, when I set the Ice.Override.Compress to true ... Im getting an error that it cannot find libbzip2 (please see attached image). I believe this has something to do with the slow performance since were talking about an average of 1.5mb of data being transfered from server to client.

I'm not giving up on this. I may just setup a sample using my project although I would love to use iPLEX project since it has the data and the hardware to test but its not my project and I'm just "helping out".

My plan right now is to build a C++ server and look into the IceBox implemention as soon as I have some freetime with my dayjob.

Any comments or reaction or pointers would be helpfull. Thanks.

r/Alex

[matthew]

256 threads is a huge number. The performance will doubt be terrible unless you have loads of CPUs and loads of memory. Some sensible value like 5 or 10 will get you better performance.

If you are not getting compression then you must not have the libbz2 library in your PATH.

[michi]

Could you let me know if you are using Visual Studio 2005 with .NET 2.0, or Visual Studio 2003 with .NET 1.1 please? It appears that there are changes in .NET 2.0 in the socket implementation that affect the behavior of non-blocking I/O. I'm still tracking down the precise problem, and would appreciate knowing which compiler you are using.

Thanks,

Michi.

[amrufon]

Originally Posted by michi

Could you let me know if you are using Visual Studio 2005 with .NET 2.0, or Visual Studio 2003 with .NET 1.1 please? It appears that there are changes in .NET 2.0 in the socket implementation that affect the behavior of non-blocking I/O. I'm still tracking down the precise problem, and would appreciate knowing which compiler you are using.

Thanks,

Michi.

Im using Visual Studio 2005 Professional with .NET 2.0

[amrufon]

Originally Posted by matthew

256 threads is a huge number. The performance will doubt be terrible unless you have loads of CPUs and loads of memory. Some sensible value like 5 or 10 will get you better performance.

If you are not getting compression then you must not have the libbz2 library in your PATH.

Hello Matthew.

I'll lower the number.

Btw, about the compression, I thought that adding the BIN directory of ICE to the PATH variable would allow ICE to find libbz2. I even put it in front of the PATH declaration.

r/alex

[michi]

Originally Posted by amrufon

Im using Visual Studio 2005 Professional with .NET 2.0

Is there any chance for you to try this with Visual Studio 2003 with .NET 1.1? I have some indication that socket performance has suffered with .NET 2.0.

Cheers,

Michi.

[amrufon]

Originally Posted by michi

Is there any chance for you to try this with Visual Studio 2003 with .NET 1.1? I have some indication that socket performance has suffered with .NET 2.0.

Cheers,

Michi.

Our VS2003 is on a VMWare machine since we've upgraded to 2005. Unfortunately, the earliest I can do this is around tuesday next week. I'll post the results as soon as I got it.

r/Alex

[michi]

Btw, about the compression, I thought that adding the BIN directory of ICE to the PATH variable would allow ICE to find libbz2. I even put it in front of the PATH declaration.

OK, bug in Ice: the code looks for libbz2, but the installer installs it as bzip2.dll. We'll do something about this for the next version of Ice.

As a work-around, you can copy libbz2.dll to bzip2.dll. That will make sure that the library is found (as long as the directory in which the library lives is in your path).

Please let me know if that doesn't fix your problem.

Cheers,

Michi.