Performance

[gthaker]

Since we are on this topic, let us once again return to just the octet performance. (I think Marc has prev. said that for large msg sizes Ice prob. has some optimizations that can be done; this seems esp. true when structs are involved.)

Attached graphic shows TAO and ORBexpress (1.3.1 and 2.3.5 versions, respectively), with default concurrency and with "single threaded" configuration. Here one sees that TAO 1.3.1 "default" configurations is slower than Ice 1.0.1. However, there is a cross over for larger msg sizes. You will be able to gain this back with some optimizations.

The purpose of my work is not to understand performance issues in the large and in the small. Thus, I hope the numbers speak for themselves - we have no commercial interests in any of the stuff we study.

Gautam

[marc]

Originally posted by gthaker
Marc,

First of all, I want to reiterate that all data I have is online so these graphs can be reproduced. What I mean is that one can regenerate different graphs showing different comparisons. There are so many different ways to look at the data. That said, i will try to address the points you make.

1) THe purpose of my last post is to compare "struct" marshalling cost. So I used TAO 1.2.2 results for which I have both octet and struct data. I don't have TAO 1.3.1 struct results yet. TAO 1.3.1 is bit slower than TAO 1.2.2. Thus, in prev. graph Ice 1.0.1 and TAO 1.3.1 were close the the low end. Now I show TAO 1.2.2 results and that is a bit faster. (BUt as I have said, the first factor of 1.5-2 is not always that important.)

Thanks for the clarification. It is clear that there is a problem in Ice with long sequences, and we will definitely improve this.

I believe, however, that for latency tests, a factor of 1.5-2 is very significant. All I wanted to say is that I don't understand why TAO is 1.5-2 times faster in your tests, when Ice is faster in our tests. Again, I believe this is because of different concurrency models.

Originally posted by gthaker

2) I don't know how you are concluding about the graph showing ORBexpress to be > 10 faster than either TAO or Ice. The Y axis is indeed logscale but there is not an entire factor of 10 difference in the curves.

Yes, you are right, I misread the scale. I guess it's more like a factor of 5 perhaps, which is still surprising. I know that OrbExpress is probably the fastest ORB around, but I wouldn't have expected such a big difference. However, while I don't believe that Ice can match OrbExpress performance, I think it will be a lot closer if OrbExpress uses the same concurrency models. (But this is pure speculation, as I don't have access to OrbExpress.)

Originally posted by gthaker

From my website I generated a new plot comparing time it takes for two processes to exchange octets of information. Fastest is shared memory (this is an SMP machine). It bypasses the network stacks and at low end it is as fast as two context swithces. Next comes TCP/IP, after that is ORBexpress, than TAO and than ICE. You can see this in the attached graphic.

I hope this is clear.

Thanks, this is very helpful.

A great test to add for your test suite would be a nested test, which is not possible with simple concurrency models. For example, an application that has a nesting level of 5, and then measure the overall time needed for all calls. You'll find a nested demo in demo/Ice/nested.

[marc]

Originally posted by gthaker
Since we are on this topic, let us once again return to just the octet performance. (I think Marc has prev. said that for large msg sizes Ice prob. has some optimizations that can be done; this seems esp. true when structs are involved.)

Attached graphic shows TAO and ORBexpress (1.3.1 and 2.3.5 versions, respectively), with default concurrency and with "single threaded" configuration. Here one sees that TAO 1.3.1 "default" configurations is slower than Ice 1.0.1. However, there is a cross over for larger msg sizes. You will be able to gain this back with some optimizations.

Thanks a lot! This is the result I would have expected. We will definitely work on improving the large-message problem.

Originally posted by gthaker
The purpose of my work is not to understand performance issues in the large and in the small. Thus, I hope the numbers speak for themselves - we have no commercial interests in any of the stuff we study.

Gautam

I wasn't suggesting anything like this. It's just that we take pride in our work here at ZeroC, so we want to make sure that if Ice is compared to some other product, that such comparison is "fair". This latest test definitely is fair, and it shows better performance for Ice for small messages, and better performance for TAO for large messages.

As for OrbExpressRT, I must admit that we can't reach their performance.

[gthaker]

Actually, I made a typo. Our purpose *IS* to understand performance issues in the large and small scales. Thus I try to collect data such as shared memory, TCP/IP, various ORBs and other MW, RMI, EJB etc. And when doing a lot of testing of course configurations are important. (Hopefully we are not making any gross errors but constant cross checking helps - for example, IIOP based ORBs better always be slower than TCP, etc.)

I will try to see about the "nested" test that you mentioned.

Gautam

[CatOne]

Hi Gautam,

I was thinking last night -- one of the things that Ice has which is "new and different" from other middleware is the ability to compress messages. For use cases such as yours (varying packet sizes) I'd guess this could make a significant difference, if the payload is compressible.

What are you using for your messages? Any chance you could run that same benchmark with protocol compression on and see how it performs? I'm quite curious about this -- I'd be interested to know if the bzip2 compression and trading cpu cycles for network latency has a reasonable effect.

Of course if you're sending chunked .jpeg files or something it won't help at all ;-)

[marc]

Over a fast network (or even loopback), compression will rather slow down requests. Compression is intended to be used for slow connections, like modems.

Anyway, we already know what the problem with long messages is. We will fix this in the next release.

[CatOne]

Marc,

That's interesting to know. The documentation doesn't mention anything about that fact (that compression is for an intended use case) -- perhaps it would be useful to note there? Do you expect there's any payload size where it would make a difference over TCP (say, 5 MB)?

With your post in the announcements forum about Mutable Realms and their "Wish" game I have a guess as to which customer requested this feature.

[marc]

Originally posted by CatOne
Marc,

That's interesting to know. The documentation doesn't mention anything about that fact (that compression is for an intended use case) -- perhaps it would be useful to note there? Do you expect there's any payload size where it would make a difference over TCP (say, 5 MB)?

While I don't have any concrete measurements for this, I think for a fast network, the time to compress the data will always be longer than the actual transmission time. So compression on a fast network is not recommended.

You can make a simple test: Take a 5MB file, and compress it with bzip2 -1, and measure the time. Compare this time to the time needed to send the 5MB file over your network. I'm pretty sure the compression time will be longer in this case.

Originally posted by CatOne
With your post in the announcements forum about Mutable Realms and their "Wish" game I have a guess as to which customer requested this feature.

Your guess is correct For Mutable Realms' game project, lowering the bandwith consumption is very important. They don't use compression within the backend, but it is used for all communications between backend and the game client. Using Glacier, this can all be done transparently.

[gthaker]

For sake of completeness I repeated my prev. roundtrip latency measurements with Ice version 1.1.0. In attached graphs I show roundtrip latencies for "octet" and "struct" messages of different sizes. (Details are in prev. postings of these thead.)

While the compiler was kept constant at GCC 3.2.1 the hardware underwent an OS change from Linux 2.4.18 to 2.4.20. So I also show on each chart corresponding TCP/IP roundtrip latencies. While Ice 1.1.0-octet measurement is slower than Ice 1.0.1 measurement the difference is essentially the same by which the kernel has slowed down. One sees similar diff for 1.1.0-struct at small message sizes, but it seems that 1.1.0 has been improved over 1.0.1 for array of structs being transported.

Note: I just realized that I think I can only attach 1 file per posting. So I will "octet" graphic here and will attach "struct" data with next post and "both" on the 3rd graphic. Sorry if I missed a way to attach multiple .png files in one posting.

As usual, all data is at:

http://www.atl.external.lmco.com/projects/QoS -> "MW_Comparator" and

Ice and TCP subset of results can be reached more easily at:

http://www.atl.external.lmco.com/pro....*transport%29

Gautam Thaker

[gthaker]

Here is the "struct" graph.

Gautam

[gthaker]

Here is the final graph which shows all data together.

Gautam

[marc]

Thanks a lot for the performance tests. This is very helpful for us, as we continue to try to further improve performance.

[skolem23]

Hi,

in my current project i employed TAO, after some simple tests it turned out that TAO is by several times slower than omniORB. (i've moved to omniORB right now)

Please consider the following results provided by

http://nenya.ms.mff.cuni.cz/projects...ts-xampler/All

(Distributed Systems Research Group, Charles University, Prague)

Real time does not mean high performance. In my opinion it doesn't make much sense to compare TAO with ICE.

An extensive comparison between omniORB and ICE would be extremely interesting. (Since I'm with embedded stuff the memory footprints are of a major interest too)

Anyway, congratulations to the ICE developers, it seems to be great^^

[marc]

We might include omniORB in future performance comparisons, but don't hold your breath. It's a lot of work to do these tests thoroughly.

As for realtime, I yet have to understand what this really means in the context of non-realtime operating systems and non-realtime transports...