Huge Virtual Memory Size in a multi-connections client

[dscardaci]

Hi,
I'm developing a test-suite to test a backend component (the server in my scenario).
My test-suite has to simulate a group of clients connecting to the backend. The backend component uses Glacier2.

I work in Ubuntu environment on a 64 bit machine.

To reach my target I developed the following steps:
1) Create the communicator:
ic= communicator();

2) Get the default router
router = Glacier2::RouterPrx::checkedCast( ic->getDefaultRouter() );

then for each client:

1) Create a new connection
router = Glacier2::RouterPrx::uncheckedCast(rt->ice_connectionId(ide_router));

2) Create a session
base = router->createSession(serial, user);
session = ade::argo::iGl2SessionPrx::uncheckedCast( base->ice_connectionId(ide_router)->ice_router(router) );

3) Clients use the AMI interface to interact with the backend

Examining the status of the machines I saw that the virtual memory used by my test-suite with about 15000 parallel connections is huge. In a test-case the virtual memory size is 235 GB after all the 15000 clients completed the connection, in another testcase the virtual memory size reach the 255 GB and the test crashes.

Is this behavior normal?
Is there a way to reduce the virtual memory size used by my test_suite?

Thanks in advance for your help.

[benoit]

Hi Diego,

Which Ice version do you use? I'll assume you use Ice 3.3.0. Establishing 15000 connections shouldn't require that much memory. However, I'm afraid it's difficult to figure out what could be wrong without more information.

One possible reason for this memory usage is that your client sends more AMI requests than the server can handle and as a result the AMI request queue keeps growing eventually consuming a lot of memory. You should consider adding some flow control if this is the case (see here for more information).

Another possible reason is that you create a communicator per connection/Glacier2 session and each communicator creates a number of threads that ends up consuming a lot of memory if you use the default thread stack size. If that's the case, you should consider using a single communicator instead.

Cheers,
Benoit.

[matthew]

I'll go a bit further, and say that lots of connections does not require lots of memory. During Ice 3.3 scalability testing, which I detailed in my blog post Glacier2 Scalability Improvements in Ice 3.3 - Matthew Newhook - ZeroC Blogs, I established over 8000 connections on a very old 32-bit Linux machine. The problem in the end wasn't memory, it was CPU. You must be doing something to eat all that memory in your application.

[dscardaci]

Hello,
thanks for your answers.

I understood the problem. The huge virtual memory has been allocated by a big number of threads.
In my test suite, I associated to each connection a class implementing IceUtil::TimerTask because I have to do some periodic operations. In this class I wrongly added a timer instance. Then, my test suite created a timer (and a thread) for each connection.
I fixed the problem creating only one timer for all the connection.

However, with only one timer I experienced another problem. In some tests, when the execution time of the operation involved is not short, the periodicity is not guaranteed.

To solve this problem now I'm using a set of timers (for example one timer for each 100 connections) to try to guarantee the periodicity without consume all the memory.

Do you think this is the correct way?

Thanks in advance,
Diego

[matthew]

I'm glad to hear you are making progress! I would think a better solution is to ensure that each timeout handler completes in a timely manner. For example, if the timeout handlers make remote invocations, you can ensure that by using AMI. If that isn't possible, then perhaps I would recommend adding a pool of threads behind the timer. Currently, however, that is not possible with our Timer class.