Number of threads per ICE server/client

[kitovyj]

Hello!

Our solution is supposed to have many ( > 1000) small Ice servers as separate processes working on the same machine. So we are interested in lowering system resource consuption as much as possible.

I noticed what simple server with source code listed below spawns 6 threads.
As far as I understand they are:

1. main thread
2. 1 thread in client thread pool
3. 1 thread in server thread pool
4. host resolver thread
5. selector thread
6. ?

My question is: is it possible to lower the number of threads spawned? Is it possible to terminate resolver thread after the resolving is done? Is it possible to use "main" thread as selector thread? It would be ideal to have only one thread waiting on "epoll_wait"

Best regards,
Alexander Ivanenko

Simple server code:

communicator = Ice::initialize(argc, argv);
Ice::ObjectAdapterPtr adapter = communicator->createObjectAdapterWithEndpoints("client", "tcp");
WorkerPtr w = new WorkerI();
WorkerPrx prx = WorkerPrx::checkedCast(adapter->add(w, communicator->stringToIdentity(lexical_cast<string>(getpid()))) );
adapter->activate();
HostPrx host = HostPrx::checkedCast(communicator->stringToProxy("xcoder:tcp -p 10000"));
host->registerWorker(prx);
communicator->waitForShutdown();
communicator->destroy();

[kitovyj]

It seems like 3.4.2 spawns 5 threads, not 6.

[marc]

No, you can't terminate the internal threads, other than by shutting down the Communicator as a whole. Why do you use so many small servers as separate processes? This doesn't seem to be a very efficient approach. Ice servers are designed to handle many clients, so it should not be necessary to create so many servers.

[kitovyj]

Strictly speaking they are not servers, just clients registering callback interfaces on the host.
We are developing video transcoder, each transcoding job is isolated in a separate process, being an Ice client. When such a client starts, it register itself on an Ice server as "worker" with a callback interface, something like :

interface worker {

void start_transcoding(...
void stop_transcoding(...
...
etc

The problem is what we will run thousands of such workers on the same machine and want to minimize their system resources usage.

No, you can't terminate the internal threads, other than by shutting down the Communicator as a whole. Why do you use so many small servers as separate processes? This doesn't seem to be a very efficient approach. Ice servers are designed to handle many clients, so it should not be necessary to create so many servers.

[marc]

I would recommend to write a single process (or a few processes) that can handle multiple transcoding processes, utilizing a thread pool that you can configure to optimize performance depending on the number of physical CPUs you have. Using thousands of processes might be convenient, but it is not efficient, with or without Ice.

[kitovyj]

I would recommend to write a single process (or a few processes) that can handle multiple transcoding processes, utilizing a thread pool that you can configure to optimize performance depending on the number of physical CPUs you have. Using thousands of processes might be convenient, but it is not efficient, with or without Ice.

It would be much easier to use a thread-per-transcoder model instead of process-per-transcoder. But it is a question of fault-tolerance. Video encoders/decoders are too complex to be bug-free, and we don't want to let the seg. fault in one transcoding job crash the whole server. That is why we want to have each xcoding job in a separate process.

[marc]

Well, I guess you could choose some middle ground here, by using a few processes that each handle multiple transcoding jobs. You could also use IceGrid to manage these processes, so that they are activated on demand. And of course if the transcoding processes crash so often, then I would look into why this is the case and debug them. No application is too complex to make crashes acceptable

Anyway, I'm afraid I can't offer you any advice beyond what I wrote already. I do not believe you will be able to have an efficient system with so many processes, regardless of whether you use Ice or not.

[kitovyj]

Well, I guess you could choose some middle ground here, by using a few processes that each handle multiple transcoding jobs. You could also use IceGrid to manage these processes, so that they are activated on demand. And of course if the transcoding processes crash so often, then I would look into why this is the case and debug them. No application is too complex to make crashes acceptable

Anyway, I'm afraid I can't offer you any advice beyond what I wrote already. I do not believe you will be able to have an efficient system with so many processes, regardless of whether you use Ice or not.

Thank you for your answers! I think we will manage it one way or another.