Scalability Questions

[feline]

First, let me say that ICE is very impressive. Although I have not used it yet, I have read all the documentation and examined various bits of the source. All very impressive.

I have some scalability questions. I have a need to develop a server which will need to have several thousand simultaneous connections. About 2000-5000 on average with up to about 8000 connected users during peak loads. Based on what I have seen in the source it is not possible for an ICE server to handle more than 1024 connections due to the usage of select(). Now I realize there are some work around which can increase the FD_SETSIZE. I consider this a brute force solution....albeit an easy one on some platforms.

Have you considered using poll() instead of select() on the platforms which support poll()? (most unix platforms support poll(). The poll() usage would solve two issues....(a) number of connections, (b) if I open 500 files...then ICE would only accept ~500 connections....as the maximum fd supported by most select() calls is FD_SETSIZE (which is 1024 on most platforms by default).

Now I have read....select() and poll() do not scale well for >1000 connections in many cases.....mainly due to user/kernel context switching I think?? So achieving scalability via poll() or increasing FD_SETSIZE might not be the best way to support many thousands of simultaneous connections. What other solutions do you recommend? I looked at glacier...and it comes close to supporting what I had in mind...this was my thought....and I was curious if glacier already supported it...or if ice came close to supporting it. The general idea is that I have one server which runs on a well-defined port. Its sole purpose in life is to accept connections and re-direct them to a router process (like glacier starter appears to do). This server would keep track of how many connections each server contains and appropriately start more routers or kill routers depending on current load. Each router would connect to my main backend server and simply forward client messages to the backend server. Effectively I could get say 1000 connections to each router, and connect 1000 routers to my main backend server.....giving me more than enough connections.

Is there already support in glacier and/or ice for this? Is this feasible in ICE?....ideally I do not want to spend much time on the router part of implementation. It looks like Glacier comes very close to this....and it is likely I will need to use glacier anyway....what is not clear is if the glacier starter can handle the load-balancing/connection arbitration part of it all.

Is this clear? Has anyone tackled similar problems in the past? Anyone got better ideas?

Thanks in advance for your help

[marc]

We didn't think about using poll() instead of select() yet. I have to do more reading on what the advantages of poll() over select() are.

The first question that comes to my mind is, are all 8000 clients active at peak time, i.e., do they send requests all the time, or are they mostly idle? If they are mostly idle, you can simply use active connection management. Then connections will be transparently closed and re-opened.

If they are mostly active, then you might have a problem that goes beyond the number of connections only. If 8000 clients send requests all the time to a single server, and the server has to process all these requests, then you will need a very fast machine to process all these requests! If this is the case, then perhaps an architecture with multiple servers would be better? You could then simply give your clients endpoints to all server instances, and a client would pick one of them randomly, meaning that you have a load-balancing mechanism.

Glacier can serve as a connection concentrator, as long as you don't use callbacks with bi-directional connections. In case you use Glacier for such bi-directional connections, there is a 1:1 relationship between a Glacier router instance and a client. However, if you do not have callbacks, then you can use one Glacier router to server any number of clients (or up to the limit of open connections your operating system supports).

Another possibility, which involves more work, is to design your own application-specific connection concentrator. There are many ways to do this, and to list them all would be beyond the scope of advice I can give in a single email.

One example is to write a session manager, which creates a session object for each new client. You could then have several such session managers, which then serve as a connection concentrator. All requests would be send through the session objects, meaning that the server would only see one connection per session manager, and not one per client.

This could be done in a generic manner, i.e., simply have your session object forward all requests as "blobs". Or you could move some of your "business logic" into the session objects, and have them do some of the pre- or post-processing.

Again, there are many possibilities here, and to give you better advice, I would need to know more about the nature of your application.

[feline]

To give you some more context. Think of the server as a chess server say. That is basically a big group of people either chatting or playing games of chess. Sometimes users will simply be chatting, other times playing a game, other times watching games etc. In all cases I think all connections will be active, e.g. if you are chatting you are obviously active, if you are playing you are either making moves or waiting for your opponents move (typically not a long period of time), other times you will be watching a game(s). In all cases the connections will be active....but there is not alot of data exchanged. E.g. a move is not big...sending or receiving, chatting is not alot of data to or from a single connection.

I do not think Active Connection management is suitable here...right? Sure I will get some users who log on and then walk away for periods....but I would not expect that to be common.

I had thought of breaking it all up into seperate servers....but the goal it is allow all 2000-5000 see each other and have the possibility of playing each other. I may change that if my current goals are not attainable. Note that for the peak 8000 load will only likely occur when some event is occuring...world championship relay of games via the server or something like that.

Essentially the server will maintain the state of all games, maintain some chat channels, maintain conversations between two users, maintain some configuration for each user etc. All the communications will likely be very small....less that 200 bytes in many cases.

Thanks for the quick reply.

[benoit]

Originally posted by feline
To give you some more context. Think of the server as a chess server say. That is basically a big group of people either chatting or playing games of chess. Sometimes users will simply be chatting, other times playing a game, other times watching games etc. In all cases I think all connections will be active, e.g. if you are chatting you are obviously active, if you are playing you are either making moves or waiting for your opponents move (typically not a long period of time), other times you will be watching a game(s). In all cases the connections will be active....but there is not alot of data exchanged. E.g. a move is not big...sending or receiving, chatting is not alot of data to or from a single connection.

I do not think Active Connection management is suitable here...right? Sure I will get some users who log on and then walk away for periods....but I would not expect that to be common.

Yes, sounds like active connection management wouldn't really help since there will always be a little traffic.

I had thought of breaking it all up into seperate servers....but the goal it is allow all 2000-5000 see each other and have the possibility of playing each other. I may change that if my current goals are not attainable. Note that for the peak 8000 load will only likely occur when some event is occuring...world championship relay of games via the server or something like that.

Essentially the server will maintain the state of all games, maintain some chat channels, maintain conversations between two users, maintain some configuration for each user etc. All the communications will likely be very small....less that 200 bytes in many cases.

Thanks for the quick reply.

Distributing your application over several servers would probably be the best thing to do, remember, Ice is about distributed computing .

First, you would have front end processes to process client requests, these processes would concentrate client connections and also filter incoming requests (firewall). You could probably use Glacier for this (but you still have the possibility to write your own routing service if Glacier doesn't suite your needs).

Then, you would have back end processes to handle the logic of the game. The clients would communicate with the back end servers only through the front end processes. So, you wouldn't have to worry about network scalability here. There would be only few connections between the back ends and the front ends. These back end processes could handle different aspect of the game, such as the chat system, the chest playing games, the meeting rooms, ...

This architecture is very similar to the architecture of Wish, an Ultra Massive Online Game. You might want to have a look at this poster, http://www.zeroc.com/documents/wishposter.pdf, which describes the architecture of Wish. Michi also wrote an article about Wish, see http://www.zeroc.com/acmArticle.html

How you distribute the logic of the game in separate services depends in part on how you can distribute the state of the game. For example, the chat system state is probably independent of the playing games so you can separate these 2 systems. Playing game states are also probably independent from each other so you could distribute playing games over several servers if you need to.

Instead of having a single highly scalable server (hard to do!) that handles all the communications and logic, you distribute the services over several smaller servers. This is much more flexible and will allow you to better scale. If one machine isn't enough to handle the increasing load, it's much easier to add a second machine with a distributed design

Benoit.

[marc]

Originally Posted by marc

Glacier can serve as a connection concentrator, as long as you don't use callbacks with bi-directional connections. In case you use Glacier for such bi-directional connections, there is a 1:1 relationship between a Glacier router instance and a client. However, if you do not have callbacks, then you can use one Glacier router to server any number of clients (or up to the limit of open connections your operating system supports).

Note that the above referred to Glacier 1, which is not supported anymore. The new Glacier 2 always serves as a connection concentrator, even if bi-directional connections are used. With Glacier 2, there is a 1:n relationship between the router and the clients, and therefore it concentrates all connections from the clients to one single connection to the server (or a single connection pair, if there are callbacks). Glacier 2 is stateless, and can therefore easily be replicated, so that several Glacier 2 instances can work as connection concentrators and split the load of the clients.

[OrNot]

Hi, Marc,
I have some vague understandings about Glacier2 and need your clarificaion.

Assuming 5000 clients will connect one server through
a glacier2router and bidir is necessary and every connection
can only be closed by client manually.

1) The interanl implementation of Glacier2Router seems to adopt thread-per-connection strategy. That is ,if i set buffer mode (client and server ), every client connection will bring 3 threads up? If so, 5000 clients will need 15,000 threads in glacier2router? This seems incredible.

2) Since Glacier2Router acts as a concentrator, it logically serialize the concurrent requests from clients. If 5000 clients send requests simultaneously, even these requests are of samll size, the Glacier2Router socket send buffer will be filled up quickly and block here regardless of oneway or AMI. If unbuffered mode, this block seems more severe.

3) One of the purposes of using glacier2router is to split the load of the server computer into other server computers? But if we have to run the glacier2router with the server process on one computer, can we gain somethings from glacier2router ?

4) Since a threadpool only support at most 1024 sockets, to overcome this , is it ok for me to create , say, 5 , object adapter replica with different ports and their own threadpool in one server ?



Best Regards ---OrNot

[marc]

Originally Posted by OrNot

1) The interanl implementation of Glacier2Router seems to adopt thread-per-connection strategy. That is ,if i set buffer mode (client and server ), every client connection will bring 3 threads up? If so, 5000 clients will need 15,000 threads in glacier2router? This seems incredible.

Yes, in this case there would be 15,000 threads. However, in most cases you would only use buffered server->client mode, which would reduce the number of threads to 10,000. With a modern operating system, having 10,000 threads is no big deal, as long as you configure a reasonable stack size per thread.

However, keep in mind that with 5,000 clients, the bottleneck will not be the number of threads, but the load that these 5,000 clients put on the Glacier router in general. If the clients are mostly idle, then you don't have a problem. But if they are all very busy, then you better distribute the load from the 5,000 clients among many front-ends running Glacier routers.

Originally Posted by OrNot

2) Since Glacier2Router acts as a concentrator, it logically serialize the concurrent requests from clients. If 5000 clients send requests simultaneously, even these requests are of samll size, the Glacier2Router socket send buffer will be filled up quickly and block here regardless of oneway or AMI. If unbuffered mode, this block seems more severe.

This would happen with our without Glacier. If the server cannot keep up with the number of requests, then clients start to block. This has nothing to do with connection concentration or Glacier.

Originally Posted by OrNot

3) One of the purposes of using glacier2router is to split the load of the server computer into other server computers? But if we have to run the glacier2router with the server process on one computer, can we gain somethings from glacier2router ?

You do not have to run glacier2router with the server process on the same computer. A common scenario for a high-load system is to have several redundant front-ends running Glacier, and then several back-ends, running the various server processes.

Originally Posted by OrNot

4) Since a threadpool only support at most 1024 sockets, to overcome this , is it ok for me to create , say, 5 , object adapter replica with different ports and their own threadpool in one server ?

Why can a threadpool only support at most 1024 sockets? Ice has no such limitation. In any case, to answer your question, yes, you can create multiple object adapters with different thread pools.

[OrNot]

Originally Posted by marc

Yes, in this case there would be 15,000 threads. However, in most cases you would only use buffered server->client mode, which would reduce the number of threads to 10,000. With a modern operating system, having 10,000 threads is no big deal, as long as you configure a reasonable stack size per thread.

Sorry Marc, what you said makes me puzzled again. Do you mean if a server has to deal with thousands of clients, which is common in many cases, thread-per-connection may be a preferable strategy since the programing in this thread strategy seems easier and more straightforward?
Then, when should I use thread pool? Can you give me a generic principle?

Originally Posted by marc

However, keep in mind that with 5,000 clients, the bottleneck will not be the number of threads, but the load that these 5,000 clients put on the Glacier router in general. If the clients are mostly idle, then you don't have a problem. But if they are all very busy, then you better distribute the load from the 5,000 clients among many front-ends running Glacier routers.

Yeah, I got it.

Originally Posted by marc

This would happen with our without Glacier. If the server cannot keep up with the number of requests, then clients start to block. This has nothing to do with connection concentration or Glacier.

I mean if many clients send even small size requests simultaneously, since they all use one connection to server, the send buffer may be filled and these threads will be blocked. In this stage, it is simlar to the situation dicussed in
ICE::AMI is useful?
Where Benoit had said " Since you're sending many of these requests, the network buffers eventually quickly fill up. When the buffers are full, the client sayHello_async call will block."
But I think this is inevitable due to the role of the concentrator acted by glacier2.

Originally Posted by marc

You do not have to run glacier2router with the server process on the same computer. A common scenario for a high-load system is to have several redundant front-ends running Glacier, and then several back-ends, running the various server processes.
.

OK. I got it.

Originally Posted by marc

Why can a threadpool only support at most 1024 sockets? Ice has no such limitation.

You mean I can change some #define to increase the number of sockets used by "Select()" function? I just read the initial post of this thread , which said the limitation of 1024.

[marc]

Originally Posted by OrNot

Sorry Marc, what you said makes me puzzled again. Do you mean if a server has to deal with thousands of clients, which is common in many cases, thread-per-connection may be a preferable strategy since the programing in this thread strategy seems easier and more straightforward?
Then, when should I use thread pool? Can you give me a generic principle?

The reason why Glacier uses thread per connection, is because it isolates clients from each other, meaning that one misbehaving client cannot impact other clients. However, for most other use cases, thread pool is more appropriate.

Originally Posted by OrNot

I mean if many clients send even small size requests simultaneously, since they all use one connection to server, the send buffer may be filled and these threads will be blocked. In this stage, it is simlar to the situation dicussed in
ICE::AMI is useful?
Where Benoit had said " Since you're sending many of these requests, the network buffers eventually quickly fill up. When the buffers are full, the client sayHello_async call will block."
But I think this is inevitable due to the role of the concentrator acted by glacier2.

Sorry, but I'm afraid I don't understand what you mean, and how the thread you quote relates to the discussion.

In general, if the server can keep up with the clients, then everything is fine, regardless of whether you use Glacier or not. If it cannot keep up with the clients, then clients will start to block, also regardless of whether you use Glacier or not.

Originally Posted by OrNot

You mean I can change some #define to increase the number of sockets used by "Select()" function? I just read the initial post of this thread , which said the limitation of 1024.

The limit for the number of sockets that select() can handle depends on the operating system. On most operating systems, including Windows XP, you can increase the limit by defining FD_SETSIZE to a higher value, and recompiling Ice.

However, as I already wrote in this thread, I doubt that this will be so useful. If you have more than 1000 clients that are busy sending requests all the time, then you will have problems handling the load with a single server. If, on the other hand, clients are idle for most of the time, then you can use active connection management to close idle connections to reduce the number of sockets to a reasonable level.

If you need further advice with how to partition and distribute a server so that it can handle thousands of clients simultaneously, then I recommend that you contact us at info@zeroc.com for commercial consulting services. Unfortunately, there are limits on how much free support we can give here in this newsgroup.

[OrNot]

Originally Posted by marc

The reason why Glacier uses thread per connection, is because it isolates clients from each other, meaning that one misbehaving client cannot impact other clients. However, for most other use cases, thread pool is more appropriate.

Yes, Thank you so much.

Originally Posted by marc

Sorry, but I'm afraid I don't understand what you mean, and how the thread you quote relates to the discussion.

In general, if the server can keep up with the clients, then everything is fine, regardless of whether you use Glacier or not. If it cannot keep up with the clients, then clients will start to block, also regardless of whether you use Glacier or not.
.

I understand what you said clearly,while I can not express myself as clearly as you did due to my poor English. .But you are definitely correct . Thank you any way.

Originally Posted by marc

The limit for the number of sockets that select() can handle depends on the operating system. On most operating systems, including Windows XP, you can increase the limit by defining FD_SETSIZE to a higher value, and recompiling Ice.
.

It is what I expected and hope it is possile for NT, 2K.

Originally Posted by marc

However, as I already wrote in this thread, I doubt that this will be so useful. If you have more than 1000 clients that are busy sending requests all the time, then you will have problems handling the load with a single server. If, on the other hand, clients are idle for most of the time, then you can use active connection management to close idle connections to reduce the number of sockets to a reasonable level.
.

I can not agree with you any more . But if the case is not as bad as you said, say, all the clients just are sending requests of small size periodically (let's say 60 s ), ACM seems not very suitable especially in bidir , I think.

Originally Posted by marc

If you need further advice with how to partition and distribute a server so that it can handle thousands of clients simultaneously, then I recommend that you contact us at info@zeroc.com for commercial consulting services. Unfortunately, there are limits on how much free support we can give here in this newsgroup.

Yes, I know. I am very happy getting your so patient explaining and I have learned so much. Thank you for your time. But commericial consulting services are too expensive for me, a student, to enjoy.


Best regards
OrNot