error: filedescriptor 1257 not registered with `Ice.ThreadPool.Server'

[kent]

Hi,
I have an Im Front Server that is to accept and manage client's connections, and communicates with Main Server and Session Server to maintain online stats, etc. The Im Front Server has 2 Ice::Communicator, one is to response the Main Server's request and the other is to request needed information from the Third Server. When Im Front Server accepts more than 1000 clients(we have reset the connection limit to 60,000), it gets the following error (error: filedescriptor 1257 not registered with `Ice.ThreadPool.Server'), in which, the fd (1257) could ranges from 1025 to any other fd bigger (for example 1257). What's the problem?

[matthew]

What operating system and version of Ice are you using?

[kent]

OS:Red Hat Enterprise Linux AS release 4 (Nahant)
Ice: 3.0.1

I do a little test as following:

Server:
single thread
use epoll to dispatch events from client and provide a simple Ice service.

Client:
multi thread for each socket connection and Ice service calling
connect to Server and send a message every 1 second, call Ice service every 1 second.

if Client make more than 1024 connection first and then call Ice service, crash after soon,
if Client call Ice service first and then make more than 1024 connection, it works,
if Client make less than 1024, it works in spite of startup order

[matthew]

Sorry, I don't understand your setup. What exactly are you doing with epoll, and what does that have to do with Ice? Ice internally uses select, not epoll. There is no hard limit at 1024 -- if there was you would reach this before anyway due to some of the other fd's that your process opens (stdin, stdout, etc). How are you forcing so many connections to be established? What exactly are you doing? Posting an example that duplicates this problem would help.

[kent]

The server is a Instance Messenger Front Server, so it has to accept many connections, receive messages from each connection and post it to another Ice Server. The Front Server also include a Ice Adapter to response other server's request(message routing, etc).

We find that the problem occurs only when the max fd number is larger than 1024(or other) of Ice's connection. we fcntl accepted connection's fd larger than 1000, and reserve fd less than 1000, there's no problem even accepts more than 2000 connections.

so, it is like that fd number is the key.

[kent]

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/queue.h>
#include <unistd.h>
#include <sys/time.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <unistd.h>
#include <string>
#include <arpa/inet.h>
#include <errno.h>
#include <netinet/in.h>
#include <limits.h>
#include <fcntl.h>
#include <event.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <unistd.h>

#include <iostream>
#include <Hello.h>
#include "Channel.h"

using namespace std;
using namespace MyUtil;

class HelloI : virtual public im::Hello
{
public:
virtual void say(const string& message, const Ice::Current&) {
//cout << message << endl;
}
};

void
socket_read(int fd, short event, void *arg)
{
char buf[255];
int len;
struct event *ev = (struct event*)arg;

//
// fprintf(stderr, "socket_read called with fd: %d, event: %d, arg: %p\n",
// fd, event, arg);
len = read(fd, buf, sizeof(buf) - 1);

if (len == -1) {
delete ev;
perror("read");
close(fd);
return;
} else if (len == 0) {
fprintf(stderr, "Connection closed\n");
delete ev;
close(fd);
return;
}

buf[len] = '\0';
//fprintf(stdout, "Read: %s\n", buf);

/* Reschedule this event */
event_add(ev, NULL);
}

void
socket_accept(int socket, short event, void* arg)
{
struct event *ev = (struct event*)arg;
/* Reschedule this event */
event_add(ev, NULL);

int fd = accept(socket, NULL, NULL);
if (fd < 0) {
cout << "accept: " << strerror(errno) << endl;
return;
}
struct event* evsocket = new struct event;
event_set(evsocket, fd, EV_READ, socket_read, evsocket);

/* Add it to the active events, without a timeout */
event_add(evsocket, NULL);
}

int
setlimit(uint32_t filemax)
{
int ret;
struct rlimit rl;

ret = getrlimit(RLIMIT_NOFILE, &rl);
if (ret != 0) {
fprintf(stderr, "Unable to read open file limit: (%d, %s)\n",
errno, strerror(errno));
return 1;
}

fprintf(stderr, "Limit was %d (max %d), ",
(int) rl.rlim_cur, (int) rl.rlim_max);

if (rl.rlim_cur >= filemax) {
fprintf(stderr, "unchanged\n");
return 0;
} else {
fprintf(stderr, "setting to %d\n", filemax);
}

rl.rlim_cur = rl.rlim_max = filemax;
ret = setrlimit(RLIMIT_NOFILE, &rl);
if (ret != 0) {
fprintf(stderr, "Unable to set open file limit: (%d, %s)\n",
errno, strerror(errno));
return 1;
}

ret = getrlimit(RLIMIT_NOFILE, &rl);
if (ret != 0) {
fprintf(stderr, "Unable to read _NEW_ open file limit: (%d, %s)\n",
errno, strerror(errno));
return 1;
}

if (rl.rlim_cur < filemax) {
fprintf(stderr, "Failed to set open file limit: "
"limit is %d, expected %d\n", (int) rl.rlim_cur, filemax);
return 1;
}

return 0;
}

int
main (int argc, char **argv)
{
struct event* evsocket = new struct event;
int socket;

setlimit(60000);

ChannelPtr channel = new Channel(argc, argv);
Ice::ObjectAdapterPtr adapter = channel->getCommunicator()
->createObjectAdapterWithEndpoints("Server", ":tcp -p 8889");
adapter->add(new HelloI, Ice::stringToIdentity("Hello"));
adapter->activate();

cout << "Adapter initialized" << endl;
/* step1: create a socket */
if ((socket = ::socket(AF_INET, SOCK_STREAM, 0)) < 0) {
cout << "socket: " << strerror(errno) << endl;
cout.flush();
return -1;
}
cout << "Socket created" << endl;
int value = 1;
socklen_t size = sizeof(value);
if (setsockopt(socket, SOL_SOCKET, SO_REUSEADDR, &value, size) < 0) {
cout << "setsockopt: " << strerror(errno) << endl;
return -1;
}

struct sockaddr_in addr;
addr.sin_addr.s_addr = INADDR_ANY;
addr.sin_family = AF_INET;
addr.sin_port = htons(8888);
if (bind(socket, (struct sockaddr*)&addr, sizeof(struct sockaddr)) < 0) {
cout << "bind: " << strerror(errno) << endl;
return -1;
}

if (listen (socket, 2000) < 0) {
cout << "listen: " << strerror(errno) << endl;
return -1;
}

/* Initalize the event library */
event_init();

event_set(evsocket, socket, EV_READ, socket_accept, evsocket);

/* Add it to the active events, without a timeout */
event_add(evsocket, NULL);

event_dispatch();

return (0);
}

[matthew]

Without detailed knowledge of your application its basically impossible to determine what exactly is happening. This is certainly nothing that is Ice specific. If you require more help with this problem please contact sales@zeroc.com to enquire about our consulting services.

[kent]

The above is Server.cpp, it uses libevent to dispatch events.
If i change socket_accept like this:

void
socket_accept(int socket, short event, void* arg)
{
struct event *ev = (struct event*)arg;
/* Reschedule this event */
event_add(ev, NULL);

int fd = accept(socket, NULL, NULL);
if (fd < 0) {
cout << "accept: " << strerror(errno) << endl;
return;
}
int newfd = fcntl(fd, F_DUPFD, 1000);
if (newfd <= 0) {
cout << "cannot move fd: " << fd << endl;
continue;
} else {
close(fd);
fd = newfd;
}
struct event* evsocket = new struct event;
event_set(evsocket, fd, EV_READ, socket_read, evsocket);

/* Add it to the active events, without a timeout */
event_add(evsocket, NULL);
}


Force the new incoming connection's fd > 1000 to reserve the lower fd for Ice, there's no problem

[matthew]

You may be running into a kernel limitation on the number of file descriptors that select will support. You should look at your kernel configuration to determine what the hard limit is. At any rate, if you opt to use our consulting services we can look into this in more detail.

[kent]

I think that is not the reason, because Ice only establishes 1 connection, and select a few file descriptors in Ice.ThreadPool. The other lots of connection established by epoll have no limits, and we have set max connection limit to 60000. The problem occurs only if Ice's connection's fd number is larger than 1000(or other number). Like the message I have post above, we have tested it is the key.

I suggest that you will make a same test.

[kent]

I look into ice/ThreadPool.cpp, the following codes are the problem source that puzzle me, please give some suggestion.

//
// Round robin for the filedescriptors.
//
if(_lastFd < _minFd - 1 || _lastFd == INVALID_SOCKET)
{
_lastFd = _minFd - 1;
}

int loops = 0;
do
{
if(++_lastFd > _maxFd)
{
++loops;
_lastFd = _minFd;
}
}
while(!FD_ISSET(_lastFd, &fdSet) && loops <= 1);

if(loops > 1)
{
Error out(_instance->logger());
out << "select() in `" << _prefix << "' returned " << ret
<< " but no filedescriptor is readable";
continue;
}

[marc]

I'm sorry, but explaining Ice internals is beyond the free support we can provide here in these forums, and so is helping to debug applications that do their own non-Ice network communications. As Matthew said, we can provide you with consulting services to help you solve your problem. Please contact us at info@zeroc.com if you are interested.