A question about ChatRoom in Connections Issue #2

[rc_hz]

The following words is quoted from Connections Issue #2:

1)The server’s invocation of message is delivered in two steps: the server sends the invocation to the Glacier2 router, and then the router sends the invocation to the chat client.

2)the recommended configuration for the chat server is to use a twoway message to the Glacier2 router. This provides maximum reliability and does not cause problems for the message implementation because Glacier2 is always well-behaved
and does not block the server for any length of time. Ideally we should not hold the lock inside message for the duration of the RPC (to allow for better concurrency), but we will address this in a future article

3)For the communication between Glacier2 and the chat clients, we’ll use oneway messages. This provides maximum throughput and avoids problems due to network latency or malicious clients. In addition, the Glacier2 router must use buffered mode for server-to-client calls.

4)The client’s local TCP/IP stack has a limited amount of buffer space to accept data. If a oneway request is too large to fitintothe remaining TCP/IP buffer space, the kernel suspends the caller in its write system call on the socket until enough buffer space becomes available. The remaining buffer space can be consumed by previously buffered requests or even by a single request, if it is large enough.

My question: If the one-way message from Glacier2 to client is blocked, then the two-way message from Server to Glacier2 has to wait, so I think this is also a problem.

[benoit]

Hi,

No this can't happen if you use Glacier2 buffered mode, even if the oneway blocks the twoway call to the router won't block thanks to Glacier2 buffered mode. Please see the manual for more information.

Benoit.

[rc_hz]

Thank you. I have read this chapter about Glacier2, but I still think that buffered mode just cached the two-way call from Server, and that the calling thread in Server will be blocked until the response is returned from Glacier2, which is dependent on the one-way call from Glacier2 to Client. Could you explain a little more ? Or do you mean that once Glacier2 has cached the two-way request from Server, then give a response to Server immediately ? If so, it's of course OK.

[marc]

Yes, the response to the server will be given immediately, because the call to the client is oneway.

[benoit]

That's correct, the twoway call will return as soon as the call is queued in Glacier2. This is of course possible only because the call is forwarded to the client as a oneway call and there's no need to wait for a response from the client.

Benoit.

[rc_hz]

Thanks, I know now. So if the invocation from Glacier2 to client is a two-way one , then even if Glacier2 uses buffered mode, the calling thread in Server will still be blocked until the invocation from Glacier2 to client has returned. Is it right?

If so, another question . Please check these two cases first:
Case 1: 5 threads in Client send two-way requests to Server simultaneously. Server use thread-pool dispatch mode(the pool is large enough). So in the Server side, there are also 5 threads to these requests simultaneously.

Case 2: 5 threads in Client send two-way requests to Server by Glacier2. Suppose requestss from Client to Glacier2 and from Glacier2 to Server are both two-ways. Glacier2 is in buffered mode. According the Ice's manul:

In buffered mode, the router queues incoming requests and creates an extra thread (or two) dedicated to processing the request queue of each connected client.

According the quoting, there are two threads in Glacier2 to serve for the Client: one for receiving requests from Client and caching them, the other for taking out request and sending it to Server. As the result, there is only one thread at a time in Server to process the request.

So, if we want that there are also 5 threads to process Client's requests simultaneously, what we should do?

[benoit]

You don't need to do anything, there will be 5 threads in your server (assuming it's properly configured) to process the 5 client requests.

Without getting too much into details... Glacier2 is using AMI/AMD to forward twoway invocations: the thread that forwards the request to the server doesn't need to wait for the answer to process another request. I'm afraid I can't give you much more details on the implementation of Glacier2, this is a bit ouf the scope of the support we can give on the forums. I recommend taking a look at Glacier2 source code for more details.

Benoit.

[rc_hz]

...I'm afraid I can't give you much more details on the implementation of Glacier2, this is a bit ouf the scope of the support we can give on the forums. I recommend taking a look at Glacier2 source code for more details.

Thank you! I will see the Glacier2's source code tomorrow.

BTW, I think that readers can not find the above questions' answer by just reading Ice's manul. So can you improve it ?

[matthew]

I'm not sure what you want to improve in the manual... if you send a one way request then the server does not need to wait for a reply. That's the whole idea, after all

Regards, Matthew

[rc_hz]

You don't need to do anything, there will be 5 threads in your server (assuming it's properly configured) to process the 5 client requests.

Without getting too much into details... Glacier2 is using AMI/AMD to forward twoway invocations: the thread that forwards the request to the server doesn't need to wait for the answer to process another request.

I am sorry that I have to bother you again. I have read part of Glacier2's source code , but I don't think that Glacier2 has used AMI to forward request to Server:

-- src/Glacier2/RequestQueue.cpp
void
Glacier2::Request::invoke()
{
    bool ok;
    ByteSeq outParams;
    
    try
    {
		if(_forwardContext)
		{  //This is not a AMI invocation
		    ok = _proxy->ice_invoke(_current.operation, _current.mode, _inParams, outParams, _current.ctx);
		}
		else
		{
		    ok = _proxy->ice_invoke(_current.operation, _current.mode, _inParams, outParams);
		}
		
		if(_proxy->ice_isTwoway())
		{
		    _amdCB->ice_response(ok, outParams);
		}
    }
    catch(const LocalException& ex)
    {
		if(_proxy->ice_isTwoway())
		{
		    _amdCB->ice_exception(ex);
		}
    }
}

[marc]

Congratulations, you have discovered a bug

We have also recently discovered this bug, and it is fixed for 2.1.2. The bug only shows up if the server sends two callbacks at the same time, and one of them blocks (for example, because the first callback triggers the second callback and waits for its completion, i.e., nested callbacks).

namespace Glacier2
{

class AMI_Object_ice_invokeI : public AMI_Object_ice_invoke
{
public:

    AMI_Object_ice_invokeI(const AMD_Object_ice_invokePtr& amdCB) :
	_amdCB(amdCB)
    {
	assert(_amdCB);
    }

    virtual void
    ice_response(bool ok, const std::vector<Byte>& outParams)
    {
	_amdCB->ice_response(ok, outParams);
    }

    virtual void
    ice_exception(const Exception& ex)
    {
	_amdCB->ice_exception(ex);
    }

private:

    const AMD_Object_ice_invokePtr _amdCB;
};

}

// ...

void
Glacier2::Request::invoke()
{
    if(_proxy->ice_isTwoway())
    {
	AMI_Object_ice_invokePtr cb = new AMI_Object_ice_invokeI(_amdCB);
	if(_forwardContext)
	{
	    _proxy->ice_invoke_async(cb, _current.operation, _current.mode, _inParams, _current.ctx);
	}
	else
	{
	    _proxy->ice_invoke_async(cb, _current.operation, _current.mode, _inParams);
	}
    }
    else
    {
	try
	{
	    ByteSeq outParams;
	    if(_forwardContext)
	    {
		_proxy->ice_invoke(_current.operation, _current.mode, _inParams, outParams, _current.ctx);
	    }
	    else
	    {
		_proxy->ice_invoke(_current.operation, _current.mode, _inParams, outParams);
	    }
	}
	catch(const LocalException&)
	{
	}
    }
}

[rc_hz]

Another question: A serverProxy is created in the following code. I know that its Identity's category is a flag , so it demands that different serverProxy should have a different catetory, but the algorithm can not promise this. Why not choose something else like generateUUID() ? Thanks.

Glacier2::RouterI::RouterI(const ObjectAdapterPtr& clientAdapter, const ObjectAdapterPtr& serverAdapter,
			   const ConnectionPtr& connection, const string& userId, const SessionPrx& session) :
//...
{
	//...

    if(serverAdapter)
    {
		ObjectPrx& serverProxy = const_cast<ObjectPrx&>(_serverProxy);
		Identity ident;
		ident.name = "dummy";
		ident.category.resize(20);
		for(string::iterator p = ident.category.begin(); p != ident.category.end(); ++p)
		{
		    *p = static_cast<char>(33 + rand() % (127-33)); // We use ASCII 33-126 (from ! to ~, w/o space).
		}		
		serverProxy = serverAdapter->createProxy(ident);
		
		ServerBlobjectPtr& serverBlobject = const_cast<ServerBlobjectPtr&>(_serverBlobject);
		serverBlobject = new ServerBlobject(_communicator, _connection);
    }
}

[marc]

The algorithm creates a 20-character random string. The probability of two equivalent strings being created within the same Glacier instance is so small, that it's safe to say that these strings are guaranteed to be unique. It is certainly not less safe than using a UUID.

[rc_hz]

Congratulations, you have discovered a bug

We have also recently discovered this bug, and it is fixed for 2.1.2. The bug only shows up if the server sends two callbacks at the same time, and one of them blocks (for example, because the first callback triggers the second callback and waits for its completion, i.e., nested callbacks).

namespace Glacier2
{

class AMI_Object_ice_invokeI : public AMI_Object_ice_invoke
{
public:

    AMI_Object_ice_invokeI(const AMD_Object_ice_invokePtr& amdCB) :
	_amdCB(amdCB)
    {
	assert(_amdCB);
    }

    virtual void
    ice_response(bool ok, const std::vector<Byte>& outParams)
    {
	_amdCB->ice_response(ok, outParams);
    }

    virtual void
    ice_exception(const Exception& ex)
    {
	_amdCB->ice_exception(ex);
    }

private:

    const AMD_Object_ice_invokePtr _amdCB;
};

}

// ...

void
Glacier2::Request::invoke()
{
    if(_proxy->ice_isTwoway())
    {
	AMI_Object_ice_invokePtr cb = new AMI_Object_ice_invokeI(_amdCB);
	if(_forwardContext)
	{
	    _proxy->ice_invoke_async(cb, _current.operation, _current.mode, _inParams, _current.ctx);
	}
	else
	{
	    _proxy->ice_invoke_async(cb, _current.operation, _current.mode, _inParams);
	}
    }
    else
    {
	try
	{
	    ByteSeq outParams;
	    if(_forwardContext)
	    {
		_proxy->ice_invoke(_current.operation, _current.mode, _inParams, outParams, _current.ctx);
	    }
	    else
	    {
		_proxy->ice_invoke(_current.operation, _current.mode, _inParams, outParams);
	    }
	}
	catch(const LocalException&)
	{
	}
    }
}

AMI calls can also be blocked just like one-way calls if the Server runs very slowly, is this right ? If so, there may be some other problem in Glacier2 model.

[marc]

I'm not sure I understand the question. All calls can block if the server doesn't respond, this is nothing specific to Glacier2. The same would happen if you would call the server directly. To avoid this, you typically set timeouts.