Glacier2 will block all session with callback,when a session hung.

[netfish]

Glacier2's session blocking problem.

Hardware: cpu p4,memory 512M
Software: Windows 2000,Ice3.2.1,ANT
I have create a project which is based on ice3.2.1/demoj/Glacier2/CallBack example,to make sure Icestorm can work with Glacier.In my test project,in order to pass firewall's NAT, icestorm publishs messages to subscriber with a callback proxy object ,which is provided by subscriber before.This is the same as Glacier2/CallBack example.Below is my project diagram .

Subscribers----Clacier2----IceStorm----Publisher

My problem is : Glacier2 can send all messages to a lot of subscribers successfully and quickly,But if a subscriber hungs,Glacier2 sends messages to other subscribers more and more slowly,and finally,Clacier2 will not sending any messages.
Does Glacier2 has solution to send messages to other subscribers,even when some subscribers hung or blocked?

To Test my problem:

1)In a separate window:

$ glacier2router --Ice.Config=config.glacier2


2) Start the IceStorm service:

$ icebox --Ice.Config=config.icebox


3) In a separate window,starting two subscriber

$ ant run-Subscriber
$ ant run-Subscriber

4) In another window

$ ant run-Publisher

5)In another window, starting a subscriber that will hung.
$ ant run-BlockSubscriber

I enclose all my codes and config below.

================================================== ============================
// Clock.ice
================================================== ===========================
#ifndef CLOCK_ICE
#define CLOCK_ICE

module Demo
{

struct Structure
{
int index;
string name;

};


interface Clock
{
void tick(string time);
void tick2(Structure ct);
};

};

#endif



================================================== ============================
// config.glacier2
================================================== ===========================
Glacier2.InstanceName=DemoGlacier2
Ice.ThreadPerConnection.StackSize=262144
Glacier2.Client.Endpoints=tcp -p 10005

Glacier2.Server.Endpoints=tcp
Glacier2.PermissionsVerifier=DemoGlacier2/NullPermissionsVerifier

Glacier2.Client.Buffered=0
Glacier2.Server.Buffered=0

Glacier2.Client.ForwardContext=1
Glacier2.Server.ForwardContext=1

Glacier2.Client.SleepTime=500
Glacier2.Server.SleepTime=500

Glacier2.Client.AlwaysBatch=0
Glacier2.Server.AlwaysBatch=0

Glacier2.Client.Trace.Request=1
Glacier2.Server.Trace.Request=1
#Glacier2.Client.Trace.Override=1
#Glacier2.Server.Trace.Override=1
Glacier2.Client.Trace.Reject=1
Glacier2.Trace.Session=1
Glacier2.Trace.RoutingTable=1

Ice.Warn.Connections=1

Ice.Trace.Protocol=1
================================================== ============================
// config.sub.glacier2
================================================== ===========================
Ice.Default.Router=DemoGlacier2/router:tcp -p 10005 -h 127.0.0.1

Clock.Subscriber.Router=DemoGlacier2/router:tcp -p 10005 -h 127.0.0.1

Clock.Subscriber.Endpoints=tcp


Ice.ACM.Client=0
Ice.ACM.Server=0

Ice.MonitorConnections=60


Ice.RetryIntervals=-1


Ice.Warn.Connections=1
IceStorm.TopicManager.Proxy=DemoIceStorm/TopicManager:default -p 10000
IceStorm.Trace.Subscriber=2
================================================== ===========================
// config.icebox
================================================== ===========================
IceBox.ServiceManager.Endpoints=tcp -p 9998

IceBox.Service.IceStorm=IceStormService,32:createI ceStorm --Ice.Config=config.service

Ice.Trace.Network=3
Ice.Trace.Protocol=1

#IceStorm.SubscriberPool.Size=10
#IceStorm.SubscriberPool.SizeMax=100
#IceStorm.SubscriberPool.SizeWarn=50
#IceStorm.SubscriberPool.Timeout=5

[benoit]

Hi,

By just looking at the Glacier2 configuration, it's very likely that the problem is caused by using the non-buffered mode. In the Glacier2 configuration file you should try setting the Glacier2.Client.Buffered and Glacier2.Server.Buffered properties to 1. See here in the Ice manual for more information on Glacier buffered mode.

Cheers,
Benoit.

[netfish]

If I use Glacier2's Buffered and Batch configuration as below, i find Subscriber can get messages but very slow,and will lost message.Because
i want to send and receive message quickly, Do you have other solution while not use batch or buffer model?


Glacier2.Client.Buffered=1
Glacier2.Server.Buffered=1
Glacier2.Client.AlwaysBatch=1
Glacier2.Server.AlwaysBatch=1

[netfish]

In my project, publisher call subscriber's callback object frequently. I find
all subscribers which do not hung will get messages promptly at first and then more and more slow if i hung a subscriber, Does os's socket buffer affect Glacier2? or Does Glacier has some special way for frequently callback invoke?

[benoit]

Hi,

Unless you use UDP, messages shouldn't be dropped when using buffered mode. It's also not clear to me why it would be slower when using the buffered mode. How many clients are connected to Glacier2 and how many messages are forwarded through Glacier2?

In unbuffered mode, a client can affect the message delivery of other clients for several reasons:

• Glacier2 is forwarding a twoway invocation and the client doesn't send the response in a timely manner.
• The socket TCP/IP send buffer of the client connection is full (see also this FAQ)

You should definitely use the buffered mode if you don't want a client to affect the message delivery of other clients.

Note that this won't be the case anymore with Ice 3.3.0. With Glacier2 from Ice 3.3.0, the unbuffered or buffered mode will offer the same guarantees in this respect, that is, a slow client or misbehaving client won't affect the delivery of messages to other clients. The buffered mode will still be useful to allow the batching of oneway messages (which can improve throughput: it's more efficient to send a batch of N oneway messages than N oneway messages separately).

Cheers,
Benoit.

[netfish]

Thanks Benoit! You are right!
In my project ,the reason why it would be slower is i set batch config as below. Moreover, my publisher publishs messages very very frequently,just for testing.
Glacier2.Client.Buffered=1
Glacier2.Server.Buffered=1
Glacier2.Client.AlwaysBatch=1
Glacier2.Server.AlwaysBatch=1

if i turn off batch config ,and doesnot publish so frequently,then Glacier2 can send message quickly.

thanks again.

[netfish]

Thanks !
this problem has been solved!