Application add is very slow, what happened?

[eaglecn]

Hi,
We use icegrid to manage several applications, find that adding some an application is slow. So what happened when adding an application or how can we trace these activities?
thanks!

[benoit]

Hi,

You can add the following traces on the registry:

IceGrid.Registry.Trace.Server=3
IceGrid.Registry.Trace.Object=1
IceGrid.Registry.Trace.Adapter=1

This should trace the newly added objects, adapters and servers and the deployment of each server on the nodes.

Cheers,
Benoit.

[eaglecn]

I add cout code to function of "IceProxy::IceGrid::Admin::addApplication" in file of Ice-3.2.1/src/IceGrid/Admin.cpp just now as following,

            cout<<"before checktwoway=="<<endl;
            __checkTwowayOnly(__IceGrid__Admin__addApplication_name);
            cout<<"before getDelegate=="<<endl;
            __delBase = __getDelegate();
            cout<<"before dynamic_cast=="<<endl;
            ::IceDelegate::IceGrid::Admin* __del = dynamic_cast< ::IceDelegate::IceGrid::Admin*>(__delBase.get());
            cout<<"before _del->addApp=="<<endl;
            __del->addApplication(descriptor, __ctx);

and find no exception.

After tried several times, the same symptom reappears, namely, the add application command line is hanging.

this time, it reports
"IceGrid.Registry.Client.ThreadPool' is running low on threads" frequently,

I got no idea with this parameter.
after checking, the registry process has more than 18 threads.

where is wrong?

[benoit]

Sorry, without more information it's impossible to say what could be wrong. Did you try adding the tracing suggested in my previous post? Can you copy/paste the tracing output here?

Cheers,
Benoit.

[eaglecn]

The following info is all I got from a rsh terminal, is it enough?
the last application need 4 minutes to be loaded, while others almost need no time.

[ 01/13/09 17:03:58.078 Adapter: added adapter `ddl_yf-softnode3.DDLAdapter' ]
[ 01/13/09 17:03:58.079 Object: added object `ddl_yf-softnode3' ]
[ 01/13/09 17:03:58.079 Server: added server `ddl_yf-softnode3' (`2C521B8F-2B06-4146-A121-4A128D8B8903', `1') ]
[ 01/13/09 17:03:58.079 Server: loading `ddl_yf-softnode3' on node `yf-softnode3' ]
[ 01/13/09 17:03:58.081 Server: loaded `ddl_yf-softnode3' on node `yf-softnode3' ]
before checktwoway==
before getDelegate==
before dynamic_cast==
before _del->addApp==
[ 01/13/09 17:03:58.117 Adapter: added adapter `load_YF-softnode4.LoadAdapter' ]
[ 01/13/09 17:03:58.117 Object: added object `load_YF-softnode4' ]
[ 01/13/09 17:03:58.117 Server: added server `load_YF-softnode4' (`95E14DCE-AF6B-47D0-9515-BB0F952E0903', `1') ]
[ 01/13/09 17:03:58.117 Server: loading `load_YF-softnode4' on node `YF-softnode4' ]
[oracle@YF-softnode4 script]$ [ 01/13/09 17:07:07.092 Server: loaded `load_YF-softnode4' on node `YF-softnode4' ]

and only this application locates at the other server

[benoit]

From the traces, it appears that the server loading on YF-softnode4 hangs since you're not getting the "Server: loaded `load_YF-softnode4' on node `YF-softnode4'" message. Is the host where YF-softnode4 runs accessible from the registry host?

Also, you should add timeouts to the IceGrid registry and node endpoints to avoid requests to block for too long if something goes wrong. See the configuration files from cpp/demo/IceGrid/replication for an example on how to set timeouts.

Cheers,
Benoit.

[eaglecn]

these machines can access each other and not all loading of this application are not successful. In fact, adding process is slow but can be done finally.

So, I am wonder during the hanging, what operation is ongoing? Is it just try to accessing remote server?

Why other applications never have this problem? Processing properites in program has relation with it? But I think no processes are to be started during that time.

[eaglecn]

I got the following message when removing this application,

[ 01/13/09 17:21:51.357 Server: couldn't unload `load_YF-softnode4' on node `YF-softnode4':
  Network.cpp:624: Ice::ConnectTimeoutException:
  timeout while establishing a connection ]

[ 01/13/09 17:53:43.501 Server: couldn't unload `load_YF-softnode4' on node `YF-softnode4':
  Network.cpp:664: Ice::ConnectionRefusedException:
  connection refused: Connection refused ]

why it is timout to establish a connection?

[benoit]

For some reasons, the registry can't contact the node YF-softnode4. You could try adding network tracing on the IceGrid registry with Ice.Trace.Network=2 to figure out on which IP address the registry tries to contact the node.

If you didn't specify any host with the -h HOST option in the IceGrid node endpoints, it's possible that the registry tries to connect to an unreachable IP address if the host where the node is running has multiple network interfaces.

You can try to specify explicitly the IP address in the node endpoints, for example:

   IceGrid.Node.Endpoints=tcp -h 192.168.1.2

See 28.4.6 Endpoints in the Ice manual for more information on object adapter endpoints.

Cheers,
Benoit.

[eaglecn]

I believe the problem is solved after i set the icegridnode configure file.
thanks a lot!