Slow grid start - node seems to be unreachable

[jesse]

I added the proper host endpoint to the Icestorm config. Here is the new one:

#
# This property is used by the administrative client to connect to IceStorm.
#
# IceStorm.TopicManager.Proxy=BDR/TopicManager:default -p 20000

#
# This property defines the endpoints on which the IceStorm
# TopicManager listens.
#
IceStorm.TopicManager.Endpoints=default -p 20000 -h 10.110.3.60

#
# The IceStorm service instance name.
#
IceStorm.InstanceName=BDR

#
# This property defines the endpoints on which the topic
# publisher objects listen. If you want to federate
# IceStorm instances this must run on a fixed port (or use
# IceGrid).
#
IceStorm.Publish.Endpoints=default -p 20001:udp -p 20001 -h 10.110.3.60

#
# TopicManager Tracing
#
# 0 = no tracing
# 1 = trace topic creation, subscription, unsubscription
# 2 = like 1, but with more detailed subscription information
#
IceStorm.Trace.TopicManager=2

Ice.Default.Locator=BDR/Locator:default -p 12000 -h 10.110.3.60

#
# Topic Tracing
#
# 0 = no tracing
# 1 = trace unsubscription diagnostics
#
IceStorm.Trace.Topic=1

#
# Subscriber Tracing
#
# 0 = no tracing
# 1 = subscriber diagnostics (subscription, unsubscription, event
#     propagation failures)
#
IceStorm.Trace.Subscriber=1

#
# Flush Tracing (for batch mode transfer flushing)
#
# 0 = no tracing
# 1 = trace activity of flusher thread
#
IceStorm.Trace.Flush=1

#
# Amount of time in milliseconds between flushes for batch mode
# transfer. The minimum allowable value is 100ms.
#
IceStorm.Flush.Timeout=2000

#
# Network Tracing
#
# 0 = no network tracing
# 1 = trace connection establishment and closure
# 2 = like 1, but more detailed
# 3 = like 2, but also trace data transfer
#
# Ice.Trace.Network=2

#
# This property defines the home directory of the Freeze 
# database environment for the IceStorm service.
#
Freeze.DbEnv.IceStorm.DbHome=/usr/bright/apps/bdr/db/icestorm

The client is hanging on the following line:

IceStorm.TopicManagerPrx manager = IceStorm.TopicManagerPrxHelper.checkedCast( Ice.Application.communicator()
				.propertyToProxy( "IceStorm.TopicManager.Proxy" ) );

[benoit]

Ok, so the proxy causing the hang is "IceStorm/TopicManager" (the value of the IceStorm.TopicManager.Proxy property in the client configuration file).

This is a well-known proxy. How did you register this well-known proxy with the IceGrid registry? I suppose you used the icegridadmin "object add" command or the IceGrid GUI since IceStorm isn't managed by an IceGrid node, is this correct?

If this is the case, most likely, the -h option for the well-known proxy hasn't been specified when it was added to the IceGrid registry. You can check the endpoints of this well-known proxy with the "object describe" command or the IceGrid administrative GUI. Updating the endpoints of this well-known proxy to include the -h option should fix the hang.

Cheers,
Benoit.

[jesse]

I am starting the IceStorm server in my application.xml.

Here is the line:

<server-instance template="IceStorm" instance-name="IceStorm">
      </server-instance>

I tried adding the endpoint property. It didn't help.

<server-instance template="IceStorm" instance-name="IceStorm">
      	<properties>
    		<property name="IceStorm.Publish.Endpoints" value="default -p 20001:udp -p 20001 -h 10.110.3.60"/>
		</properties>
      </server-instance>

[benoit]

Hi Jesse,

I'm afraid I don't understand your deployment. I thought you were not deploying IceStorm on an IceGrid node with a deployment descriptor but instead was manually starting IceStorm from the command line using a configuration file. If you're indeed using a deployment descriptor to deploy IceStorm on an IceGrid node, you need to update the XML file for configuration changes (and run the icegridadmin "application update" command after each update). Could post your application.xml file? I suspect the problem is coming from the definition of the "endpoints" attribute for the TopicManager adapter (missing -h option).

Cheers,
Benoit.

[jesse]

I am starting IceStorm automatically via IceGrid. Here is my application.xml:

<icegrid>
  <application name="BDR" import-default-templates="true">
    <node name="localhost">
      <server id="BDRServer" exe="/usr/bright/apps/bdr_server_start.sh" activation="always" user="root">
		<adapter name="CpService" endpoints="tcp -h 10.110.3.60" register-process="true">
		  <object identity="BDRCpService" type="::BDRCp::BDRCpService" property="Identity"/>
		</adapter>
		<adapter name="CustomerInfoService" endpoints="tcp -h 10.110.3.60" register-process="true">
		  <object identity="BDRCustomerInfoService" type="::BDRCustomerInfo::BDRCustomerInfoService" property="Identity"/>
		</adapter>
		<adapter name="DriveInfoService" endpoints="tcp -h 10.110.3.60" register-process="true">
		  <object identity="BDRDriveInfoService" type="::BDRDriveInfo::BDRDriveInfoService" property="Identity"/>
		</adapter>
		<adapter name="CustomerInitService" endpoints="tcp -h 10.110.3.60" register-process="true">
		  <object identity="BDRCustomerInitService" type="::BDRCustomerInit::BDRCustomerInitService" property="Identity"/>
		</adapter>
		<adapter name="MountService" endpoints="tcp -h 10.110.3.60" register-process="true">
		  <object identity="BDRMountService" type="::BDRMount::BDRMountService" property="Identity"/>
		</adapter>
		<adapter name="UnmountService" endpoints="tcp -h 10.110.3.60" register-process="true">
		  <object identity="BDRUnmountService" type="::BDRUnmount::BDRUnmountService" property="Identity"/>
		</adapter>
      </server>
      
      <server-instance template="IceStorm" instance-name="IceStorm">
      	<properties>
    		<property name="IceStorm.Publish.Endpoints" value="default -p 20001:udp -p 20001 -h 10.110.3.60"/>
    		<property name="IceStorm.TopicManager.Endpoints" value="default -p 20000 -h 10.110.3.60"/>
		</properties>
      </server-instance>
    </node>
  </application>
</icegrid>

I have already ran the "appliation update" after updating my application.xml.

[benoit]

The properties defined in the <server-instance> element are properties that will be defined in the IceBox server configuration file not the IceStorm service configuration file and will therefore have no effect on the IceStorm service.

You can simply use the following instead:

      <server-instance template="IceStorm" instance-name="IceStorm" topic-manager-endpoints="default -p 20000 -h 10.110.3.60" publish-endpoints="default -p 20001:udp -p 20001 -h 10.110.3.60"/>

Or:

     <server-instance template="IceStorm" instance-name="IceStorm">
       	 <properties service="IceStorm">
    		<property name="IceStorm.Publish.Endpoints" value="default -p 20001:udp -p 20001 -h 10.110.3.60"/>
    		<property name="IceStorm.TopicManager.Endpoints" value="default -p 20000 -h 10.110.3.60"/>
         </properties>
      </server-instance>

Make sure to run "application update" on the updated descriptor to push the changes to the IceGrid registry.

You can also use the "adapter endpoints IceStorm.TopicManager" command to check the runtime endpoints of the topic manager adapter once the IceBox server is started. In theory, it should only contain the 10.110.3.60 IP address.

Cheers,
Benoit.

[jesse]

It looks like that was the issue. Thanks for your help.