Glacier2.Client.Endpoints, SSL, and ICE-E

[bartley]

We've run across a strange (to us, at least) problem when using clients compiled for Ice-E for C++ to connect to a Glacier server that's listening for both TCP and SSL connections (I'm aware that Ice-E doesn't support SSL, thus the need for our Glacier server to listen for TCP).

Here's our Glacier2 config file:

Glacier2.InstanceName=TerkGlacier

#
# We must set the stack size of new threads created by Glacier2. The
# default on Linux is typically in the 10MB range, which is way too
# high.
#
# Since Glacier2 always uses thread-per-connection mode, we must use
# the property below to set the thread stack size. Internal Glacier2
# threads also use this property value.
#
# The Ice manuals says that only the C++ run time uses this property.
#
Ice.ThreadPerConnection.StackSize=262144

#
# The client-visible endpoint of Glacier2. This should be an endpoint
# visible from the public Internet, and it should be secure.  This is
# where requests from routed proxies are sent.
#
Glacier2.Client.Endpoints=ssl -h glacier.terk.ri.cmu.edu -p 10005:tcp -h glacier.terk.ri.cmu.edu -p 10004

#
# The server-visible endpoint of Glacier2. This endpoint is only
# required if callbacks are needed (leave empty otherwise). This
# should be an endpoint on an internal network (like 192.168.x.x), or
# on the loopback, so that the server is not directly accessible from
# the Internet.
#
Glacier2.Server.Endpoints=tcp -h 127.0.0.1

#
# This configures the session manager. If no external session manager
# is used, sessions are only handled by Glacier2 internally.
#
Glacier2.SessionManager=TerkSessionManager:tcp -h 127.0.0.1 -p 10001

#
# The permissions verifier is collocated with the session server process.
#
Glacier2.PermissionsVerifier=TerkPermissionsVerifier:tcp -h 127.0.0.1 -p 10001

#
# The timeout for inactive sessions. If any client session is inactive
# for longer than this value, the session expires and is removed. The
# unit is seconds.
#
Glacier2.SessionTimeout=60

#
# We don't want to allow access to any server back-end objects
# that do not have the _userId as their category.
#
Glacier2.AddUserToAllowCategories=2

#
# Glacier can forward requests buffered or unbuffered. Unbuffered
# means a lower resource consumption, as buffering requires one
# additional thread per connected client or server. However, without
# buffering, messages cannot be batched and message overriding doesn't
# work either. Also, with unbuffered request forwarding, the caller
# thread blocks for twoway requests.
#
Glacier2.Client.Buffered=1
Glacier2.Server.Buffered=1

#
# These two lines instruct Glacier2 to forward contexts both for
# regular routing, as well as for callbacks (reverse routing).
#
Glacier2.Client.ForwardContext=1
Glacier2.Server.ForwardContext=1

#
# To prevent Glacier2 from being flooded with requests from or to one
# particular client, Glacier2 can be configured to sleep for a certain
# period after all current requests for this client have been
# forwarded. During this sleep period, new requests for the client are
# queued. These requests are then all sent once the sleep period is
# over. The unit is milliseconds.
#
Glacier2.Client.SleepTime=10
Glacier2.Server.SleepTime=10

#
# With the two settings below, Glacier2 can be instructed to always
# batch oneways, even if they are sent with a _fwd/o instead of a
# _fwd/O context.
#
Glacier2.Client.AlwaysBatch=0
Glacier2.Server.AlwaysBatch=0

#
# Glacier2 always disables active connection management so there is no
# need to configure this manually. Connection retry does not need to
# be disabled, as it's safe for Glacier2 to retry outgoing connections
# to servers. Retry for incoming connections from clients must be
# disabled in the clients.  NOTE: Actually, they don't have to be, and
# can actually provide a work around to a bug.  See here for more:
# http://www.zeroc.com/vbulletin/showpost.php?p=8319&postcount=4
#

#
# Various settings to trace requests, overrides, etc.
#
Glacier2.Client.Trace.Request=1
Glacier2.Server.Trace.Request=1
Glacier2.Client.Trace.Override=1
Glacier2.Server.Trace.Override=1
Glacier2.Client.Trace.Reject=1
Glacier2.Trace.Session=1
Glacier2.Trace.RoutingTable=1

#
# Other settings.
#

Ice.Trace.Network=1
Ice.Trace.Protocol=1

# Print warning messages for certain exceptional conditions in connections
Ice.Warn.Connections=1


#
# SSL Settings.
#
#IceSSL.Trace.Security=1
Ice.Plugin.IceSSL=IceSSL:create
IceSSL.Server.CertPath=./ssl
IceSSL.Server.Config=sslconfig.xml
IceSSL.Client.CertPath=./ssl
IceSSL.Client.Config=sslconfig.xml

We're not doing any port forwarding, so, if I understand correctly, there should be no need for defining PublishedEnpoints.

We've found that if Glacier2.Client.Endpoints is set as shown above (both SSL and TCP), then clients compiled for Ice-E for C++ die with this error (I believe it's somewhere in the call to createObjectAdapter()):

helloTest: ../../src/TcpTransport/TcpEndpoint.cpp:279: virtual bool IceInternal::TcpEndpoint::operator<(const IceInternal::Endpoint&) const: Assertion `p' failed.
Aborted

Clients compiled for Ice for Java work fine. We haven't had a chance to try clients compiled for Ice for C++ yet.

The problem goes away if we simply remove the SSL clause from the Glacier2.Client.Endpoints property, like this:

Glacier2.Client.Endpoints=tcp -h glacier.terk.ri.cmu.edu -p 10004

Am I not defining Glacier2.Client.Endpoints properly? Any other ideas?

many thanks,

chris

[mes]

Hi Chris,

The presence of the Glacier2.Client.Endpoints property shouldn't have any adverse affect on an Ice-E client. Are you perhaps including an SSL endpoint in the Ice.Default.Router property defined in the client?

If you're still having trouble, it might be useful for us to see your client's configuration.

Take care,
- Mark

[benoit]

Hi Chris,

I've posted a patch here that should fix the assert you mentioned in your post. This should help solving your problem.

Cheers,
Benoit.

[bartley]

Mark wrote:

The presence of the Glacier2.Client.Endpoints property shouldn't have any adverse affect on an Ice-E client. Are you perhaps including an SSL endpoint in the Ice.Default.Router property defined in the client?

If you're still having trouble, it might be useful for us to see your client's configuration.

To be clear, it's not really the presence of the Glacier2.Client.Endpoints property that's causing the problem, but rather the value. When the value contains an only a TCP endpoint, everything works fine. But when we also include the SSL endpoint, the client dies. Weird.

Here's the config file that the client is using:

Ice.ProgramName=RobotClient

Ice.Package.peer=edu.cmu.ri.mrpl
Ice.Package.TeRK=edu.cmu.ri.mrpl
Ice.Default.Package=edu.cmu.ri.mrpl

#
# The proxy to the Glacier2 router for all outgoing connections. This
# must match the value of Glacier2.Client.Endpoints.
#
Ice.Default.Router=TerkGlacier/router:tcp -h glacier.terk.ri.cmu.edu -p 10004

#
# The proxy for the Glacier2 router, installed in the client's
# object adapter named Robot.Client. This router proxy must
# match the value of Glacier2.Client.Endpoints.
#
Robot.Client.Router=TerkGlacier/router:tcp -h glacier.terk.ri.cmu.edu -p 10004

#
# We don't need any endpoints for the client if we use a
# router. Incoming requests are received through connections
# established from the client to the router.
#
Robot.Client.Endpoints=

#
# No active connection management is permitted with Glacier2.
# Connections must remain established.
#
Ice.ACM.Client=0
Ice.ACM.Server=0

#
# Ice.MonitorConnections defaults to the smaller of Ice.ACM.Client or
# Ice.ACM.Server, which we set to 0 above. However we still want the
# connection monitor thread for AMI timeouts (for completeness, even
# if this application doesn't use AMI).
#
Ice.MonitorConnections=60

# DO NOT SET THIS PROPERTY!!!  See this thread for why:
# http://www.zeroc.com/vbulletin/showthread.php?p=9557
#
# Ice.RetryIntervals=

# Print warning messages for certain exceptional conditions in connections
Ice.Warn.Connections=1
Ice.Trace.Network=1
Ice.Trace.Protocol=1


#
# SSL Settings.
#
Ice.Plugin.IceSSL=IceSSL.PluginFactory
Ice.ThreadPerConnection=1
IceSSL.Client.Keystore=ssl/client.jks
IceSSL.Client.Password=password
IceSSL.Client.Certs=ssl/certs.jks
IceSSL.Server.Keystore=ssl/server.jks
IceSSL.Server.Password=password
IceSSL.Server.Certs=ssl/certs.jks
IceSSL.Server.ClientAuth=2
IceSSL.Trace.Security=0

After pasting it in, I see that we have a bunch of IceSSL stuff still in there (left over from when I hacked up a demo of the client in Ice for Java. A coworker ported it to Ice-E for C++). But would that matter? The client is C++ running under Ice-E. Since Ice-E doesn't support SSL, wouldn't the IceSSL stuff just get ignored? I'll take it out and see whether it makes a difference.

Benoit wrote:

I've posted a patch here that should fix the assert you mentioned in your post. This should help solving your problem.

Great, thanks, we'll give it a try and letcha know what happens.

Thanks for all your help!

chris

[bartley]

I wrote:

After pasting it in, I see that we have a bunch of IceSSL stuff still in there (left over from when I hacked up a demo of the client in Ice for Java. A coworker ported it to Ice-E for C++). But would that matter? The client is C++ running under Ice-E. Since Ice-E doesn't support SSL, wouldn't the IceSSL stuff just get ignored? I'll take it out and see whether it makes a difference.

No difference with the IceSSL stuff commented out.

We'll try the patch in a bit.

thanks!

chris