Problems when distributing messages through IceStorm to multiple replicas

[lmgarcia]

Hi everybody,

I'm having an issue when using Icestorm in my application, which involves a replication mechanism in my own application-domain service. The followed approach is similar to highly available Icestorm, that is, a master-slave scheme where only the master updates the state but read-only operations can be attended by any replica.

I use Icestorm to distribute changes from the master to the slaves and to decide who the master is (bully algorithm). Basically, the problem is that the messages sent by a replica do not reach the others when are distributed through IceStorm. In fact, only the replica that sends a message receives it. However, if a replica receives a message via an indirect proxy, for instance, instead of using a topic, then there is no problem at all.

All the replicas are publishers and are subscribed to the topics for the replication system and the master election. I hope the following code clarifies my explanation:

// The names starting with an underline are private attributes of the class.

void AMSI::setPublisherAndSubscriber(const Ice::CommunicatorPtr& communicator)
{
  Ice::ObjectPrx obj = communicator->stringToProxy("FIPA.IceStorm/TopicManager");
  _topicManager = IceStorm::TopicManagerPrx::checkedCast(obj);

  _amsTopicManagerProxy = _adapter->addWithUUID(
    new AMSTopicManagerI(e_priority, this))->ice_twoway();

  string topicName[] = {"electionsAMS", "replicaInfoAMS"};
  IceStorm::TopicPrx topic[2];

  for (int i = 0; i < 2; ++i)
  {

    // Set the publishers.

    while (!topic[i])
    {
      try
      {
        topic[i] = _topicManager->retrieve(topicName[i]);
      }

      catch (const IceStorm::NoSuchTopic&)
      {
        try
        {
          topic[i] = _topicManager->create(topicName[i]);
        }
        catch (const IceStorm::TopicExists&) {}
      }
    }

    Ice::ObjectPrx pub = topic[i]->getPublisher()->ice_twoway();

    if (i == 0)
      _electionPublisher = FIPA::AMSTopicManagerPrx::uncheckedCast(pub);
    else
      _replicaPublisher = FIPA::AMSTopicManagerPrx::uncheckedCast(pub);

    // Subscribers.

    IceStorm::QoS qos;
    qos["reliability"] = "ordered";
    qos["retryCount"] = AMS_ICESTORM_RETRY; // this is 3
    topic[i]->subscribeAndGetPublisher(qos, _amsTopicManagerProxy);

  } // end for

}

AMSTopicManagerI handles the received messages and also discards the messages that come from itself. The server is an Icebox service and all the application is deployed with Icegrid.

I suspect I'm missing something when using objects that are publishers and subscribers at the same time.

Thanks a lot for your help in advance.

[bernard]

Hi Luis,

I didn't spot anything wrong with your code. If IceStorm cannot reach your (other) subscribers for some reason, you should turn on some tracing to figure it out, for example:
Ice.Trace.Network
IceStorm.Trace.Subscriber

Is _adapter a direct adapter? What is its endpoint configuration?

Best regards,
Bernard