IceGrid replica group fail-safe design question

[spsoni]

Hi,

A generic question.

I am using Replica Group on ice grid to designate same server on each node (for load distribution purpose). (S1, S2, S3 and S4 as server names on N1, N2, N3 and N4 nodes respectively, all associated with S as replica group)

For each server (master, say S1) there are two backup servers(say S1B1 and S1B2) on any two random nodes (to make my each replicated server fail-safe).

If, master fails temporarily, I have mechanism to switch any of the one backup server to take a master role.

So, my question is, what replica group shall I put in backup servers, such that I do not have to modify my clients for replica group S? Such that, backup server is accessible to those clients, as soon some backup server becomes master in case of real master failure, as available replicated servers (4 in total in my case as above) for replica group S.

Or, what naming convention(Repica grouping) or IceGrid design, shall I use to actuate solution to this problem.

Note 1: When actual master recovers/restarts, possibly it becomes backup to current master in its sub-group.

Note 2: Master is a read/write berkeley db server and Backups are replicated Berkeley DB clients which can be used for read only purpose. Therefore, I need to keep exclusive sub-set of replica group for this fail-safe purpose.

I even thought of keeping two adapter per server and associate each adapter one in master replica group and other in sub-set replica group (including 1 master and 2 backup servers). But not sure, how to switch any backup adapter to master replica group on the fly. I was bit lost in this vague idea.

Example Server Proxies (I call backup servers as client due to BDB replication client thing):

Proxy naming convention for mater server:
dbNodeIdentity -t @ DBNode-<Node Index>-master.DBNodeAdapter

Proxy naming convention for backup server:
dbNodeIdentity -t @ DBNode-<Node Index>-client-<Master Node Index>.DBNodeAdapter

dbNodeIdentity -t @ DBNode-1-master.DBNodeAdapter  : Master (writeable)
dbNodeIdentity -t @ DBNode-1-client-2.DBNodeAdapter : Backup (read-only)
dbNodeIdentity -t @ DBNode-1-client-3.DBNodeAdapter : Backup (read-only)
dbNodeIdentity -t @ DBNode-1-client-4.DBNodeAdapter : Backup (read-only)
dbNodeIdentity -t @ DBNode-2-client-1.DBNodeAdapter : Master (writeable)
dbNodeIdentity -t @ DBNode-2-client-4.DBNodeAdapter : Backup (read-only)
dbNodeIdentity -t @ DBNode-2-master.DBNodeAdapter  : Master (writeable)
dbNodeIdentity -t @ DBNode-3-client-1.DBNodeAdapter : Backup (read-only)
dbNodeIdentity -t @ DBNode-3-client-2.DBNodeAdapter : Backup (read-only)
dbNodeIdentity -t @ DBNode-3-master.DBNodeAdapter  : Master (writeable)
dbNodeIdentity -t @ DBNode-4-client-3.DBNodeAdapter : Backup (read-only)
dbNodeIdentity -t @ DBNode-4-master.DBNodeAdapter  : Master (writeable)

Thanks in advance.

Surya

[benoit]

Hi Surya,

If I understand it correctly, you use the replica group S for load balancing purpose only and S1, S2, S3 and S4 are not replicas sharing the same state. Each server has its own state which is replicated on S1B1, S1B2 (read-only replicas). Is this correct?

Assuming you have clearly split your interfaces to not mix the replicated functionality with the non-replicated functionality, you could indeed use two object adapters per server.

One would host Ice objects member of the replica group S (members would be S1, S2, S3) and the other would host Ice objects of the replica group Sn (member of this replica group would be Sn, SnB1, SnB2). Clients would first talk to objects members of the replica group S to obtain a proxy from one of the replica group Sn. This way when invoking on proxies of the replica group Sn, your clients would be able to fail-over to SnB1 or SnB2 if Sn fails.

Although it doesn't directly discuss your use-case, you might want to take a look at my article on replication in the issue #23 of the Ice newsletter, here. It does discuss how to design your Slice interfaces to make it easier to replicate a service.

Cheers,
Benoit.

[spsoni]

Hi Benoit,

If I understand it correctly, you use the replica group S for load balancing purpose only and S1, S2, S3 and S4 are not replicas sharing the same state. Each server has its own state which is replicated on S1B1, S1B2 (read-only replicas). Is this correct?

Yes, your understanding is correct and I am properly managing independent state for each repla within group S and proper replication within S1, S1B1 and S1B2.

To be precise, for replication I am using replication feature provided by BerkeleyDB (as also referred in your article) and usual Replica Group for master databases.

Assuming you have clearly split your interfaces to not mix the replicated functionality with the non-replicated functionality, you could indeed use two object adapters per server.

One would host Ice objects member of the replica group S (members would be S1, S2, S3) and the other would host Ice objects of the replica group Sn (member of this replica group would be Sn, SnB1, SnB2). Clients would first talk to objects members of the replica group S to obtain a proxy from one of the replica group Sn. This way when invoking on proxies of the replica group Sn, your clients would be able to fail-over to SnB1 or SnB2 if Sn fails.

So, as you would be aware, if master database fails in BerkeleyDB, clients(replicated DB) undertake their own elections and one of them becomes master. I have so far properly wrapped these master and client replicated BerkeleyDB. But, I am now trying to make it transparent for actual Ice client to be unaware of underlying BerkeleyDB replication. So that for any master failure, client(replicated DB (wrapped in Ice interfaces), usually read only, becomes master - writeable). So, in your suggestion, I could not get, how would SnB1, if selected as next master on Sn failure, would register itself into group S. So that, this SnB1 (now master) would be transparently selected by client. I could not understand, " Clients would first talk to objects members of the replica group S to obtain a proxy from one of the replica group Sn". As, my curiosity is, as if Sn fails, how will it further resolve to group Sn. Or, if we by default keep SnB1, SnB2 already part of group S, how would we distinguish, which one is master, which I am currently doing by exposing interface (isMaster()). But I need IceGrid to resolve for me, to transform this logic from client to IceGrid. Please elaborate a little bit further.

Many thanks.

Surya

[benoit]

Hi,

You could add an object adapter to the SnB1, SnB2 servers which would be a member of the replica group S. This object adapter would only be activated if the server is promoted to master and deactivated when the server is a read-only slave again.

Cheers,
Benoit.

[spsoni]

Hi Benoit,

Yes, activation/deactivation was simple and straight forward. Didn't thought of that

Many thanks.

Surya

[spsoni]

Hi Again,

Following are the different possible object adapter states.

1. hold - where requests are queued/timedout and not forwarded to servant until adapter is activated again.
2. active - this state is for normal communication
3. inactive - no further communication, adapter is destroyed or about to be destroyed.

Using inactive(using deactivate) state would not be a good idea for any master adapter degrading to slave state. As, this would permanently destroy the adapter.

If I use hold to temporary deactivate replica S adapter whenever master server becomes slave then all the messages for this adapter would be queued until TimeoutException or ConnectTimeoutException exception. Which is again not desired. Rather what is desired is membership of adapters with replica groups dynamically changes with servant internal state (master/slave berkeley db). So that, if I use IceGrid.Query.getAllReplicas(proxy) should return me currently registered master or slave adapters only depending on proxy to be master or slave respectively. Or, is there any other API, which can give me more control over this Replica thing or give me current state of remote adapter. As, I do not want to put the timeout thing, currently (or by default) this time is too long.

I am open to suggestions.

Thanks.

Surya

[benoit]

Hi Surya,

Using inactive(using deactivate) state would not be a good idea for any master adapter degrading to slave state. As, this would permanently destroy the adapter.

Hmm, why is destroying the adapter an issue? Can't you set it up again once the slave is promoted to master?

If I use hold to temporary deactivate replica S adapter whenever master server becomes slave then all the messages for this adapter would be queued until TimeoutException or ConnectTimeoutException exception. Which is again not desired. Rather what is desired is membership of adapters with replica groups dynamically changes with servant internal state (master/slave berkeley db). So that, if I use IceGrid.Query.getAllReplicas(proxy) should return me currently registered master or slave adapters only depending on proxy to be master or slave respectively. Or, is there any other API, which can give me more control over this Replica thing or give me current state of remote adapter. As, I do not want to put the timeout thing, currently (or by default) this time is too long.

The getAllReplicas method returns all the replicas regardless of their status. To get the state of an deployed object adapter, you would have to use the getAdapterInfo method of the IceGrid::Admin interface (if the proxy member is 0, the adapter is inactive).

Cheers,
Benoit.

[spsoni]

Hi Benoit,

Today, it took me quite long to identify multiple adapter problem.

In my server template I have defined two adapters (one for master, other for slave). So, until I start both adapters, server is not put in state of started ("Active"). Hence, it times out. I have got some factory server to start these master/slave servers by instantiating from template. So, when I look into icegridadmin gui, I can see that server is marked as "Activating..." if I only start one adapter depending on my case.

So, in my one of the experiments, I did start both adapters, then server started properly. But, say after 5 secs, I deactivate one of the adapters, server state also changes to "Deactivating.." and finally terminates.

Please advice, where is the problem.

Update:
======

After reading forum discussion, I got little bit better interpretation that if adapter name is specified in server template then those adapters must be active in order for server to be in active state. To avoid this, I shall create adapters on the fly, rather than keeping adapter reference in server template.

But with that, I have two problems:
1. I have two mutually exclusive adapters, so cant specify any one of them as default in the server template and create other one on the fly.
2. Both adapters are associated with different replica groups. Would it be possible to create adapters on the fly and at the same time associate them to some specific replica group?

Regards,

Surya

[spsoni]

Following is my DBNode application template XML. I instantiate server from this template from another factory server.

<icegrid>
  <application name="DBNodeApp">
    <server-template id="DBNodeTemplate">
      <parameter name="index"/>
	  <parameter name="nodeType"/>
	  <parameter name="dbhome"/>
	  <parameter name="dbport"/>
	  <parameter name="others"/>
	  <parameter name="indexLevel"/>
      <server
			id="DBNode-${index}-${nodeType}"
			exe="python"
			activation="on-demand"
			user="ssoni"
			deactivation-timeout="10"
			activation-timeout="60" >
        <option>dbNode.py</option>
        <adapter name="DBMasterNodeAdapter" endpoints="tcp" register-process="true" replica-group="ReplicatedDBMasterNodeAdapter">
		  <property name="Identity" value="dbNodeIdentity" />
		  <property name="NodeIndex" value="${index}" />
		  <property name="NodeType" value="${nodeType}" />
		  <property name="dbhome" value="${dbhome}" />
		  <property name="dbport" value="${dbport}" />
		  <property name="others" value="${others}" />
		  <property name="indexLevel" value="${indexLevel}" />
		  <property name="DatabasesPath" value="dbs" />
        </adapter>
        <adapter name="DBSlaveNodeAdapter" endpoints="tcp" register-process="true" replica-group="ReplicatedDBSlaveNodeAdapter">
		  <property name="Identity" value="dbNodeIdentity-${dbport}" />

		  <property name="NodeIndex" value="${index}" />
		  <property name="NodeType" value="${nodeType}" />
		  <property name="dbhome" value="${dbhome}" />
		  <property name="dbport" value="${dbport}" />
		  <property name="others" value="${others}" />
		  <property name="indexLevel" value="${indexLevel}" />
		  <property name="DatabasesPath" value="dbs" />
        </adapter>
      </server>
    </server-template>

    <replica-group id="ReplicatedDBMasterNodeAdapter">
      <load-balancing type="round-robin"/>
      <object identity="dbNodeIdentity" type="::NextDigital::DBNode"/>
    </replica-group>
    <replica-group id="ReplicatedDBSlaveNodeAdapter">
      <load-balancing type="round-robin"/>
    </replica-group>

  </application>
</icegrid>

I havent yet optimised few of my parameters. But i liked the idea of these parameters is that, once my server crashes and if I just restart it, it will retrieve all those parameters, as they might be getting stored as instance in icegrid registry.

My question is in reference and continuation to the last post.

As, you might observe, I need to pass same parameters to both of the adapters properties. Can I have some shared property thing for each server? As, I was thinking to start/create adapters on the fly, I was thinking, where will I get those properties from, if I dont specify any of the adapter detail in above xml.

[benoit]

Hi Surya,

Specifying <property> elements within the scope of an <adapter> element is the same as specifying them at the <server> level. So the second definitions of the properties in your XML descriptor just override the first ones.

It's not quite clear to me what you mean by creating object adapters on the fly. In any case, if you don't want a given object adapter to take an active role into figuring out the server state, you should set its "server-lifetime" attribute to false.

So in your case, I believe the DBMasterNodeAdapter adapter should have this attribute set to false:

<adapter name="DBMasterNodeAdapter" endpoints="tcp" server-lifetime="false" replica-group="ReplicatedDBMasterNodeAdapter"/>

With this setting, IceGrid won't pay attention to the status of the object adapter when it comes to figure out whether or not the server finished activating or started deactivating. Your code will be free to activate/deactivate this adapter anytime.

See the manual for more information on this attribute.

Cheers,
Benoit.

[spsoni]

Hi Benoit,

It works!

Property thing also works as well as adapter activation/deactivation due to server-lifetime="false".

Many thanks, you made my day.

Regards,

Surya

[benoit]

Great! I'm glad to hear that it works now .

Benoit.