Consistent Backups For Wish

[Ken Carpenter]

Has any progress been made on making a consistent "snapshot" style backup of the game databases for the Wish game?

If so, is the process/technique something that you can discuss? Is it applicable to any Ice application or specific to Wish?

Is there any requirement to "pause" the game/application while the backup is done?

Thanks,


Ken

[shaver]

Originally posted by Ken Carpenter
Has any progress been made on making a consistent "snapshot" style backup of the game databases for the Wish game?

If so, is the process/technique something that you can discuss? Is it applicable to any Ice application or specific to Wish?

I can't speak for Wish, since I'm not one of the ZeroC/MR people working on it, but the Ice distribution includes a program (Ice-1.2.0/demo/Freeze/backup) demonstrating how to do hot backups of Freeze databases. I've seen posters and papers indicating that Wish uses Freeze for its databases, but they may have other layers atop of it as well.

(I've never used the hot-backup facility myself, other than seeing that the demo worked on my system.)

Mike

[Ken Carpenter]

I think backing up a database on a single machine is quite simple, since the database is in control of the transactional consistency.

The problem is that MMO games like Wish have hundreds of servers running, each with its own database, which means that a mechanism of ensuring that the backups are consistent amongst servers is also required.

It depends on the game/application as well. For some, it might be "OK" for the backups to be initiated a few seconds apart. For others, this could mean absolute chaos.

[bernard]

Hi Ken,

Since Ice 1.2, all Freeze writes are transactional, so it's possible to perform hot backups on any Freeze/Berkeley DB environment, without any application pause. Hot-backups are a Berkeley DB feature for transactional Berkeley DB environments, nothing specific to Ice.

A hot backup contains all committed transactions in the backed up Berkeley DB log files; incomplete transactions will be cancelled if you perform Berkeley DB recovery on this backup.

Now, how does this translate in Freeze terms?
If you use Freeze Maps, you have direct control over Berkeley DB transactions: either you enclose your updates in transactions that map to Berkeley DB transactions, or you update without transactions meaning a transaction per write operation (or read-write iterator).
If you use a Freeze Evictor, everything is cached and a background thread writes from time to time the modified objects (really facets) to the underlying Berkeley DB database using transactions. The application has no control over which facets are included in such transactions, and there is not even any ordering guarantee. For example if you update facet A then facet B then facet A again, the latest facet A update can be saved (by some transaction) before the latest facet B update is saved (by a subsequent transaction).

So let's assume you use only Freeze Maps in several Berkeley DB environments, and you want a snapshot of all these Maps at a given point in time. To do this, you would need to block all transactions while you perform the hot backups. Freeze does not provide such "block all transactions" feature; the application would need to do it.
Note that blocking all transactions does not necessarily prevent writes to the underlying Berkeley DB files; each Freeze/Berkeley DB environment has a checkpoint thread that periodically checkpoints the Berkeley DB environment (i.e. flush the modified database pages to disk and write a checkpoint record to the log). Even without checkpoints, a database read can trigger the writting of modified pages back to disk to free some Berkeley DB cache. Here is a good post on this topic: http://groups.google.com/groups?selm...ing.google.com

Another interesting topic is how you perform these hot backups. The most basic solution is to copy the Berkeley DB files, typically with dd. You can also use snapshots provided by a Logical Volume Manager (e.g. http://tldp.org/HOWTO/LVM-HOWTO on Linux) or your storage device.

Cheers,
Bernard