The use of ICE for Massive Multiplayer Online Games

[Corman]

I am currently working with a company reviewing technology for their planned MMOG. I am more of a 3D graphics and physics programmer, but do have some experience with networking. But I have been given the task to review different networking technologies. I say that another MMOG was using ICE but I have several questions about ICE, its ueses, and Mutable Realms possible uses of it. So here are my views and possibly questions...

First I am going to make some assumtions and please correct me if I am wrong. First I am going to lay down what I understand of previous MMOG setups. Since i know the terms Client and Server in what ICE uses them as can be on the same machine, here I am not unless I say otherwise (and even then I may forget so bare with me). So in previous MMOGs the Client is the 3D interactive interface to the game World which is the Server. The World is usually Sharded into parallel worlds then each has corisponding Zones. So their way of load balancing was to have seprate Sharded Worlds on top of having this Zones. So players usually happen to play on the Shards of their choice and when travelling the World they Zone, which is the connection and user data transfer to this new world along with being updated on their new Zones info as well.

So with that aside this is how I see how ICE would be used, and again please correct me if I am wrong our inform me of other ideas. I see its use for highly scalable sServer cluster Server side. What I see is that now instead of Shards and Zones you have a single large World. But even this has problems just being describe that simply. So one would now need a way for balacing this. In comes the World Cell. A Cell is a way of mapping Server Node space to logical World space. In a typical MMOG if whole lot of players were in one Zone, that Zone Server would be bogged down beyond belief with now way of balancing this load. They might of had simple fixes or hacks but the problem was still there. But with ICE and the idea of a Cell, a less bogged Server Node could load balace this region by now Serving part of this Cell. So now instead of one Server taking all the load, now several Server Nodes could be running this. And from what I understand players would still percieve this and one area as how the data is being sent from Cell bounds to each Server Node then to the players. This would cut Zoning totaly out and allowing for alot more users being that it is scalable and auto balancing.

Do understand alot of this is how I understand how MicroForte's Big World Technology works, but is outside the realm of our companys resources to license. So I am here trying to undrstand how to create such a system, or some other system for large worlds and large number of players, hopefully using ICE.

I know this looks more like a statement that questions, but I wanted this out so you know what I meen when I do ask the questions and for further forum talking.

So I guess my questions are...

1. Am I properly viewing the correct possible use of this system?
2. How would one go about creation such a system?
3. Compared to other systems I may have not described, how could ICE be used to help support large games worlds and large number of players in a MMOG.
4. I know you may not beable to talk about it, but how is Mutable Realms using this system for Wish?
5. Am I as lost as I think I am?

hopefully someone from the forums or even ZeroC will start up a conversation up with me on these or other topics related to ICE and MMOG game development. Thank you all for your time and understanding ahead of time, I look forward to hearing from all of you.

-Corman

[marc]

I'm afraid to answer your questions in detail, I would have to write a very long post

In general, Ice is multi-purpose communications middleware, that allows you to build all kinds of applications, including online games. It has all the ingredients needed for such online games: a solid object model, high scalability, various protocols, firewalls, security, persistence, fail-over, etc. See the Ice manual for a complete list

However, Ice is not a ready-to-use solution for online games. It can only serve as a solid foundation. So if you are looking for a finished solution, Ice is not for you. If you are looking for a powerful tool to implement your own online gaming solution, then Ice is exactly right for you.

[Corman]

Thank you Marc for your quick response. I fully understand the drive not to answer all my questions, but even then alot of them I do know the answers for and just want to get a discussion going knowing that they can be some what long, inspiring, and helpful for anyone looking at ice for such a sollution. As with all online games there is no one single already made package to fit their needs otherwise it be done over and over with games feeling and doing the same things. So with that I am looking at using ICE to fully implement our own online gaming solutionand , so far just like you said Marc, ICE is exactly right for us.

Again with that said what I am looking forward to get done with this thread is to get a variety of cross talk going on using ICE and its wonderous systems as a pallet to dicuss and create a framework of ideas for such a online gaming solution. It not my goal at this very minute to start coding such a massive beast, but just get the flow charts going in everyones head and make this a useful tool creation place just not for me but for anyone looking at starting such a undertaking.

So again this is not so much a thread on questioning the internals of ICE but the apllied uses of them to roughly sketch out this sort of solution for online gaming.

Thanks again Marc for you reply, and I look forward to yours and many others again as I try to get the ball rolling here. If I see that this sort of thread needs more of its own community site or that it souldnt really be on here at all I will try to get a site going on my company's web site if I can get permission. But since this does have to deal with ICE and I do get the felling that others might be looking at or are currently working on such systems I just felt it needed voiced.

Yet again thank you all ahead of time, I look forward to everyones input.

-Corman

[Ken Carpenter]

Hi Corman.

I was also investigating ICE for use in an MMOG before our company ran into financial problems. I'd be happy to discuss some design ideas we had worked through if you are interested.

You mentioned that BigWorld is beyond your means. How much were you quoted? Also keep in mind that ICE is not free unless you plan to GPL your entire game.

To answer your questions somewhat:

1. ICE could certainly be a component in the system you described. It does NOT, however, provide the type of "World Cell" load balancing you described. You would need to implement yourself.

2. ICE can be used for communication between servers. For example, you could have servers do load balancing by handing off players and their data from cell to cell. Just define the interfaces in SLICE and write the code. You would also use ICE for 3D-client-to-server communications. ICE provides authentication and firewall abilities here. IcePack provides you with infrastructure for managing your huge server farm.

3. ICE itself doesn't provide you with any directly usable abilities to create large worlds. It DOES, however, provide lower-level infrastructure to build on top of to provide these services to your game. If you have a well-designed object hierarchy and a well-designed distribution of objects to servers, ICE provides the ability for them all to communicate seamlessly and coordinate their actions, regardless of which server has which object at a particular time.

4. I don't know the answer for this one, although I have spoken with Matthew and Marc about Wish, and I believe it is fairly similar to what I described above. I don't know, however, how they are handling world partitioning. I had been considering distributed Sphere Trees or Circle Trees for a while.

5. Probably not! :-)

You may also want to consider code-generation for certain aspects of the system. I designed an XML-based format that described the game objects that would be sent between client and server. I used this format to describe the valid ranges for numbers, valid lengths for strings and arrays, how frequently a client was allowed to make a particular call, etc. I then code generated the required ICE Slice files and validation routines used by the Slice types. I generated code for both client and server side validation.

Anyway, I hope this helped somewhat.

[Corman]

Hello Ken (and everyone else again),

I am happy to see people responding to my not-so-random mind blurbs. I would have responded sooner (yes, I think I fall under the catagory of forum buzzard) but I was out of town with only limited technology available to me, got to love country life every now and then.

Top of the order, as for BigWorld, even as they say, thats a project to project basis (meening "how much you got" most of the time) so I will not list it. As what I said before about being beyond us it not totaly true, it just seems like our board like to direct money elsewhere.

I do know as you pointed out that ICE is not the MMOG "be all". I am looking at it objectively as the foundation more so for the servers. I know this may not be the correct term to use, I see it as a wrapper for all the non-game specific network layer so to say. I know these terms of super simplefying this can lead to over generalizationa and misunderstanding, but I use them for ease of communicating my ideas to others. So I am not looking at ICE to be the the large world solution, I am looking at it to be the part of the solution I do not have to create myself and get to the game specific issues.

Like I said before, I am not all here just to talk about ICE, but to do mind experaments on what can be built on ICE. I started it here since there may be others on the same path or are seeing this possible use of ICE and want to have a common ground to dream out loud.

Now to game stuff...

We have worked with World Cells for some time. We are now also trying another end. Instead of having many duplicate applications (ai, items, combat, etc.) on each server world cell (this is almost like zones, but seamless... so it still has some of its downfalls). We are not having distributed services. So we have a Item service running on systems, Combat on another, and so forth. So this has some cons as well, having to deal with different service for different things, can add som latency. But there are many pros as well. One of wich is a more gracefully recovery if something does go boom. For examples, if a standard cell or zone system, the server goes..... all its state data is lost since its last state save. But for a distributed service on the other had.... if AI service bombs, the monsters just seem to lost their train of thought as another AI service comes on line or takes over... or for the items, so you loose the ability to move equip something for a second or two. So it allows for gracefull saves, and keeps like thinks alike. Also for moving from cell to cell, cells are now purely spatial, nothing more. so now you dont have to proxy or transfer data from server to server on all their stats items and the like, the load for the item server stays at its 350,000 items or so no matter how many players are mobbing in the town square. The analogy might now work so gracefully as I meen it to be, but its like what I have learned from many people in 3D programming... "Have a structure of arrays, not an array of structures"... this is more ment for the linear friendlyness of memory, but it seems to hold partial true to what I have said here I believe. So instead of having a server cluster of duplicated applications (the array of structures) have the cluster have specialized services (the structure of arrays). I know all this is a rant outside the true meenings of this forum, but I want to share .

And as for your XML comment Ken, we have been using it in a somewhat similar way but more towards data-driving system [Riley03] than a code generator, well not fully true, we have been working on some mobile code using it.

As for partioning, I have been playing with momentum based sphere trees (well circle trees, we ignore up for now since it hasnt made much of a difference). We track players and player groupings and how they are moving, so we keep the players that are more likely to be interactive with each other visually or otherwise are kept on the same server even though parts of the group may be over physical cell bounds, and so on and so forth, so even with the dynamic nature of the cells they have special use cases so to say when to expand or contract, and not just by physical space, but by the chance the players are going to stay or leave. So momentum here is no use for it physics terms I believe. Its a long story on how we came to use that term if I remember correctly.

So Ken, yes I would like to further discuss design ideas and the like if you are up to it. And that goes equally to anyone else out there yet to join in.

As always, thank you for joining in this talk with me and I am looking forward to your future posts along with anyone elses as well. Until next time, take it easy .

-Corman

[Edited for grammar problems caused by to much coffee at 2 in the morning]

[marc]

I would love to give you more details about the Wish design, and how it uses Ice. However, I cannot do this here for confidentialiy reasons. We would have to take this to private email and/or a phone conference. Furthermore, you would have to sign an NDA with Mutable Realms.

Please let me know if you would like to start such a discussion. Just email us at info@mutablerealms.com.

[Corman]

Howdy Marc,

I do not know if it is a weird feeling or some shred of truth, I must be pushing on some close buttons to get an offer to talk about this . (Or it might be the coffee... yup I am still awake from my last post, a programmers job seems to be never done... or a forum buzzards work for that matter.)

I am pleased you are offering this chance to talk on the back lots so to say. I am up to this and will take the offer up as soon as I come down of this coffee rush and crash and get some sleep. I had to catch up on my work since I was out of town for a few days like my last post stated.

Thanks Marc, you will hear from me soon.

And as always to my tradition, I look forward to more posts from the world around me here.

[Hopefully no edits will be need for poor little sleep and coffee grammar]
[Nope, still bad grammar]

[MarkM]

I think with the massive influx of indie MMRPG development, most of which, unfortunately, are a complete sham, a lot of people are looking for a zero cost plug-in to bring their world to life. I am somewhat guilty of this myself but after many years toying with MMRPG design I dove in with both feet a year and some ago and my project has finally arrived at the network layer phase =)

Like the original poster I am looking for some discussion on ICE as it pertains to massive/ultra massive simultaneous client/server handling. I am out of my league as far as my programming ability goes but I am trying to get a vague grasp on the realities and physical limitations of the network possibilities that the coding team is going to deal with. Being an indie developer with only amateur team members at this point we have decided on the less strenuous method of world handling by choosing Zones instead of seamless cell integration. I have purchased a very simplistic engine to script the worlds and build the landblocks but we are looking for a practical backbone to base our network module on. Ice is one of the options.

In all honesty our entire goal is to develop a real solid alpha project with all core world engines fully funcitonal to use for wooing some of the next gen MM online engine developers to our cause. Our network needs will not be for a full production 1000+ simultaneous connection engine but more for a 500-1000 user alpha testing world so that we can fine tune our core designs.

Ice is obviously capable of this and more. The guys at Muteable are proof of that but I can see already that with the amount of code we are going to have to write to implement ICE and SLICE it may be a little more than a bunch of amateurs can handle. I've currently got 2 coders working on inserting ICE into our server design doc ATM. I promised them I'd delve into it myself so we could have an intelleigent conversation about it once they got it sorted.

I anticipate I'll have more than a few questions regarding our chosen infrastructure but untill then I will try and mask my ignorance with silence

I will keep this thread earmarked though and post our design once we get some headway.

Cheers guys,
Mark

[madborg]

My interest is in MMO/DVE (distributed virtual environments). It looks like there are at least a couple people here that are interested in learning more on what it takes to make ICE into a viable package for building various worlds. So I hope this thread stays alive or spins off into a subtopic.

maddy

[Corman]

Hello MarkM and madborg,

Thanks for joining in on the talks here, normaly I would have replied as soon as each one posted, I like having a solid welcoming feel. But I have been bussy doing work since we are currently hiring to fill two positions and going over everything and making choices with the rest of the crew is kinda nerve shaking if none the least. So welcome, and I hope this forum or the spin offs of will help you in your efforts. On a side note, like I said in a previous post I am working on getting a dedicated forum going to be hosted on my company's website, and it will not be restricted to ICE alone. Also anyone is free to ask me any questions or for help at any time and I will be glad to help within the bounds of my NDAs at this current time. So I will be here hovering around and help guide talks and will help them mature when I can. So thanks for joining in and I hope to see so productive idea farming here.

-Corman

[marc]

At the Integrate.2003 conference, ZeroC demonstrated Wish as an example for a complex Ice application (and also because computer games tend to attract people ). The attached poster was displayed to give an overview of how Wish uses Ice.

[MarkM]

Thanks for the responses guys. George is incharge ofthe network integration into our current client engine and after a light discussion last night he has decided we are going to move forward with the ICE implementation.

So with that I would like to thank the entire ICE community for all of the support we are going to need and also the developers of ICE for making the software available to the average shmos like myself.

We are not sure what hurdles we are going to run into but we are quite confident that with time each little problem is really nothing more than an opportunity to learn something and improve.

Thanks for the welcome and I will be posting more info as we progress.

Mark

[MarkM]

I will not profess to be an expert on GPL, but while we were integrating our first version of our ICE server we found we had to load several components to get ICE to compile. These components were GPL license components and I just wonder, if someone has a moment, to clarify for me that this will not have any effect on our final product.

Again I really am not a legal professional but I have taken great pains to this point to make sure that all my code was self produced, or royalty free commercial code that I have purchased. Introducing even one GPL snipet of code has the potential to cause me much grief later on.

I assume that I am way off track as you guys wouldn't be licensing your software if it had GPL dependancies. Just wanted clarification.

Thanks in advance and sorry if this has been answered before I just didnt see it.

Mark

[marc]

Ice itself is GPL. But the libraries it depends on are not GPL. Berkeley DB is GPL-like, but our commercial license includes a commercial license for Berkeley DB (we have a contract with Sleepycat software).

[MarkM]

I should've put 2 and 2 together Marc. Thanks for clearing that up. I knew there was a commercial license for something included I should've dug a little deeper.

As for ICE being GPL I am not concerned about that as when I license it I am not bound by those restrictions. Thanks for setting my mind at ease. Hope we can do the network engine justice

Mark