Peer to Peer problem

[CharlesHan]

Hello, i am new to ICE and using Slice for my current C++ project under WindowsXP with Visual Studio 2003. The project has a peer to peer network architecture. The problem now is when the third peer joins the system just crashed ramdomly.

I also wrote up a simple test program and had the same problem. The protype code is like this:

...
...
while(true)
{
if(time > TIME_INTERVAL)
{
for(int i = 0; i < sizeofDataobject; ++i)
{
sendInt(dataObejct[i]->getInt());
sendDouble(dataObejct[i]->getDouble());
}
}
}

.....

Two peers can communicate very well but when the third peer joins a windows crash dialog pops. It seems to me that the nested while-for loop caused some threading problem but not so sure. Please let me know if you need more information.

Thanks in advance.

Charles

[matthew]

I'm afraid that this description doesn't give us much to go on. In order to help you the best thing would be to give us a small complete compilable test case that reproduces the issue.

[CharlesHan]

Hey Matthew,

Thanks for your prompt reply, here is more details,

The Slice module file we created:

module osgart
{
module net
{

sequence <double> DoubleSeq;

struct Matrix
{
DoubleSeq vals;
};

class Contact
{
void receiveMatrix(Matrix m, int msgTypeID, int peerID);
void receiveDouble(double d, int msgTypeID, int peerID);
void receiveInt(int i, int msgTypeID, int peerID);
void receiveString(string s, int msgTypeID, int peerID);
void announce(Contact* c);
};


};
};

In the application we have CommunicationManager Class that init the number of contacts as such:

int argc = 0;
char* argv[1] = {""};
ic = Ice::initialize(argc, argv);

// Create and register contact
ContactI* contact = new ContactI();
ss << "tcp -p " << port << " -h " << host << ":udp -p " << port << " -h " << host;
Ice::ObjectAdapterPtr adapter = ic->createObjectAdapterWithEndpoints("OSGART.Collab ", ss.str());
adapter->add(contact, ic->stringToIdentity("contact"));
adapter->activate();

// Iterate through peers, grab their contacts.
for (vector<Peer>::const_iterator it = peers.begin() ; it != peers.end() ; it ++)
{
if (peerID != it->id)
{
stringstream ss;
ss.flush();
ss << "contact:tcp -h " << it->host << " -p " << it->port << "\n";
std::cout << ss.str();
try
{
ContactPrx cp = ContactPrx::checkedCast(ic->stringToProxy(ss.str()));
cp->announce(ContactPrx::uncheckedCast(adapter->createProxy(ic->stringToIdentity("contact"))));
contacts.push_back(cp);
}
catch (...)
{
std::cout << "failed to connect to host " << it->host << ":" << it->port << "\n";
std::cout << "This is the server." << std::endl;
}

}
}

Then in the main program:

int main(int argc, char* argv[]) {

CommunicationManager::instance()->start();

osg::Timer_t s, t = osg::Timer::instance()->tick();

osg::MatrixTransform* targetTrans = new osg::MatrixTransform();
targetTrans->setMatrix(osg::Matrix::translate(osg::Vec3d(0, 0, 25)));

while(true)
{
s = osg::Timer::instance()->tick();
if (osg::Timer::instance()->delta_m(t, s) > NET_UPDATE_INTERVAL) {
t = s;
for(int i = 0; i < 3; ++i){
CommunicationManager::instance()->sendMatrix(targetTrans->getMatrix(), 0);
CommunicationManager::instance()->sendInt(1, PEER_VISIBILITY);
}
}
}


return 0;
}

One of the sending data methods look like this:

void CommunicationManager::sendInt(int i, int msgTypeID)
{
int peerID = this->getPeerID();
for (vector<ContactPrx>::iterator it = contacts.begin() ; it != contacts.end() ; it ++)
{
(*it)->receiveInt(i, msgTypeID, peerID);
}
}

This test program is similar to my project and has the same problem.

[bernard]

Hi Charles,

I don't see any obvious problem in your code. As Matthew suggested, the best would be to post a small yet complete test-case that reproduces the problem.

If this is not feasible, I'd recommend to create debug build of you application and to try to get a stack trace when you get this crash. This should give an idea of where the problem is.

Note that Ice is fully thread-safe, and you get concurrent requests only if you start threads yourself (in a client) or you configure Ice to use several threads (on the server side). By default, Ice uses only one thread to dispatch all requests on the server side.

Best regards,
Bernard

[CharlesHan]

Hey Bernard,

Thanks for your reply.

I have attached a zip file and please have a look the small test case.

You might want to run the example for several times if it does not crash at the first time.

Cheers

Charles

[matthew]

Just a couple of notes on your implementation. I don't think any of these issues in particular are causing the problem you mention, however, addressing these issues might help you understand Ice a little better and simplify your application design:

// Create and register contact
ContactI* contact = new ContactI();

If the code returns before you assign contact to a smart pointer this will result in a memory leak (in your code you do this by calling adapter->add). It is better to assign the result to a smart pointer immediately to avoid this situation -- see Michi's article in the latest edition of the connections newsletter for details.

adapter->add(contact, ic->stringToIdentity("contact"));
//...
ss << "contact:tcp -h " << it->host << " -p " << it->port << "\n";

Here you use the same identity for all objects, including your callback objects. This probably isn't really what you want. Conceptually all unique Ice objects have a unique object identity. You can see the Ice manual for full details on the Ice object model.

Furthermore, it looks to me like your communications manager class is more or less an event distributor. You would be better off using IceStorm to do this.

[matthew]

I took a look at your code. The problem is that your client is also a server and thus has two threads of control. The first is the main thread where you are calling CommunicatorManager::sendDouble and such. The second is the dispatch threads in which the Ice runtime calls upon the servant (that is the thread that calls ContactI::receiveDouble).

Your problem comes in when ContactI::announce is called. This modifies the contact list at the same time as the main thread is calling CommunicationManager::sendDouble. This concurrent modification is not thread safe and results in the issue that you are seeing.

In short, you need to mutex protect the contact list.

Futhermore, you are not correctly dealing with exceptions when calling the proxy methods, so any client that drops will cause the entire application to terminate. If you used IceStorm instead then this would be a non-issue (as would concurrent modification of the contact list for that matter since that would all be managed by IceStorm itself).

I also don't think that busy looping in main is so good either

[CharlesHan]

Thank you so much for your help. Now i see the light in my life

Thanks again!

Charles