Help required with Python as client and server

[bobc]

Hi

I've just started to evaluate ice. I'm using Python for the evaluation. Although the final system will be a language mix it will still use Python for some nodes.

I can get simple calls to work fine and IceStorm also fine with a client and server. What I want to do for the first test is two Python nodes that are both a client and a server using IceStorm and also have a one-way direct call connection. I am aware of the threading issues with Python so I am not expecting true concurrency, just that the three mechanisms can operate together.

It's not clear from the documentation how one should arrange this. The architecture seems to require one class derived from Ice.Application. If I create this instance in a separate thread in the server and the client they both hang in the thread (although the actual operation works fine) i.e. Thread.run() does not return.

Can anyone help please.

Thanks
Bob

[bobc]

Now I've corrected my own silly mistake it seems that I can't use Ice in a thread as it uses signals which will only work in the main thread. Is there a way around this?

Bob

[mes]

Hi Bob,

Welcome to the forum.

Ice.Application is a convenience class and applications are not required to use it. The Ice manual states that you must destroy your communicator before the main thread exits otherwise you risk undefined behavior. Aside from that, you can initialize a communicator and use it from any thread you like.

Hope that helps,
Mark

[bobc]

Hi Mark

Thanks for the info. I'm a little lost at the moment. I can't seem to find anything that really explains how things work. I can read what a communicator is but it doesn't really help. I'm not sure how to create one in Python as I can't find anything that looks like the Python API. Can you point me to any examples or documentation that might help. At the moment I'm feeling my way around in the dark.

Thanks
Bob

[dwayne]

Hi Bob,

The minimal demo (py/demo/Ice/minimal in the source distribution) is an example of a python Ice application that does not use the Ice.Application class. You can also find instructions for writing a simple Hello World python application in the Ice manual here as well as more detailed information about using Ice for Python in Chapters 22-25 of the manual.

[bobc]

Thanks for the input. Ice looks really good and I am sure it can do what I need but I'm having trouble getting over the first hurdles. Bear with me, this is going to be quite long.

I think I need to explain a little more about what I am doing. I have looked at the simple examples and they are fine, no problems with those. The example in the manual again is fine but doesn't move me forward. The chapters on Python in the manual concentrate on mapping with a little explanation of some of the classes but not what I would call an API reference. The chapters on IceStorm only have C++ and Java examples which I can follow but haven't helped me to get this working.

The first real code I want to get working is with IceStorm and a simple client (publisher) and server (subscriber) in Python that both run in a Python thread. When I have achieved that the way will be clear to add other threads to publish/subscribe in the other direction and have a high throughput direct channel as well.

I was using the Clock example as a starting point. I have attached just the simple server.py code and associated files. I am not using Ice.Application so get a communicator in the way shown in the simple example. However, I now have no way to give it the subscriber configuration file so I didn't expect it to work. If I run it I get an 'unknown exception' thrown.

UnknownException: exception ::Ice::UnknownException
{
unknown = ../../../cpp/include/IceUtil/Handle.h:46: IceUtil::NullHandleException
}

It would be great if you could fix the code so it works and show me how it should be done.

Thanks
Bob

[matthew]

You need to provide the configuration to your server, otherwise you cannot read the configuration properties. For example:

manager = IceStorm.TopicManagerPrx.checkedCast(communicator.propertyToProxy('TopicManager.Proxy'))

This code cannot work with the property "TopicManager.Proxy" defined. Of course, you are not required to use properties. You could equally well do something like:

manager = IceStorm.TopicManagerPrx.checkedCast(communicator.stringToProxy('IceStorm/TopicManager:tcp -p 12345 -h 192.168.1.1"))

Using configuration properties is often more convenient, as you can change various aspects of your application without modifying the code. To provide configuration properties in your example, you can run with --Ice.Config=config.sub, or you can manually load the configuration using something like:

Ice.InitializationData initData = Ice.InitializationData()
initData.properties = Ice.createProperties(args)
initData.properties.load("config.sub")

Ice.Communicator comm = Ice.initialize(data=initData)

The structure of server.py is pretty strange also. Why do you want to put the subscriber in a thread?

[bobc]

Matthew

Thanks for the pointers. I now know how to initialize properly. I have added that code but unfortunately it makes no difference and I still get the same exception as below.

Just to clarify this is the beginning of a proof of concept for a peer network. Each node can be a publisher, a subscriber or both and also perform bulk data transfers. That is why it is threaded because I need threads to add the other modes of operation. I have had success with streaming audio data using jack at each end so the bulk data transfer is looking good. I need to crack the publish/subscribe issues now. I would just reiterate that the documentation is pretty hard to fathom for someone just starting out, especially if not using the C++ interface. I looked for InitializationData but I don't think I would have figured how to use it without your help even if I knew it existed.

Any additional help from anyone would be very much appreciated.

Regards
Bob


----
Starting server side subscriber
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
self.run()
File "server.py", line 36, in run
manager = IceStorm.TopicManagerPrx.checkedCast(communicator. propertyToProxy('TopicManager.Proxy'))
File "/opt/Ice-3.3.0/python/IceStorm_IceStorm_ice.py", line 349, in checkedCast
return _M_IceStorm.TopicManagerPrx.ice_checkedCast(proxy, '::IceStorm::TopicManager', facetOrCtx, _ctx)
UnknownException: exception ::Ice::UnknownException
{
unknown = ../../../cpp/include/IceUtil/Handle.h:46: IceUtil::NullHandleException
}

[matthew]

I don't think you need threads to handle both modes of operation. If you follow the route you have done with the sample code, you'll end also end up with multiple communicators which makes your life mode complex, and isn't very resource efficient. I would advise not structuring your application in this way.

With respect to the error you are getting, I suspect that you have still not initialized correctly. Please post a complete example, and I'll tell you what you are doing wrong. Furthermore, in your prior example you did not include the slice; you included instead the slice2py output. It is more convenient for us for you to include everything in your examples.

With respect to the documentation, you should check out the section on communicator initialization for the relevant information. http://www.zeroc.com/doc/Ice-3.3.0-I...ties.31.6.html

[bobc]

Matthew

I appreciate the help. Clearly I'm doing something wrong but I'm not sure what yet. I have attached the client and server and associated files (please let me know if I missed any again as I have a lot of files I am testing with). This worked fine until I moved away from the demo code and took out the base class Ice.Application.

At present the client works in that it will publish from its loop provided I don't start it as a thread i.e. just calling run(). If I run it as a thread I get Client Exception below which was the original problem I was trying to avoid.

The server has not worked since I changed it to not use Ice.Application. It throws the exception in Server Exception below if I don't use a config file, but throws unknown exception if I do use one.

As to threading I don't see how I can get away without. Consider one node type which is a GUI. This will sit in an event loop waiting for user events and then publishes those events but also needs to wait for subscriber events. Yes, some GUI's give you an idle event, others don't. Even harder is a node type that streams audio or receives audio and also needs to publish and subscribe. If threads won't work then I need to know so I can figure if there is a viable way out of that. I already have such a scheme working using net_jack for the audio stream and Pyro for the publish/subscribe paths. What I want to do is replace both those with Ice as a better solution and also so that I can bring in C++ easily to nodes that need it.

Regards
Bob

Client Exception:
Traceback (most recent call last):
File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
self.run()
File "client.py", line 74, in run
pub.main(self.argv, "config.pub")
File "/opt/Ice-3.3.0/python/Ice.py", line 768, in main
Application._ctrlCHandler = CtrlCHandler()
File "/opt/Ice-3.3.0/python/Ice.py", line 626, in __init__
signal.signal(signal.SIGHUP, CtrlCHandler.signalHandler)
ValueError: signal only works in main thread

Server Exception:
Traceback (most recent call last):
File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
self.run()
File "server.py", line 54, in run
adapter = communicator.createObjectAdapter("Command.Subscrib er")
File "/opt/Ice-3.3.0/python/Ice.py", line 306, in createObjectAdapter
adapter = self._impl.createObjectAdapter(name)
InitializationException: exception ::Ice::InitializationException
{
reason = object adapter `Command.Subscriber' requires configuration
}

[matthew]

I'll look at the problem in the morning. In the meantime, some comments:

As to threading I don't see how I can get away without. Consider one node type which is a GUI. This will sit in an event loop waiting for user events and then publishes those events but also needs to wait for subscriber events. Yes, some GUI's give you an idle event, others don't.

This scenario does not need an Ice specific thread, as in your demo. Activating the object adapter causes the Ice runtime to start worker threads to dispatch incoming events.

Even harder is a node type that streams audio or receives audio and also needs to publish and subscribe

It also is not clear to me why your application must create threads for Ice in this scenario. Note that any asynchronous calls that you make on Ice proxies are guaranteed NOT to block, therefore you can safely make Ice invocations from the UI thread.

[bobc]

Matthew,

I think I understand what you mean. So once I say adapter.activate() I will get callbacks into my subscriber methods on a worker thread regardless of what my main thread is doing and I just need to make sure I am thread safe. I assume I can also publish events from within a callback. If that is what you mean then I agree I don't need threads.

Regards
Bob

[matthew]

Yes, that's what I mean. If you want to update the GUI state from an invocation on a servant you need to ensure that the update is done in a method that is safe for the GUI toolkit in question. In general, this means the update must be somehow on the main thread. You can look at our hosted chat demo for examples of such applications.

[matthew]

I tried your latest example, and with the exception of having to remove the call to communicator.shutdownOnInterrupt (there is no such method on the communicator), it worked.

It looks to me like you have a problem with your Ice or python installation. What operating system, and which Ice installation package are you using?

[bobc]

Matthew

I'm not at home at the moment so can't check exactly but I'm using Ubuntu 7.x, Python 2.5 and I built from the latest Ice sources as Ubuntu came with Python 2.5 but the Ice build in the repos was with Python 2.4. I'm pretty sure I cleaned up ok but it's possible I have some remnants of the Unbuntu Ice install. I will try doing the same as you did when I get home to see what it does.

Bob