Known issues with mixing 32bit/64bit systems?

[joshmoore]

Currently we are having issues crossing 32/64 bit boundaries. Our server is written in Java. When a C++ client on a 64 bit machine accesses the server on a 32bit machine one of two exceptions are raised in indeterminate order (32/32 and 64/64 combinations work fine):

(1) Most frequently:

Exception:
BasicStream.cpp:363: Ice::NegativeSizeException:
protocol error: negative size for sequence, dictionary, etc.

at

(gdb) bt
#0  0x00002aaaac2330e0 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.6/libstdc++.so.6
#1  0x00002aaaaac59ace in IceInternal::BasicStream::skipSlice () from /usr/lib/libIce.so.32
#2  0x00002aaaaac5f4fa in IceInternal::BasicStream::read () from /usr/lib/libIce.so.32
#3  0x00002aaaab9ce2d1 in omero::model::__read () from ../lib/64bit/libOMERO_common.so.0
#4  0x00002aaaab9ce544 in omero::model::Event::__read () from ../lib/64bit/libOMERO_common.so.0
#5  0x00002aaaaac5fd98 in IceInternal::BasicStream::read () from /usr/lib/libIce.so.32
#6  0x00002aaaaac60246 in IceInternal::BasicStream::readPendingObjects () from /usr/lib/libIce.so.32
#7  0x00002aaaabbad637 in IceDelegateM::omero::api::IUpdate::saveAndReturnObject () from ../lib/64bit/libOMERO_common.so.0
#8  0x00002aaaabb53fe6 in IceProxy::omero::api::IUpdate::saveAndReturnObject () from ../lib/64bit/libOMERO_common.so.0
#9  0x0000000000418645 in IceProxy::omero::api::IUpdate::saveAndReturnObject (this=0x574a60, obj=@0x7fffff991260) at ../include/OMERO/API.h:1848
#10 0x0000000000409ed8 in main (argc=1, argv=0x7fffff991538) at SDK.cpp:261

or

(gdb) bt     
#0  0x00002aaaac2330e0 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.6/libstdc++.so.6
#1  0x00002aaaaac59798 in IceInternal::BasicStream::throwNegativeSizeException () from /usr/lib/libIce.so.32
#2  0x00002aaaab9dee4a in omero::model::__read () from ../lib/64bit/libOMERO_common.so.0
#3  0x00002aaaab9def97 in omero::model::Experimenter::__read () from ../lib/64bit/libOMERO_common.so.0
#4  0x00002aaaaac5fd98 in IceInternal::BasicStream::read () from /usr/lib/libIce.so.32
#5  0x00002aaaaac60246 in IceInternal::BasicStream::readPendingObjects () from /usr/lib/libIce.so.32
#6  0x00002aaaabbad637 in IceDelegateM::omero::api::IUpdate::saveAndReturnObject () from ../lib/64bit/libOMERO_common.so.0
#7  0x00002aaaabb53fe6 in IceProxy::omero::api::IUpdate::saveAndReturnObject () from ../lib/64bit/libOMERO_common.so.0
#8  0x0000000000418645 in IceProxy::omero::api::IUpdate::saveAndReturnObject (this=0x574a60, obj=@0x7ffffff754b0) at ../include/OMERO/API.h:1848
#9  0x0000000000409ed8 in main (argc=1, argv=0x7ffffff75788) at SDK.cpp:261

(2) less frequently

Exception:
BasicStream.cpp:1654: Ice::NoObjectFactoryException:
protocol error: no suitable object factory found for `':
class sliced to ::Ice::Object, which is abstract

at

(gdb) bt
#0  0x00002aaaac2330e0 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.6/libstdc++.so.6
#1  0x00002aaaaac5fbd7 in IceInternal::BasicStream::read () from /usr/lib/libIce.so.32
#2  0x00002aaaab9daa41 in omero::model::__read () from ../lib/64bit/libOMERO_common.so.0
#3  0x00002aaaab9dac8e in omero::model::ExperimenterGroup::__read () from ../lib/64bit/libOMERO_common.so.0
#4  0x00002aaaaac5fd98 in IceInternal::BasicStream::read () from /usr/lib/libIce.so.32
#5  0x00002aaaaac60246 in IceInternal::BasicStream::readPendingObjects () from /usr/lib/libIce.so.32
#6  0x00002aaaabbad637 in IceDelegateM::omero::api::IUpdate::saveAndReturnObject () from ../lib/64bit/libOMERO_common.so.0
#7  0x00002aaaabb53fe6 in IceProxy::omero::api::IUpdate::saveAndReturnObject () from ../lib/64bit/libOMERO_common.so.0
#8  0x0000000000418645 in IceProxy::omero::api::IUpdate::saveAndReturnObject (this=0x574a60, obj=@0x7fffff87c760) at ../include/OMERO/API.h:1848
#9  0x0000000000409ed8 in main (argc=1, argv=0x7fffff87ca38) at SDK.cpp:261

For a server:

java version "1.6.0_01"
Java(TM) SE Runtime Environment (build 1.6.0_01-b06)
Java HotSpot(TM) Server VM (build 1.6.0_01-b06, mixed mode)

Also tried on :

java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)

Linux valewalker 2.6.16-gentoo-r13 #3 SMP PREEMPT Wed Apr 25 15:55:21 BST 2007 i686 Intel(R) Xeon(TM) CPU 1.60GHz GenuineIntel GNU/Linux

        libIce.so.32 => /homes/jmoore/lib/ice/lib/libIce.so.32 (0xb7d18000)
        libIceUtil.so.32 => /homes/jmoore/lib/ice/lib/libIceUtil.so.32 (0xb7ce5000)
        libSlice.so.32 => /homes/jmoore/lib/ice/lib/libSlice.so.32 (0xb7bb1000)
        libGlacier2.so.32 => /homes/jmoore/lib/ice/lib/libGlacier2.so.32 (0xb7b36000)
        libstdc++.so.6 => /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libstdc++.so.6 (0xb7a55000)
        libm.so.6 => /lib/libm.so.6 (0xb7a30000)
        libc.so.6 => /lib/libc.so.6 (0xb7912000)
        libgcc_s.so.1 => /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libgcc_s.so.1 (0xb7907000)
        libbz2.so.1 => /lib/libbz2.so.1 (0xb78f6000)
        libdl.so.2 => /lib/libdl.so.2 (0xb78f2000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb78a0000)
        /lib/ld-linux.so.2 (0x80000000)

and a client:

Linux warlock 2.6.15-gentoo #3 SMP Wed Jan 11 10:31:26 GMT 2006 x86_64 AMD Opteron(tm) Processor 252 AuthenticAMD GNU/Linux

        libIce.so.32 => /usr/lib/libIce.so.32 (0x00002aaaab936000)
        libIceUtil.so.32 => /usr/lib/libIceUtil.so.32 (0x00002aaaabc77000)
        libSlice.so.32 => /usr/lib/libSlice.so.32 (0x00002aaaabdad000)
        libGlacier2.so.32 => /usr/lib/libGlacier2.so.32 (0x00002aaaabfec000)
        libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.6/libstdc++.so.6 (0x00002aaaac176000)
        libm.so.6 => /lib/libm.so.6 (0x00002aaaac366000)
        libc.so.6 => /lib/libc.so.6 (0x00002aaaac4ed000)
        libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.6/libgcc_s.so.1 (0x00002aaaac713000)
        libbz2.so.1 => /lib/libbz2.so.1 (0x00002aaaac81e000)
        libdl.so.2 => /lib/libdl.so.2 (0x00002aaaac92e000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00002aaaaca31000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

If I reverse these, using

java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_11-b03, mixed mode)

on the 64-bit server, I get similar exceptions:

Exception:
BasicStream.cpp:200: Ice::UnmarshalOutOfBoundsException:
protocol error: out of bounds during unmarshaling

at

0x40e56265 in __cxa_throw () from /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libstdc++.so.6
#0  0x40e56265 in __cxa_throw () from /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libstdc++.so.6
#1  0x4008f1f6 in IceInternal::BasicStream::skipSlice () from /homes/jmoore/lib/ice/lib/libIce.so.32
#2  0x40094517 in IceInternal::BasicStream::read () from /homes/jmoore/lib/ice/lib/libIce.so.32
#3  0x408e7456 in omero::model::__read () from /homes/jmoore/1.1b3/build/32bit/lib/32bit/libOMERO_common.so.0
#4  0x408e82a8 in omero::model::Experimenter::__read () from /homes/jmoore/1.1b3/build/32bit/lib/32bit/libOMERO_common.so.0
#5  0x40094fab in IceInternal::BasicStream::read () from /homes/jmoore/lib/ice/lib/libIce.so.32
#6  0x400953e4 in IceInternal::BasicStream::readPendingObjects () from /homes/jmoore/lib/ice/lib/libIce.so.32
#7  0x40a8cd8e in IceDelegateM::omero::api::IUpdate::saveAndReturnObject () from /homes/jmoore/1.1b3/build/32bit/lib/32bit/libOMERO_common.so.0
#8  0x40a59bee in IceProxy::omero::api::IUpdate::saveAndReturnObject () from /homes/jmoore/1.1b3/build/32bit/lib/32bit/libOMERO_common.so.0
#9  0x0806d416 in IceProxy::omero::api::IUpdate::saveAndReturnObject (this=0x809b4c8, obj=@0xbfca59cc) at ../include/OMERO/API.h:1848
#10 0x08051a9b in main (argc=1, argv=0xbfca5f84) at SDK.cpp:261

or

 	    Exception:
 	    generated2/OMERO/Model/Experimenter.cpp:549: Ice::UnexpectedObjectException:
 	    unexpected class instance of type `::omero::model::ExperimenterGroup'; expected instance of type `::omero::model::Experimenter'

(From user. No backtrace)


I'm currently trying to produce a self-contained test that shows this behavior, but haven't yet succeeded. Our setup includes object factories (as shown in the exceptions) as well as Glacier2 (always run on the server machine).

[matthew]

Sorry, there are no known issues. In order to help you further we need more information. Ideally proving a self-contained example that demonstrates the issue would be the best way for us to help solve your problem.

[joshmoore]

Would it make sense for compiling with -fPIC (as in Make.rules.Linux) to solve the issue? (Because it is for the moment) And if so, is there a suggested list of flags for building shared libraries which use the Ice libraries?

    CXXFLAGS="-m64 -fPIC -D_REENTRANT..."

Thanks,
Josh.

[bernard]

Hi Josh,

-fPIC is required for shared libraries, but not other objects. "-m64 -fPIC -D_REENTRANT" sounds reasonable to build shared libraries on x86_64; shared libraries that use Ice don't require anything special.

What is the Slice definition of this omero::api::IUpdate::saveAndReturnObject operation?

Does this problem occur every time you call this operation? Are you sure the Slice definitions used by your client and server are the same?

Of course, the best would be to reduce your application to a small test case and post it on these forums.

Cheers,
Bernard

[joshmoore]

The slice is fairly simple:

    interface IUpdate extends ServiceInterface
      { 
	void saveObject(omero::model::IObject obj) throws ServerError;
	void saveCollection(IObjectList objs) throws ServerError;     
	omero::model::IObject saveAndReturnObject(omero::model::IObject obj) throws ServerError;
	IObjectList saveAndReturnArray(IObjectList graph) throws ServerError;
	void deleteObject(omero::model::IObject row) throws ServerError;
      };

where IObject is the base class for all OMERO model objects:

module omero { 
  module model {     
    class IObject
    {
      omero::RLong          id;
      omero::model::Details details;
      bool loaded;
      void unload();
    };
  };
};

Things get more interesting here since omero::RLong is a concrete class, but we have convenience subclasses which get sliced, and due to the unload() method, all slice generated subclasses of IObject are abstract, which means we do a good deal of code generation ourselves, so I haven't been able to make a scaled down version which reproduces this issue.

Since it's working the moment, my assumption is that it was all a matter of the compilation flags. If the exceptions arise again, I'll post a reply.

Thanks for all the consideration,
Josh.

[bernard]

Hi Josh,

The most likely explanation for this error is a mismatch between the Slice definitions used by your client and server (the Slice definition for an IObject derived class).

In terms of reducing your application to a simple test case (if/when necessary), the first thing I'd remove is all the operations on your classes. This does not affect what goes on the wire but removes lots of code since all your mapped Slice classes would become concrete.

Best regards,
Bernard

[joshmoore]

Agreed, Bernard. Unfortunately I'm nearly positive that it's not a slice mismatch since we've seen the effect both from clean check outs on two machines, as well as in our distribution given to clients (without slice to generate from).

If it's of interest, I'll proceed with a trivial example I was working on, and see if the fPIC/shared library combo will cause similar odd exceptions.

~J.