Java client C++ server fails on complex structure, but C++ client C++ server works

[Jon S]

Hi,

I wrote a C++ Server and Client, all worked fine as below

My .getStatus() and a more complex .recvRetQueryData(long,long,long) worked fine.

                    case 'g': {
                        // C++ (CLient.cpp) works
                        int b = (int)fasBitComm->getStatus();
                        cout << "client: got status '" << b << "'" << endl;
                        break;
                    }
                    case 'n': {
                        // C++ (CLient.cpp) works
                        int begrow = 0;
                        int maxfetch = 2000;
                        Demo::RetQueryData qdata = 
                             fasBitComm->recvRetQueryData(-1,begrow,maxfetch);
                        break;
                    }

Then I tried to make a Java Client (used ant and all compiled fine) The simple method .getStatus() that returns a byte works just fine BUT the more complex .recvRetQueryData(long,long,long) fails which returns a 'structure' that contains a variable that is a Sequence of another structure.

                        case 'g':
                        {
                            // JAVA (Client.java) works
                            System.out.print("client 'g': \n");
                            byte b = fbprot.getStatus();
                            System.out.print("was : "+b+ "\n");
                            break;
                        }
                        case 'n':
                        {
                            // JAVA (Client.java) fails
                            System.out.print("client 'n': \n");
                            long cn = -1L; long o = 0L; long m = 1000L;
                            RetQueryData rqd = fbprot.recvRetQueryData(cn,o,m);
                            break;
                        }

I end up getting an error as follows .. sort of stumped on what to do except
slowly build up my entire *.ice file again.

Ice.EncapsulationException
    reason = (null)
        at IceInternal.BasicStream.endReadEncaps(BasicStream.java:317)
        at Demo._FbProtDelM.recvRetQueryData(_FbProtDelM.java:138)
        at Demo.FbProtPrxHelper.recvRetQueryData(FbProtPrxHelper.java:261)
        at Demo.FbProtPrxHelper.recvRetQueryData(FbProtPrxHelper.java:236)
        at Client.run(Client.java:137)
        at Ice.Application.doMain(Application.java:200)
        at Ice.Application.main(Application.java:180)
        at Ice.Application.main(Application.java:118)
        at Client.main(Client.java:227)

My *.ice file is below "FbProt.ice" basically I am trying to return a RetQueryData to the Client which contains a Sequence of RetDataCol ( or a set of arrays or vectors that represent a column store where only one data type is active in any RetDataCol). I think this is the way to code up a union in ICE. As I said it worked in C++ but seems not to be happy in Java.

#ifndef FBPROT_ICE
#define FPPROT_ICE

#include <Ice/BuiltinSequences.ice>

module Demo {

        /** define column stor types **/
        enum RetDataType { eBoolSeq, eByteSeq, eShortSeq, eIntSeq, eLongSeq, eFloatSeq, eDoubleSeq, eStringSeq, eEOL };

        struct RetDataCol {

                string          mName;        // name of data
                RetDataType     mRetDataType; // must indicate which sequence this packet holds

                /** must have data in one of the below **/
                Ice::BoolSeq    mBoolSeq;     // can be empty
                Ice::ByteSeq    mByteSeq;     // can be empty
                Ice::ShortSeq   mShortSeq;    // can be empty
                Ice::IntSeq     mIntSeq;      // can be empty
                Ice::LongSeq    mLongSeq;     // can be empty
                Ice::FloatSeq   mFloatSeq;    // can be empty
                Ice::DoubleSeq  mDoubleSeq;   // can be empty
                Ice::StringSeq  mStringSeq;   // can be empty
        };

        sequence<RetDataCol> RetDataColSeq;

        /** define query return a squence of a single column or all columns **/
        struct RetQueryData {
                byte            mStatus;      // 0 failue, > 1 success,
                long            mBegRowId;    // offset into RowId set
                long            mNumRows;     // number of rows returned
                long            mMaxRows;     // total number of rows, (done if mBegRowId+mNumRows == mMaxRows)
                long            mPacketSz;    // approx size of non-compresses data packet

                RetDataColSeq      mRetDataCols;
        };


        interface FbProt
        {
                byte getStatus();
                RetQueryData recvRetQueryData(long col, long beg, long max);
                void shutdown();
        };
};
#endif

Thanks in Advance,

Jon S

[bernard]

Hi Jon,

Welcome to our forums!

Your recvRetQueryData operation should work fine in any language mapping.

Could you double-check that you built your client with exactly the same Slice definitions as your server? If the Slice definitions are out of sync - e.g. your Java client is using the generated code for an older version of your Slice - you could get this type of exception.

If everything is in sync, then it would be helpful if you could post a C++ server and Java client that we can use to reproduce this problem.

Best regards,
Bernard

[Jon S]

Bernard,

Your recvRetQueryData operation should work fine in any language mapping.

Could you double-check that you built your client with exactly the same Slice definitions as your server? If the Slice definitions are out of sync - e.g. your Java client is using the generated code for an older version of your Slice - you could get this type of exception.

If everything is in sync, then it would be helpful if you could post a C++ server and Java client that we can use to reproduce this problem.

It is indeed a BUG in the slice compiler definitely in the the 'slice2java' I compare a bad definition to a good definition.

testdev# diff /tmp/FbProt.ice.bad /tmp/FbProt.ice.good
20c20
<               byte               mStatus;  // comment
---
>               byte               mStatus;
testdev#

Consider the simplified slice definition with the single '//' comment this fails to make any JAVA code for mStatus, if I remove the '//' comment code is generated and it works. The failing slice definition.

/*
*  BAD 'slice2java' FAILS 'FbProt.ice.bad' THIS IS DUE TO A
*  SINGLE '// comment' PUT IN MY 'FbProt.ice' file.
*
*  Note, the one comment in 'struct RetQueryData' after 'byte mStatus;'
*  will break the slice compiler.  I think the lexer / parser needs to
*  be fixed - IMHO stipping comments is a basic operation.
*
*  testdev# slice2java -I/opt/Ice-3.4.1/slice FbProt.ice --output-dir generated
*
*  testdev# grep mStatus generated/Demo/\*
*      << nothing ... no output for mStatus in any Java Class>>
*/

#include <Ice/BuiltinSequences.ice>

module Demo {

        enum RetDataType { eShortSeq, eIntSeq, eEOL };

        struct RetDataCol {
                string          mName;
                RetDataType     mRetDataType;
                Ice::ShortSeq   mShortSeq;
                Ice::ShortSeq   mIntSeq;
        };

        sequence<RetDataCol> RetDataColSeq;

        struct RetQueryData {
                byte               mStatus;  // comment
                RetDataColSeq      mRetDataCols;
        };

        interface FbProt
        {
                byte getStatus();
                RetQueryData recvRetQueryData(long col, long beg, long max);
                void shutdown();
        };
};

#endif

Okay so I remove the '//' comment and everything works. The working slice definition

#ifndef FBPROT_ICE
#define FPPROT_ICE

/*
* BACKUP 'FbProt.ice.good' WORKS ALL I DID WAS REMOVED THE COMMENT
* FROM 'FbProt.ice'
*
* Note, the one comment in 'struct RetQueryData' after 'byte mStatus;'
* this bs removed the slice compiler now works.
*
* testdev# slice2java -I/opt/Ice-3.4.1/slice FbProt.ice --output-dir generated
*
* testdev# grep mStatus generated/Demo/\*
*     << expected data found in Java files for 'mStatus' >>
*     generated/Demo/RetQueryData.java:    public byte mStatus;
*     generated/Demo/RetQueryData.java:    public RetQueryData(byte mStatus, RetDataCol[] mRetDataCols)
*     generated/Demo/RetQueryData.java:        this.mStatus = mStatus;
*     generated/Demo/RetQueryData.java:            if(mStatus != _r.mStatus)
*     generated/Demo/RetQueryData.java:        __h = 5 * __h + (int)mStatus;
*     generated/Demo/RetQueryData.java:        __os.writeByte(mStatus);
*     generated/Demo/RetQueryData.java:        mStatus = __is.readByte();
*/

#include <Ice/BuiltinSequences.ice>

module Demo {

        enum RetDataType { eShortSeq, eIntSeq, eEOL };

        struct RetDataCol {
                string          mName;
                RetDataType     mRetDataType;
                Ice::ShortSeq   mShortSeq;
                Ice::ShortSeq   mIntSeq;
        };

        sequence<RetDataCol> RetDataColSeq;

        struct RetQueryData {
                byte               mStatus;
                RetDataColSeq      mRetDataCols;
        };

        interface FbProt
        {
                byte getStatus();
                RetQueryData recvRetQueryData(long col, long beg, long max);
                void shutdown();
        };
};
#endif

Just to be perfectly clear this is the only true difference (I removed the comment blocks /* ... */ from top of the two files. That is how I documented the issue in my opening comment.

testdev# diff /tmp/FbProt.ice.bad /tmp/FbProt.ice.good
20c20
<               byte               mStatus;  // comment
---
>               byte               mStatus;
testdev#

Thus there is really no need to provide 'C++', 'JAVA', or other items used to build my programs.

Regards,

Jon S

[bernard]

Hi Jon,

It's great you were able to isolate this problem.

There is unfortunately such a bug in mcpp 2.7.2, and we include a patched mcpp in all our binary distributions. You can also retrieve this patch (in source form) from our third-party source archive: ZeroC - Download

Which platform / build are you using?

Best regards,
Bernard

[Jon S]

Bernard,

I am using Ice-3.4.1.tar.gz (downloaded it on 2011-05-23) and built/installed the source. As far as "bug in mcpp 2.7.2" somehow the first source bundle doesn't cover that ?

OpenIndiana is my operating system (consider it the same OpenSolaris). SO let me get this straight the binary "Ice-3.4.1-bin-solaris-x86.tar.gz (x86/x64)" has updated code compared to the source "Ice-3.4.1.tar.gz"

Okay on the download page we have under the main hghlighted section "Source Distributions" there was the source download I used:

http://www.zeroc.com/download/Ice/3.4/Ice-3.4.1.tar.gz

and later on the page under "Third-Party Source Code" (why would I want a 'ThirdParty' product I thought - why would I notice it)

http://www.zeroc.com/download/Ice/3....s-3.4.1.tar.gz

FYI, I typically take the first main generic source distro over sme TBD third party item, so you are saying I should have built my full source from the later link - right ?

Or are you saying that I only need to apply "patch.mcpp.2.7.2" (from the 30+ meg) Third Party link to my source and rebuild - note this 3K patch is dated way back to 1/14/2010 thats pretty old (how often do you release bug fixes)?

Just for clarity .... what should I use ... what should I patch ... will I be fine if I don't use comments ?

Note (form the source bundle I used - first link above)
1. It looks like Ice-3.4.1.tar.gz is dated 2011-04-23 - pretty new.
2. It looks like mcpp-2.7.2.tar.gz is dated 2008-11-30 - pretty old.

testdev# ls -l /opt/SANDBOX/ice
total 6804
drwxrwxr-x 14 500 500 22 2011-05-23 14:17 Ice-3.4.1/
-rw-r--r-- 1 root root 5263510 2011-04-23 15:22 Ice-3.4.1.tar.gz
-rw-r--r-- 1 root root 7141 2011-05-25 08:47 ice.qsi
drwxrwxrwx 13 1000 1000 31 2011-05-23 13:33 mcpp-2.7.2/
-rw-r--r-- 1 root root 1533457 2008-11-30 04:24 mcpp-2.7.2.tar.gz
testdev#

My environment is as follows:

testdev# uname -a
SunOS testdev 5.11 oi_148b i86pc i386 i86pc Solaris


testdev# cat /etc/release
OpenIndiana Development oi_148b X86 (powered by illumos)
Copyright 2010 Oracle and/or its affiliates. All rights reserved.
Use is subject to license terms.
Assembled 11 April 2011
lab10# g++ --version
g++ (GCC) 3.4.3 (csl-sol210-3_4-20050802)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


testdev# gmake --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-pc-solaris2.11


testdev# cpp --version
cpp (GCC) 3.4.3 (csl-sol210-3_4-20050802)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


testdev# m4 --version
m4 (GNU M4) 1.4.12
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Rene' Seindal.
testdev#

Regard,

Jon S

[Jon S]

I'm sorry

mcpp-2.7.2.tar.gz if not from ZeroC I think I understand what is going on

Regards,

Jon

[bernard]

Hi Jon,

The first step when building Ice for sources is to obtain all the third-party packages Ice depends on.

That's straightforward on Windows, since we provide an installer with all these dependencies (in binary form). Likewise on Linux (RHEL and SLES), we use rpms that come with these systems and provide a few rpms that don't: rpms for Berkeley DB 4.8 and a rpm for mcpp (mcpp-devel).

For other platforms, you may be able to find these dependencies in binary form.

However you need to be extra careful with mcpp, and make sure your mcpp library includes all our patches.

Overall, the solution here is to:
- build mcpp 2.7.2 + patch, using the mcpp patch in our third-party source distribution
- rebuild Ice using this patched mcpp library.

If you use a static mcpp library, the patched mcpp objects will be incorporated into libSlice. I would however recommend to rebuild Ice entirely as bugs in the non-patched mcpp could affect other Ice components.

Then, everything should work well on OpenIndiana .

All the best,
Bernard

[Jon S]

Bernard,

Updating the static and shared libraries of mcpp 2.7.2 + patch, using the mcpp patch in our third-party source distribution fixed the "so called" comment issue (without touching Ice).


Silly me I though why not get the latest and greatest mcpp from mcpp -- a portable C preprocessor with Validation Suite - this happens to be the same as on your site, but I still needed the patch.

Then as per your advice I rebuilt all the Ice items to be safe


Thanks so much for following up with me

Jon

[grembo]

Hi,

since I had a similar issue recently (see
Mishap in slice compilers) I would suggest two things:

1) You contribute to getting a new release of mcpp done (all the details had been discussed in the thread above, but then you stopped replying). The developer is more than happy to agree to get it in there, but simply doesn't have the resources right now. (I copy and pasted Kiyoshi's response again at the end of this post).

2) Incorporate a simple check routine in the slice2xxx compilers to verify if a valid version of mcpp is used (either by a version check or better by compiling a simple well known snippet of Ice code and check if the result is what you expect it to be). This should be extremely easy to accomplish.

The reason why I think this is really critical is that in case a broken compiler is used, potentially terrible mistakes can happen just by adding a simple comment to a struct member (especially in dynamic/non-compiled languages)

just my 2 cents

Kiyoshi wrote:
"Hello,

I'm very sorry, but I have no time for mcpp now. I think the Zeroc
patch is correct, and has no undesirable side effect. So, if someone
make an mcpp package applying the patch, and commit it to FreeBSD or any
other system, I will agree with it.

--
Kiyoshi Matsui, the maintainer of mcpp"