IcePy can not handle buffers with longs (I64)

[DaVinci79]

Hi everyone,

For a project we are working on we wish to send a numpy array as quick as possible. We match the numpy array datatype to the type defined in our Slice definition, and wrap it in a buffer. Now this works fine for doubles and ints, and it is awesomely quick. However, for longs (64bit signed) it gives a "expected sequence value" exception. Now we've traced it down to this point in IcePy (Types.cpp, line: 1066).:

void
IcePy::SequenceInfo::marshalPrimitiveSequence(const PrimitiveInfoPtr& pi, PyObject* p, const Ice::OutputStreamPtr& os)
{
    //
    // For most types, we accept an object that implements the buffer protocol
    // (this includes the array.array type).
    //
    const void* buf = 0;
    Py_ssize_t sz;
    if(PyObject_AsReadBuffer(p, &buf, &sz) == 0)
    {
        const Ice::Byte* b = reinterpret_cast<const Ice::Byte*>(buf);
        switch(pi->kind)
        {
        case PrimitiveInfo::KindBool:
        {
            os->writeBoolSeq(reinterpret_cast<const bool*>(b), reinterpret_cast<const bool*>(b + sz));
            break;
        }
        // [ ... ]

        case PrimitiveInfo::KindLong:  // ???
        {
            PyErr_Format(PyExc_ValueError, STRCAST("expected sequence value"));
            throw AbortMarshaling();
        }
        case PrimitiveInfo::KindFloat:
        {
            os->writeFloatSeq(reinterpret_cast<const Ice::Float*>(b), reinterpret_cast<const Ice::Float*>(b + sz));
            break;
        }
        case PrimitiveInfo::KindDouble:
        {
            os->writeDoubleSeq(reinterpret_cast<const Ice::Double*>(b),
                               reinterpret_cast<const Ice::Double*>(b + sz));
            break;
        }
        // [ ....]

As you can see it accepts every primitive kind, except long. Now we modified this piece of code to accepts longs, exactly the same as with the other primitives, and it seems to work fine.

However, we do not wish to keep a separate version of Ice, and besides I do not know if there is a deeper reason why this is not possible. I would not like to bump into odd failures in the future because we didn't know what we where doing .

Regards,
Vincent

[mes]

Hi,

The only reason we don't support the buffer protocol for the Slice long type is because Python's array module doesn't support it. However, if numpy's array type supports 64-bit integers, we'll fix IcePy accordingly.

Thanks,
Mark

[DaVinci79]

numpy can handle those, yes. If that could be included in the future releases that would be great!

Thanks,
Vincent

[joshmoore]

+1 I'll add a workaround for the moment, but longs are a sizeable portion of what we'd be serializing from numpy.

[joshmoore]

Also, if possible supporting byte sequences would be beneficial:

In [80]: numpy.array([1,2,3], dtype=tables.UInt8Atom())
Out[80]: array([1, 2, 3], dtype=uint8)

In [81]: numpy.array([1,2,256], dtype=tables.UInt8Atom())
Out[81]: array([1, 2, 0], dtype=uint8)

...

In [83]: a = numpy.array([1,2,3], dtype=tables.UInt8Atom())

In [84]: print a
[1 2 3]

In [85]: omero.grid.MaskColumn(None, None, None, None, None, None, None, None, None, [a])
Out[85]: 
object #0 (::omero::grid::MaskColumn)
{
    name = None
    description = None
    imageId = {}
    theZ = {}
    theT = {}
    x = {}
    y = {}
    w = {}
    h = {}
    bytes = 
    {
        [0] = 
        {
            [0] = <invalid value - expected byte>
            [1] = <invalid value - expected byte>
            [2] = <invalid value - expected byte>
        }
    }
}

...

In [90]: a
Out[90]: array([   1,    2, -128], dtype=int8)

In [91]: a = numpy.array([1,2,127], dtype=tables.Int8Atom())

In [92]: a
Out[92]: array([  1,   2, 127], dtype=int8)

In [93]: omero.grid.MaskColumn(None, None, None, None, None, None, None, None, None, [a])
Out[93]: 
object #0 (::omero::grid::MaskColumn)
{
    name = None
    description = None
    imageId = {}
    theZ = {}
    theT = {}
    x = {}
    y = {}
    w = {}
    h = {}
    bytes = 
    {
        [0] = 
        {
            [0] = <invalid value - expected byte>
            [1] = <invalid value - expected byte>
            [2] = <invalid value - expected byte>
        }
    }
}

Or do the various ranges for Ice::Byte (-128,127 or 0, 255) make this difficult/impossible?

[mes]

These issues will be addressed in Ice 3.4

Regards,
Mark

[joshmoore]

Hi Mes, that's great to hear.

Not knowing exactly what the plans are, I'll add more feedback about what I'm running into with 3.3 here, but will re-test everything with 3.4.

Most recently we've encountered a 64bit issue. For the following slice:

module b {
  sequence<int> IntArray;
  class c {
    IntArray i;
  };
};

numpy arrays of type 'i4' are considered invalid.

import b
import os
import Ice
import numpy

os.system("uname -a")
print "Ice", Ice.stringVersion()

def test(name, array):
    d = b.c()
    d.i = array
    s = str(d)
    i = s.find("invalid")
    print name, "... ", i<0 and "Ok" or "Failed"

test("Python array", [1,2])
arr4 = numpy.array([1,2], dtype='i4')
test("numpy array i4", arr4)
test("numpy array i4 tolist", arr4.tolist())

arr8 = numpy.array([1,2], dtype='i8')
test("numpy array i8", arr8)
test("numpy array i8 tolist", arr8.tolist())

works on 32 bit mac:

Darwin mac 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386
Ice 3.3.0
Python array ...  Ok
numpy array i4 ...  Ok
numpy array i4 tolist ...  Ok
numpy array i8 ...  Ok
numpy array i8 tolist ...  Ok

and slightly less so on 32bit Linux:

Linux valewalker 2.6.24-gentoo-r2 #2 SMP Sat Feb 23 10:40:02 GMT 2008 i686 Intel(R) Xeon(TM) CPU 1.60GHz GenuineIntel GNU/Linux
Ice 3.3.1
Python array ...  Ok
numpy array i4 ...  Ok
numpy array i4 tolist ...  Ok
numpy array i8 ...  Failed
numpy array i8 tolist ...  Ok

but on a 64bit Linux machine:

Linux warlock 2.6.29-gentoo-r5 #1 SMP Wed Jul 8 21:59:17 BST 2009 x86_64 AMD Opteron(tm) Processor 252 AuthenticAMD GNU/Linux
Ice 3.3.1
Python array ...  Ok
numpy array i4 ...  Failed
numpy array i4 tolist ...  Ok
numpy array i8 ...  Ok
numpy array i8 tolist ...  Ok

If possible, it would be good to not have to manually convert numpy arrays based on whether the local machine is 32 or 64 bit.

Thanks, ~J.

[mes]

Hi Josh,

I ran your test in our Ice 3.4 development tree on Linux (x86 & x64) and it works correctly. Note however that the range of your array elements must still comply with the limits of the Slice data types:

>>> d.i = numpy.array([1,2,-9223372036854775808], dtype='i8')
>>> d.i
array([1, 2, -9223372036854775808], dtype=int64)
>>> d
object #0 (::b::c)
{
    i = 
    {
        [0] = 1
        [1] = 2
        [2] = <invalid value - expected int>
    }
}

Regards,
Mark