Frequently Asked Questions

How do I transfer a file with Ice?

The easiest approach is to simply transfer the entire file in a single RPC. Here are the Slice definitions for a simple file store:

// Slice
sequence ByteSeq;
interface FileStore
{
    ByteSeq get(string name);
    void put(string name, ByteSeq bytes);
};

This interface allows clients to read and update the contents of remote files by specifying a file name. Quite often, this approach to file transfer is entirely adequate. Unless the files get quite large (in the tens of megabytes), they can be transferred in a single RPC without problems. If you want to transfer files larger than a megabyte, you will need to increase the setting of the Ice.MessageSizeMax property though. By default, this property limits the maximum size of Ice messages to 1MB so, to transfer larger files, you will need to increase the default setting. (See also this FAQ, which discusses this property.)

If you need to transfer files that are large, you will likely need to use a different approach. The reason is that, once a single RPC transfers too much data, your machine is likely to start thrashing because all of the data for the RPC is buffered in memory before it is made available to the receiving end. In addition (unless you use zero-copy, as explained by Brent Eagles and Dwayne Boone in issue 13 of Connections), during unmarshaling, the Ice run time temporarily requires roughly twice the memory for the data in a request: once to store the data in a transport buffer, and once to make it available to the application as a byte sequence (such as a vector in C++ or a collection in C#).

For larger files, there are various ways to tackle the problem. One of the simplest and most effective is to read the file in chunks and to have the interface mimic the UNIX read and write system calls: 

interface FileStore
{
	ByteSeq read(string name, int offset, int num);
	void write(string name, int offset, ByteSeq bytes);
};

The read operation requests a number of bytes starting at the specified offset. The operation returns either the number of bytes that were requested, or a sequence containing fewer bytes. In the latter case, the server may have limited the number of bytes it returned to its setting of MessageSizeMax (which may be smaller than the client’s), so reading fewer bytes than requested does not indicate end-of-file (as it does in UNIX). Instead, the client must keep reading until it receives an empty sequence. To retrieve a file in chunks, the client simply keeps reading some number of bytes, starting at offset zero, and adds the number of bytes in the returned sequence to the offset for the next call to read, until read returns an empty sequence.

The write operation writes the byte sequence starting at the specified offset. If the specified number bytes cannot be transferred because the client’s setting of MessageSizeMax is too low, the operation raises a MemoryLimitExeption; if the operation fails because the server’s setting of MessageSizeMax is too low, the operation raises an UnknownException. If you prefer more descriptive exceptions, you can add an additional operation that allows the client to obtain the setting of the largest sequence that a server can handle (which will be smaller than the server’s MessageSizeMax setting), and make the server throw an exception such as SizeTooLarge if the client attempts to transfer too many bytes at once.

Finally, for high throughput applications, you can use a more sophisticated scheme, such as the one used by IcePatch2: instead of transferring one chunk of a file at a time, IcePatch2 uses concurrent calls to avoid idle time while chunks are in transfer. However, that approach is beyond the scope of this FAQ—you can have a look at the IcePatch2 source code if you are interested in seeing how this approach works.
Copyright © 2008 ZeroC, Inc.