Improved searchtools in freeze

[Nis Baggesen]

Wow - I wasn't expecting such an immediate reply. It was just some wishful thinking on my part.

Originally Posted by bernard

The search features offered by Freeze are constrained by the underlying database, Berkeley DB. Freeze could offer searches on multiple indices (with AND and OR expressions) but since there is no corresponding feature in Berkeley DB, performance-wise it would be like doing the individual searches and combining the results with intersections and unions. Actually it would probably be slower since Freeze would have to parse this expression.

Fair enough. I'm unfamiliar with the inner workings of BerkeleyDb. I just recalled that there was a MySQL implementation built on it, so I figured it wasn't impossible. Ofcourse that implementation might do all its work in RAM and simply use BerkeleyDb for storage.

Just to fully satisfy my curiosity - You have previously mentioned that you where considering implementing an SQL version of Freeze. Would that support more advanced searching?

My hunch is that it should be possible, but it might require a more complicated translation from IceObject to database schema. And I assume you don't want to go all-out and make and OODB implementation. Ofcourse the specification of indices might be a usefull middleground.

Originally Posted by bernard

Range-searches on keys and indexed members using Berkeley DB seems more feasible. We would mainly need to specify comparison functions (Db::set_bt_compare) to sort the corresponding Ice-encoded binary strings. Is this the kind the non-equality search you have in mind?

That was pretty much exactly the kind of 'non-equality' search I had in mind. Would this allow for custom sorting functions (For more advanced ice structures), or would it be based on the compassiron fuctions that er already automatically defined for ice objects? Or would it be restricted to indices on the basic types?

[bernard]

MySQL supports several storage engines, and Berkeley DB is one of them; however I don't know if/how MySQL uses Berkeley DB features such as secondary indices.

A new implementation of Freeze using a relational database (MySQL, PostgreSQL, Oracle, ODBC ... there are many choices) could store everything in Berkeley DB-like maps (sequence<byte> to sequence<byte>): this would be simpler to implement but would not help for queries. A more reasonable implementation would let you map Slice class data members to table columns, and then it could provide powerful SQL searches.

Regarding the range searches, it would be just for keys and members for which you've defined an index (otherwise Berkeley DB can't help).
Custom sorting? Your indexed member's type is Int and you want your own Int order? This would be strange! For indexed members with more complicated types, the data to sort is instances of this type stored as binary strings (encoded with the Ice encoding). Unmarshalling for each comparison is feasible (and flexible) but expensive; I don't know if there is a better alternative.

Cheers,
Bernard

[Nis Baggesen]

Originally Posted by bernard

MySQL supports several storage engines, and Berkeley DB is one of them; however I don't know if/how MySQL uses Berkeley DB features such as secondary indices.

I wasn't expecting you to know the inner workings of various MySQL implementations. I was just sort of thinking out loud.

Originally Posted by bernard

A new implementation of Freeze using a relational database (MySQL, PostgreSQL, Oracle, ODBC ... there are many choices) could store everything in Berkeley DB-like maps (sequence<byte> to sequence<byte>): this would be simpler to implement but would not help for queries. A more reasonable implementation would let you map Slice class data members to table columns, and then it could provide powerful SQL searches.

Obviously the latter would be preferable from my standpoint. And I guess we could just implement that ourselves. But there was no reason not to but it on the wishlist.

Originally Posted by bernard

Regarding the range searches, it would be just for keys and members for which you've defined an index (otherwise Berkeley DB can't help).
Custom sorting? Your indexed member's type is Int and you want your own Int order? This would be strange!

Yes, it wouldn't be all that typical. But there are alternative orderings that do make sense for bytepatterns - They could be ordered to reduce Hamming-distance and the like. Could be useful if the int was used to store flagbits. Likewise I could imagine several way of ordering strings - As practically anything could be encoded in a string. However I was thinking about structured types when I talked about custom sorters.

[bernard]

Originally Posted by Nis Baggesen

Obviously the latter would be preferable from my standpoint. And I guess we could just implement that ourselves. But there was no reason not to but it on the wishlist.

You're welcome to submit any wish (and please continue to do so); realistically, for a significant effort like a new Freeze implementation or support for a new language mapping, it is more likely to happen through a commercial customer with specific requirements/a specific target database, than through popular demand.

Range searches are much simpler ... just needs enough wishes

Cheers,
Bernard