Page 1 of 1

(how to) deal with large amounts of (key,value) - updated for 7.0x

Posted: 4 Apr 2006 19:22
by Gildas Menier
For the management of key->values in Visual Prolog, you may use factsDB, chainDB or some sql based solution. Each solution has benefits and drawbacks. For some reasons, I had to find another solution.

Mikio Hirabayashi has built a very nice API called 'QDBM'.

http://qdbm.sourceforge.net/

see also http://qdbm.sourceforge.net/benchmark.pdf

QDBM is well known in the Unix world, but has been ported to Win32. Practically, it is a disk-based Hashtable + Binary tree + inverted list package. QDBM offers many solutions called 'depot', 'curia', 'villa', 'odeum' and is well worth a look if you want to manage large amount of data.

In the enclosed project, you'll find a small 'depot' wrapper for Visual Prolog.

The example creates 1 000 000 (key, values) and then ask for 1 000 000 keys at random. All the key/values are stored on disk in a single file.

Using a collection of instances of the visual prolog class 'depot', you can manage a very large hashtable on disk.

I do not provide a full documentation since the real documentation can be found on the above web site. Only the port of depot is functional in this lib.

I tried to simplify the use of this lib as much as possible, and tried not to get too far away from the original API syntax :

Code: Select all

NewDepot = depot::new("dtest"), % create a depot db NewDepot:dbopen("create"), % open it (create mode) NewDepot:dbput_string(Key, Value), % put a key,value NewDepot:dbclose(), % close the file
and you can access the db as follows:

Code: Select all

ReadDepot = depot::new("dtest"), % ReadDepot:dbopen("read"), % open it in read mode Value = ReadDepot:dbget_string(Key), % tries to find the value for a key, fails if no value ReadDepot:dbclose(), %
More predicates are provided. Please, have a look on the documentation of QDBM/depot and on the depot.pro file.

The project doesn't include a copy of qdbm (because of the allowed posting size); I encourage you to download the most up-to-date version on the site above. It is provided under the GNU Lesser GPL.
Just copy the three following files into your exe directory :

libiconv-2.dll
mgwz.dll
qdbm.dll

you will find the package there :

http://qdbm.sourceforge.net/win/

Best Regards

Gildas

Posted: 7 Jul 2007 19:09
by Gildas Menier
Updated for 7.0x

regards

Gildas

Posted: 29 Aug 2007 13:31
by pegesund
Hi, this is really fast :D

Have you had a look at the Villa-api as well? I would be very interested in some code...

Cheers,

Petter

Posted: 8 Jan 2008 23:31
by Thomas Linder Puls
This has been copied to the wiki: 3rd:QDBM.