(how to) deal with large amounts of (key,value) - updated for 7.0x

Share Tips, Code Samples, etc. with the Visual Prolog community.
Gildas Menier
VIP Member
Posts: 78
Joined: 8 Jun 2004 23:01

(how to) deal with large amounts of (key,value) - updated for 7.0x

Unread post by Gildas Menier » 4 Apr 2006 19:22

For the management of key->values in Visual Prolog, you may use factsDB, chainDB or some sql based solution. Each solution has benefits and drawbacks. For some reasons, I had to find another solution.

Mikio Hirabayashi has built a very nice API called 'QDBM'.

http://qdbm.sourceforge.net/

see also http://qdbm.sourceforge.net/benchmark.pdf

QDBM is well known in the Unix world, but has been ported to Win32. Practically, it is a disk-based Hashtable + Binary tree + inverted list package. QDBM offers many solutions called 'depot', 'curia', 'villa', 'odeum' and is well worth a look if you want to manage large amount of data.

In the enclosed project, you'll find a small 'depot' wrapper for Visual Prolog.

The example creates 1 000 000 (key, values) and then ask for 1 000 000 keys at random. All the key/values are stored on disk in a single file.

Using a collection of instances of the visual prolog class 'depot', you can manage a very large hashtable on disk.

I do not provide a full documentation since the real documentation can be found on the above web site. Only the port of depot is functional in this lib.

I tried to simplify the use of this lib as much as possible, and tried not to get too far away from the original API syntax :

Code: Select all

NewDepot = depot::new("dtest"), % create a depot db NewDepot:dbopen("create"), % open it (create mode) NewDepot:dbput_string(Key, Value), % put a key,value NewDepot:dbclose(), % close the file
and you can access the db as follows:

Code: Select all

ReadDepot = depot::new("dtest"), % ReadDepot:dbopen("read"), % open it in read mode Value = ReadDepot:dbget_string(Key), % tries to find the value for a key, fails if no value ReadDepot:dbclose(), %
More predicates are provided. Please, have a look on the documentation of QDBM/depot and on the depot.pro file.

The project doesn't include a copy of qdbm (because of the allowed posting size); I encourage you to download the most up-to-date version on the site above. It is provided under the GNU Lesser GPL.
Just copy the three following files into your exe directory :

libiconv-2.dll
mgwz.dll
qdbm.dll

you will find the package there :

http://qdbm.sourceforge.net/win/

Best Regards

Gildas
Attachments
demo_largedb.zip
You also have to download qdbm for windows (see the post)
(97.13 KiB) Downloaded 1182 times
Last edited by Gildas Menier on 7 Jul 2007 19:10, edited 1 time in total.

Gildas Menier
VIP Member
Posts: 78
Joined: 8 Jun 2004 23:01

Unread post by Gildas Menier » 7 Jul 2007 19:09

Updated for 7.0x

regards

Gildas
Attachments
demo_largedb7.zip
(98.03 KiB) Downloaded 1125 times

pegesund
Posts: 3
Joined: 19 May 2005 16:55

Unread post by pegesund » 29 Aug 2007 13:31

Hi, this is really fast :D

Have you had a look at the Villa-api as well? I would be very interested in some code...

Cheers,

Petter

User avatar
Thomas Linder Puls
VIP Member
Posts: 2384
Joined: 28 Feb 2000 0:01

Unread post by Thomas Linder Puls » 8 Jan 2008 23:31

This has been copied to the wiki: 3rd:QDBM.
Regards Thomas Linder Puls
PDC

Post Reply