| LMDB_File(3pm) | User Contributed Perl Documentation | LMDB_File(3pm) |
LMDB_File - Tie to LMDB (OpenLDAP's Lightning Memory-Mapped Database)
# Simple TIE interface, when you're in a rush
use LMDB_File;
$db = tie %hash, 'LMDB_File', $path;
$hash{$key} = $value;
$value = $hash{$key};
each %hash;
keys %hash;
values %hash;
...
# The full power
use LMDB_File qw(:flags :cursor_op);
$env = LMDB::Env->new($path, {
mapsize => 100 * 1024 * 1024 * 1024, # Plenty space, don't worry
maxdbs => 20, # Some databases
mode => 0600,
# More options
});
$txn = $env->BeginTxn(); # Open a new transaction
$DB = $txn->OpenDB( { # Create a new database
dbname => $dbname,
flags => MDB_CREATE
});
$DB->put($key, $value); # Simple put
$value = $DB->get($key); # Simple get
$DB->put($key, $value, MDB_NOOVERWITE); # Don't replace existing value
# Work with cursors
$cursor => $DB->Cursor;
$cursor->get($key, $value, MDB_FIRST); # First key/value in DB
$cursor->get($key, $value, MDB_NEXT); # Next key/value in DB
$cursor->get($key, $value, MDB_LAST); # Last key/value in DB
$cursor->get($key, $value, MDB_PREV); # Previous key/value in DB
$DB->set_compare( sub { lc($a) cmp lc($b) } ); # Use my own key comparison function
NOTE: This document is still under construction. Expect it to be incomplete in places.
LMDB_File is a Perl module which allows Perl programs to make use of the facilities provided by OpenLDAP's Lightning Memory-Mapped Database "LMDB".
LMDB is a Btree-based database management library modeled loosely on the BerkeleyDB API, but much simplified and extremely fast.
It is assumed that you have a copy of LMBD's documentation at hand when reading this documentation. The interface defined here mirrors the C interface closely but with an OO approach.
This is implemented with a number of Perl classes.
A LMDB's environment handler (MDB_env* in C) will be wrapped in the LMDB::Env class.
A LMDB's transaction handler (MDB_txn* in C) will be wrapped in the LMDB::Txn class.
A LMDB's cursor handler (MDB_cursor* in C) will be wrapped in the LMDB::Cursor class.
A LMDB's Database handler (MDB_dbi in C) will be exposed as a simple integer, but because in LMDB all Database operations needs both a Transaction and a Database handler, LMDB_File provides you a convenient "LMDB_File" object that encapsulates both and mimic the syntax of other *_File modules.
In the C API, most functions return 0 on success and an error code on failure.
In this module, when a function fails, the package variable $die_on_err controls the course of action. When $die_on_err is set to TRUE, this causes LMDB_File to "die" with an error message that can be trapped by an "eval { ... }" block.
When FALSE, the function will return the error code, in this case you should check the return value of any function call.
By default $die_on_err is TRUE.
Regardless of the value of $die_on_err, the code of the last error can be found in the package variable $last_err.
This class wraps an opened LMDB environment.
At construction time, the environment is created, if it does not exist, and opened.
When you are finished using it, in the C API you must call the "mdb_env_close" function to close it and free the memory allocated, but in Perl you simply will let that the object get out of scope.
$Env = LMDB::Env->new ( $path [, ENVOPTIONS ] )
Creates a new "LMDB::Env" object and returns it. It encapsulates both LMDB's "mdb_env_create" and "mdb_env_open" functions.
$path is the directory in which the database files reside. This directory must already exist and should be writable.
ENVOPTIONS, if provided, must be a HASH Reference with any of the following options:
The size of the memory map is also the maximum size of the database. The value should be chosen as large as possible, to accommodate future growth of the database. The size should be a multiple of the OS page size.
The default is 1048576 bytes (1 MB).
This defines the number of slots in the lock table that is used to track readers in the environment.
The default is 126.
This option is only needed if multiple databases will be used in the environment. Simpler applications that use the environment as a single unnamed database can ignore this option.
The default is 0, i.e. no named databases allowed.
Incompatible with nested transactions (also known as sub transactions).
This flag may be changed at any time using "$Env->set_flags()".
This flag may be changed at any time using "$Env->set_flags()".
This flag may be changed at any time using "$Env->set_flags()".
Data is always written to disk when "$Txn->commit()" is called, but the operating system may keep it buffered. LMDB always flushes the OS buffers upon commit as well, unless the environment was opened with "MDB_NOSYNC" or in part "MDB_NOMETASYNC".
If BOOL is TRUE force a synchronous flush. Otherwise if the environment has the "MDB_NOSYNC" flag set the flushes will be omitted, and with "MDB_MAPASYNC" they will be asynchronous.
BITMASK is the flags to change, bitwise OR'ed together. BOOL TRUE set the flags, FALSE clears them.
If provided, $tflags will be passed to the constructor, if not provided, this wrapper will propagate the environment's flag "MDB_RDONLY", if set, to the transaction constructor.
In LMDB every operation (read or write) on a Database needs to be inside a transaction. This class wraps an LMDB transaction.
You must terminate the transaction by either the "abort" or "commit" methods. After a transaction is terminated, you should not call any other method on it, except "env".
If you let an object of this class get out of scope, by default the transaction will be aborted.
$Txn = LMDB::Txn->new ( $Env [, $tflags ] )
Create a new transaction for use in the environment.
Abort the transaction like "$Txn->abort()", but keep the transaction handle in the inactive state so "$Txn->renew()" may reactivate the handle.
This saves allocation overhead if the process will start a new read-only transaction soon, and also saves locking overhead if MDB_NOTLS is in use.
The reader table lock is released, but the table slot stays tied to its thread or Transaction. Use "$Txn->abort()" to discard a reseted handle, and to free its lock table slot if MDB_NOTLS is in use.
This acquires a new reader lock for a transaction handle that had been inactivated by "$Txn->reset()". It must be called before an inactive (reseted) transaction may be used again.
In this Perl implementation if you call "$Txn->renew()" in an active Transaction the method internally calls "$Txn->reset()" for you.
Nested transactions are useful for combining components that create and commit transactions. No modifications are permanently stored until the highest level "parent" transaction is committed. Nested transactions can be aborted without aborting the parent transaction and only the changes made in the nested transaction will be rolled-back.
Aborting the parent transaction will abort and terminate all outstanding nested transactions. Committing the parent transaction will similarly commit and terminate all outstanding nested transactions.
Unlike some other databases, in LMDB changes made inside nested transactions are not visible to the parent transaction until the nested transaction is committed. In other words, transactions are always isolated, even when they are nested.
If you don't provide BOOL, you are only interested in knowing the current value of this option, which is returned in every case.
This is a convenience shortcut for "LMDB_File->new( $Txn, $Txn->open(...) )" for use when you want to use the hi-level LMDB_File's OO approach.
DBOPTIONS, if provided, should be a HASH reference with any of the following keys:
You can also call this method using its values, $dbname and DBFLAGS, documented ahead.
If provided $dbname, will be the name of a named Database in the environment, if not provided (or if $dbname is "undef"), the opened Database will be the unnamed (the default) one.
DBFLAGS, if provided, will set special options for this Database and can be specified by OR'ing the following flags:
After successfully commit the transaction that created the Database, it will remains opened in the Environment so you can reuse $dbi in other transactions.
If you will need to use that Database handler in more than one transaction or want to use a more traditional (in LMDB's point of view) approach, this the method you should use.
To operate in the opened database with the returned $dbi handler you can use the methods described bellow or call "LMDB_File->new(...)" to obtain a "LMDB_File" object to operate the database in a particular transaction.
Provided for when your main concern is the raw speed.
For details of the other arguments, please see the method of the same name in LMDB_File below.
Provided for when your main concern is the raw speed.
For details of the other arguments, please see the method of the same name in LMDB_File below.
In the LMDB C API all Database operations need both an active Transaction and a Database handler. To simplify those operations and be syntax compatible with others *_File modules, this Perl API provides you a LMDB_File object that encapsulates both and implements some hi-level extensions.
LMDB_File's methods, in contrast to the LMDB::Txn's ones of the same name, perform some checks before calling the low-level C API.
This function stores key/data pairs in the database. The default behavior is to enter the new key/data pair, replacing any previously existing key if duplicates are disallowed, or adding a duplicate data item if duplicates are allowed
$key is the key to store in the database and $data the data to store.
WRITEFLAGS, if provided, will set special options for this operation and can be one of following flags:
The function will return MDB_KEYEXIST if the key already appears in the database, even if the database supports duplicates (#MDB_DUPSORT). The $data parameter will be set to the existing item.
In this particular case, you need to pass the extra $length parameter to specify how many bytes to reserve.
Please read about the "$DB->ReadMode" method caveats bellow for details that apply to the magical scalar returned in $data in this case.
No key comparisons are performed. This option allows fast bulk loading when keys are already known to be in the correct order.
NOTE: Loading unsorted keys with this flag will cause data corruption.
This method retrieves key/data pairs from the database.
If the database supports duplicate keys (#MDB_DUPSORT) then the first data item for the key will be returned. Retrieval of other items requires the use of the "LMBD::Cursor->get()" method.
The two-argument form, closer to the C API, returns in the provided argument $data the value associated with $key in the database if it exists or reports an error if not.
In the simpler, more "perlish" one-argument form, the method returns the value associated with $key in the database or "undef" if no such value exists.
This function removes key/data pairs from the database.
If the database does not support sorted duplicate data items, (MDB_DUPSORT) the $data parameter is optional and is ignored.
If the database supports sorted duplicates and the $data parameter is "undef" or not provided, all of the duplicate data items for the $key will be deleted. Otherwise, if the $data parameter is provided only the matching data item will be deleted.
CODE should be a subroutine reference or an anonymous subroutine, that like Perl's "sort" in perlfunc, will receive the values to compare in the global variables $a and $b.
The comparison function is called whenever it is necessary to compare a key specified by the application with a key currently stored in the database. If no comparison function is specified, and no special key flags were specified in "LMDB_File->open()", the keys are compared lexically, with shorter keys collating before longer keys.
Warning: This function must be called before any data access functions are used, otherwise data corruption may occur. The same comparison function must be used by every program accessing the database, every time the database is used.
$DB->Txn->commit; # Commit the current transaction.
If the method "$DB->Alive" has returned FALSE before, this method will return "undef".
You can use "$DB->Txn" as an lvalue to change the associated Transaction, but remember that, if $DB is holding the last reference of the current transaction, that transaction will be terminated.
$DB->Txn->commit; # Commit current
$DB->Alive; # FALSE
...
$DB->Txn = $Env->BeginTxn; # Start another, with same Database
...
You can use "$DB->dbi" as an lvalue to switch the associated Datbase hander:
$DB->dbi = $other_dbi;
The C documentation for the "mdb_get" function states that:
The memory pointed to by the returned values is owned by the
database. The caller need not dispose of the memory, and may not
modify it in any way. For values returned in a read-only transaction
any modification attempts will cause a SIGSEGV.
So this module implements two modes of operation for its "get" methods and you can select between them with this method.
When MODE is 0 (or any FALSE value) a default "safe" mode is used in which the data value found in the database is copied to the scalar returned, so you can do anything you want to that scalar without side effects.
But when MODE is 1 (or, in the current implementation, any TRUE value) a sort of hack is used to avoid the memory copy and a magical scalar returned that hold only a pointer to the data value found. This is much faster and uses less memory, especially when used with large values.
In a environment opened with MDB_WRITEMAP and in a transaction without the MDB_RDONLY flag, you are allowed to modify the returned scalar, and the modifications are reflected to the associated memory block and preserved in the database when the transaction is commited. Otherwise the magical scalar is marked READ-ONLY and any attempt to modify it (other than reuse it in another "$DB->get" ), will cause perl to croak.
CAVEATS: In a read-only transaction the value is valid only until the end of the transaction, and in a read-write transaction the value is valid only until the next write operation (because any write operation can potentially modify the in-memory btree). In the current implementation, you are responsible for the proper timing of usage.
NOTE: In order to achieve the zero-copy behavior desired by setting ReadMode to TRUE, you must use the two-argument form of get ("$DB->get ( $key, $data )"), use the new "$DB->Rget( $key )" or use the cursor get method described below.
Returns the previous value.
By default, all values in LMDB are simple byte buffers of certain fixed length.
So if you are storing binary data in your database all works as expected: what you put is what you get.
But when you need to store some arbitrary Unicode text value, remember that internally perl stores your strings in either the native eight-bit character set or in UTF-8, and to warrant a consistent encoding in your database you should do something like:
use Encoding;
...
$DB->put($key, Encode::encode($my_encoding, $characters));
$characters = Encode::decode($my_encoding, $DB->get($key));
For any value of $my_encoding, see Encode for the gory details.
But if you use for interchange the UTF-8 encoding, with this method you can avoid all that typing.
When MODE is 1, all values that you put in the Database will be encoded in UTF-8, And all get calls will expect UTF-8 data and it will be verified and decoded. In this mode, if malformed data is found, a warning will be emitted, the decode attempt aborted and the raw bytes returned.
In this mode, a "$foo->get(...)" call interacts with the bytes pragma in a special way: In the lexical scope under the effects of "use bytes", any get call skips the decode step, returning the fetched encoded UTF-8 data as bytes, i.e. with the internal perl UTF8 flag off, as expected by modules like JSON::XS.
To construct a cursor you should call the "Cursor" method of the "LMDB_File" class:
$cursor = $DB->Cursor
The variables $key and $data are used to return the values found.
CURSOR_OP determines the key/data to be retrieved and must be one of the following:
If the function succeeds and an item is inserted into the database, the cursor is always positioned to refer to the newly inserted item.
If the function fails for any reason, the state of the cursor will undetermined.
NOTE: Earlier documentation incorrectly said errors would leave the state of the cursor unchanged.
If the database was opened with "MDB_DUPSORT", the optional parameter DELFLAGS can be "MDB_NODUPDATA" to deletes all of the data items for the current key.
At "use" time you can import into your namespace the following constants, grouped by their tags.
MDB_FIXEDMAP MDB_NOSUBDIR MDB_NOSYNC MDB_RDONLY MDB_NOMETASYNC MDB_WRITEMAP MDB_MAPASYNC MDB_NOTLS
MDB_REVERSEKEY MDB_DUPSORT MDB_INTEGERKEY MDB_DUPFIXED MDB_INTEGERDUP MDB_REVERSEDUP MDB_CREATE
MDB_NOOVERWRITE MDB_NODUPDATA MDB_CURRENT MDB_RESERVE MDB_APPEND MDB_APPENDDUP MDB_MULTIPLE
All of ":envflags", ":dbflags" and ":writeflags"
MDB_FIRST MDB_FIRST_DUP MDB_GET_BOTH MDB_GET_BOTH_RANGE MDB_GET_CURRENT MDB_GET_MULTIPLE MDB_NEXT MDB_NEXT_DUP MDB_NEXT_MULTIPLE MDB_NEXT_NODUP MDB_PREV MDB_PREV_DUP MDB_PREV_NODUP MDB_LAST MDB_LAST_DUP MDB_SET MDB_SET_KEY MDB_SET_RANGE
MDB_SUCCESS MDB_KEYEXIST MDB_NOTFOUND MDB_PAGE_NOTFOUND MDB_CORRUPTED MDB_PANIC MDB_VERSION_MISMATCH MDB_INVALID MDB_MAP_FULL MDB_DBS_FULL MDB_READERS_FULL MDB_TLS_FULL MDB_TXN_FULL MDB_CURSOR_FULL MDB_PAGE_FULL MDB_MAP_RESIZED MDB_INCOMPATIBLE MDB_BAD_RSLOT MDB_LAST_ERRCODE
MDB_VERSION_FULL MDB_VERSION_MAJOR MDB_VERSION_MINOR MDB_VERSION_PATCH MDB_VERSION_STRING MDB_VERSION_DATE
The simplest interface to LMDB is using "tie" in perlfunc.
The TIE interface of LMDB_File can take several forms that depend on the data at hand.
The first two forms will create and/or open the Environment at $path, create a new Transaction and open a Database in the Transaction.
If provided, $options must be a HASH reference with options for both the Environment and the database.
Valid keys for $option are any described above for ENVOPTIONS and DBOPTIONS.
In the case that you have already created a transaction or an environment, you can provide a HASH reference in DBOPTIONS for options exclusively for the database.
In the forms that needs to create a Transaction, this is setted for Autocommit mode.
Salvador Ortiz Garcia, <sortiz@cpan.org>
Copyright (C) 2013-2014 by Salvador Ortiz García Copyright (C) 2013-2014 by Matías Software Group, S.A. de C.V.
This library is free software; you can redistribute it and/or modify it under the terms of the Artistic License version 2.0, see LICENSE.
| 2022-10-20 | perl v5.36.0 |