Storage APIs 

Storage interfaces 

There are various storage implementations that implement standard storage interfaces. They differ primarily in their constructors.

Application code rarely calls storage methods, and those it calls are generally called indirectly through databases. There are interface-defined methods that are called internally by ZODB. These aren’t shown below.

IStorage 

interface ZODB.interfaces.IStorage[source]

A storage is responsible for storing and retrieving data of objects.

Consistency and locking

When transactions are committed, a storage assigns monotonically increasing transaction identifiers (tids) to the transactions and to the object versions written by the transactions. ZODB relies on this to decide if data in object caches are up to date and to implement multi-version concurrency control.

There are methods in IStorage and in derived interfaces that provide information about the current revisions (tids) for objects or for the database as a whole. It is critical for the proper working of ZODB that the resulting tids are increasing with respect to the object identifier given or to the databases. That is, if there are 2 results for an object or for the database, R1 and R2, such that R1 is returned before R2, then the tid returned by R2 must be greater than or equal to the tid returned by R1. (When thinking about results for the database, think of these as results for all objects in the database.)

This implies some sort of locking strategy. The key method is tcp_finish, which causes new tids to be generated and also, through the callback passed to it, returns new current tids for the objects stored in a transaction and for the database as a whole.

The IStorage methods affected are lastTransaction, load, store, and tpc_finish. Derived interfaces may introduce additional methods.

__len__()

The approximate number of objects in the storage

This is used soley for informational purposes.

close()

Close the storage.

Finalize the storage, releasing any external resources. The storage should not be used after this method is called.

Note that databases close their storages when they’re closed, so this method isn’t generally called from application code.

getName()

The name of the storage

The format and interpretation of this name is storage dependent. It could be a file name, a database name, etc..

This is used soley for informational purposes.

getSize()

An approximate size of the database, in bytes.

This is used soley for informational purposes.

history(oid, size=1)

Return a sequence of history information dictionaries.

Up to size objects (including no objects) may be returned.

The information provides a log of the changes made to the object. Data are reported in reverse chronological order.

Each dictionary has the following keys:

time: UTC seconds since the epoch (as in time.time) that the object revision was committed.
tid: The transaction identifier of the transaction that committed the version.
serial: An alias for tid, which expected by older clients.
user_name: The bytes user identifier, if any (or an empty string) of the user on whos behalf the revision was committed.
description: The bytes transaction description for the transaction that committed the revision.
size: The size of the revision data record.

If the transaction had extension items, then these items are also included if they don’t conflict with the keys above.

isReadOnly()

Test whether a storage allows committing new transactions

For a given storage instance, this method always returns the same value. Read-only-ness is a static property of a storage.

lastTransaction()

Return the id of the last committed transaction.

For proper MVCC operation, the return value is the id of the last transaction for which invalidation notifications are completed.

In particular for client-server implementations, lastTransaction should return a cached value (rather than querying the server). A preliminary call to sync() can be done to get the actual last TID at the wanted time.

If no transactions have been committed, return a string of 8 null (0) characters.

pack(pack_time, referencesf)

Pack the storage

It is up to the storage to interpret this call, however, the general idea is that the storage free space by:

discarding object revisions that were old and not current as of the given pack time.
garbage collecting objects that aren’t reachable from the root object via revisions remaining after discarding revisions that were not current as of the pack time.

The pack time is given as a UTC time in seconds since the epoch.

The second argument is a function that should be used to extract object references from database records. This is needed to determine which objects are referenced from object revisions.

sortKey()

Sort key used to order distributed transactions

When a transaction involved multiple storages, 2-phase commit operations are applied in sort-key order. This must be unique among storages used in a transaction. Obviously, the storage can’t assure this, but it should construct the sort key so it has a reasonable chance of being unique.

The result must be a string.

IStorageIteration 

interface ZODB.interfaces.IStorageIteration[source]: API for iterating over the contents of a storage.

IStorageUndoable 

interface ZODB.interfaces.IStorageUndoable[source]

A storage supporting transactional undo.

undoInfo(first=0, last=-20, specification=None)

Return a sequence of descriptions for undoable transactions.

This is like undoLog(), except for the specification argument. If given, specification is a dictionary, and undoInfo() synthesizes a filter function f for undoLog() such that f(desc) returns true for a transaction description mapping desc if and only if desc maps each key in specification to the same value specification maps that key to. In other words, only extensions (or supersets) of specification match.

ZEO note: undoInfo() passes the specification argument from a ZEO client to its ZEO server (while a ZEO client ignores any filter argument passed to undoLog()).

undoLog(first, last, filter=None)

Return a sequence of descriptions for undoable transactions.

Application code should call undoLog() on a DB instance instead of on the storage directly.

A transaction description is a mapping with at least these keys:

“time”: The time, as float seconds since the epoch, when
the transaction committed.

“user_name”: The bytes value of the .user attribute on that
transaction.

“description”: The bytes value of the .description attribute on
that transaction.

“id`” A bytes uniquely identifying the transaction to the
storage. If it’s desired to undo this transaction, this is the transaction_id to pass to undo().

In addition, if any name+value pairs were added to the transaction by setExtendedInfo(), those may be added to the transaction description mapping too (for example, FileStorage’s undoLog() does this).

filter is a callable, taking one argument. A transaction description mapping is passed to filter for each potentially undoable transaction. The sequence returned by undoLog() excludes descriptions for which filter returns a false value. By default, filter always returns a true value.

ZEO note: Arbitrary callables cannot be passed from a ZEO client to a ZEO server, and a ZEO client’s implementation of undoLog() ignores any filter argument that may be passed. ZEO clients should use the related undoInfo() method instead (if they want to do filtering).

Now picture a list containing descriptions of all undoable transactions that pass the filter, most recent transaction first (at index 0). The first and last arguments specify the slice of this (conceptual) list to be returned:

first: This is the index of the first transaction description
in the slice. It must be >= 0.

last: If >= 0, first:last acts like a Python slice, selecting
the descriptions at indices first, first+1, …, up to but not including index last. At most last-first descriptions are in the slice, and last should be at least as large as first in this case. If last is less than 0, then abs(last) is taken to be the maximum number of descriptions in the slice (which still begins at index first). When last < 0, the same effect could be gotten by passing the positive first-last for last instead.

IStorageCurrentRecordIteration 

interface ZODB.interfaces.IStorageCurrentRecordIteration[source]

IBlobStorage 

interface ZODB.interfaces.IBlobStorage[source]

A storage supporting BLOBs.

temporaryDirectory()

Return a directory that should be used for uncommitted blob data.

If Blobs use this, then commits can be performed with a simple rename.

IStorageRecordInformation 

interface ZODB.interfaces.IStorageRecordInformation[source]: Provide information about a single storage record

IStorageTransactionInformation 

interface ZODB.interfaces.IStorageTransactionInformation[source]

Provide information about a storage transaction.

Can be iterated over to retrieve the records modified in the transaction.

Note that this may contain a status field used by FileStorage to support packing. At some point, this will go away when FileStorage has a better pack algoritm.

Included storages 

FileStorage 

class ZODB.FileStorage.FileStorage.FileStorage(file_name, create=False, read_only=False, stop=None, quota=None, pack_gc=True, pack_keep_old=True, packer=None, blob_dir=None)[source]

Storage that saves data in a file

__init__(file_name, create=False, read_only=False, stop=None, quota=None, pack_gc=True, pack_keep_old=True, packer=None, blob_dir=None)[source]

Create a file storage

Parameters:

file_name (str) – Path to store data file
create (bool) – Flag indicating whether a file should be created even if it already exists.
read_only (bool) – Flag indicating whether the file is read only. Only one process is able to open the file non-read-only.
stop (bytes) – Time-travel transaction id When the file is opened, data will be read up to the given transaction id. Transaction ids correspond to times and you can compute transaction ids for a given time using TimeStamp.
quota (int) – File-size quota
pack_gc (bool) – Flag indicating whether garbage collection should be performed when packing.
pack_keep_old (bool) – flag indicating whether old data files should be retained after packing as a .old file.
packer (callable) – An alternative packer.
blob_dir (str) – A blob-directory path name. Blobs will be supported if this option is provided.

A file storage stores data in a single file that behaves like a traditional transaction log. New data records are appended to the end of the file. Periodically, the file is packed to free up space. When this is done, current records as of the pack time or later are copied to a new file, which replaces the old file.

FileStorages keep in-memory indexes mapping object oids to the location of their current records in the file. Back pointers to previous records allow access to non-current records from the current records.

In addition to the data file, some ancillary files are created. These can be lost without affecting data integrity, however losing the index file may cause extremely slow startup. Each has a name that’s a concatenation of the original file and a suffix. The files are listed below by suffix:

.index: Snapshot of the in-memory index. This are created on shutdown, packing, and after rebuilding an index when one was not found. For large databases, creating a file-storage object without an index file can take very long because it’s necessary to scan the data file to build the index.
.lock: A lock file preventing multiple processes from opening a file storage on non-read-only mode.
.tmp: A file used to store data being committed in the first phase of 2-phase commit
.index_tmp: A temporary file used when saving the in-memory index to avoid overwriting an existing index until a new index has been fully saved.
.pack: A temporary file written while packing containing current records as of and after the pack time.
.old: The previous database file after a pack.

When the database is packed, current records as of the pack time and later are written to the .pack file. At the end of packing, the .old file is removed, if it exists, and the data file is renamed to the .old file and finally the .pack file is rewritten to the data file.

interface ZODB.FileStorage.interfaces.IFileStoragePacker[source]

FileStorage text configuration 

File storages are configured using the filestorage section:

<filestorage>
  path Data.fs
</filestorage>

which accepts the following options:

blob-dir (existing-dirpath): If supplied, the file storage will provide blob support and this is the name of a directory to hold blob data. The directory will be created if it doesn’t exist. If no value (or an empty value) is provided, then no blob support will be provided. (You can still use a BlobStorage to provide blob support.)
create (boolean): Flag that indicates whether the storage should be truncated if it already exists.
pack-gc (boolean, default: true): If false, then no garbage collection will be performed when packing. This can make packing go much faster and can avoid problems when objects are referenced only from other databases.
pack-keep-old (boolean, default: true): If true, a copy of the database before packing is kept in a “.old” file.
packer (string): The dotted name (dotted module name and object name) of a packer object. This is used to provide an alternative pack implementation.
path (existing-dirpath, required): Path name to the main storage file. The names for supplemental files, including index and lock files, will be computed from this.
quota (byte-size): Maximum allowed size of the storage file. Operations which would cause the size of the storage to exceed the quota will result in a ZODB.FileStorage.FileStorageQuotaError being raised.
read-only (boolean): If true, only reads may be executed against the storage. Note that the “pack” operation is not considered a write operation and is still allowed on a read-only filestorage.

MappingStorage 

class ZODB.MappingStorage.MappingStorage(name='MappingStorage')[source]

In-memory storage implementation

Note that this implementation is somewhat naive and inefficient with regard to locking. Its implementation is primarily meant to be a simple illustration of storage implementation. It’s also useful for testing and exploration where scalability and efficiency are unimportant.

__init__(name='MappingStorage')[source]

Create a mapping storage

The name parameter is used by the getName() and sortKey() methods.

MappingStorage text configuration 

File storages are configured using the mappingstorage section:

<mappingstorage>
</mappingstorage>

Options:

name (string, default: Mapping Storage): The storage name, used by the getName() and sortKey() methods.

DemoStorage 

class ZODB.DemoStorage.DemoStorage(name=None, base=None, changes=None, close_base_on_close=None, close_changes_on_close=None)[source]

A storage that stores changes against a read-only base database

This storage was originally meant to support distribution of application demonstrations with populated read-only databases (on CDROM) and writable in-memory databases.

Demo storages are extemely convenient for testing where setup of a base database can be shared by many tests.

Demo storages are also handy for staging appplications where a read-only snapshot of a production database (often accomplished using a beforestorage) is combined with a changes database implemented with a FileStorage.

__init__(name=None, base=None, changes=None, close_base_on_close=None, close_changes_on_close=None)[source]

Create a demo storage

Parameters:

name (str) – The storage name used by the getName() and sortKey() methods.
base (object) – base storage
changes (object) – changes storage
close_base_on_close (bool) – A Flag indicating whether the base database should be closed when the demo storage is closed.
close_changes_on_close (bool) – A Flag indicating whether the changes database should be closed when the demo storage is closed.

If a base database isn’t provided, a MappingStorage will be constructed and used.

If close_base_on_close isn’t specified, it will be True if a base database was provided and False otherwise.

If a changes database isn’t provided, a MappingStorage will be constructed and used and blob support will be provided using a temporary blob directory.

If close_changes_on_close isn’t specified, it will be True if a changes database was provided and False otherwise.

pop()[source]: Close the changes database and return the base.

push(changes=None)[source]

Create a new demo storage using the storage as a base.

The given changes are used as the changes for the returned storage and False is passed as close_base_on_close.

DemoStorage text configuration 

Demo storages are configured using the demostorage section:

<demostorage>
  <filestorage base>
    path base.fs
  </filestorage>
  <mappingstorage changes>
    name Changes
  </mappingstorage>
</demostorage>

demostorage sections can contain up to 2 storage subsections, named base and changes, specifying the demo storage’s base and changes storages. See ZODB.DemoStorage.DemoStorage.__init__() for more on the base and changes storages.

Options:

name (string): The storage name, used by the getName() and sortKey() methods.

Noteworthy non-included storages 

A number of important ZODB storages are distributed separately.

Base storages 

Unlike the included storages, all the implementations listed in this section allow multiple processes to share the same database.

NEO

NEO can spread data among several computers for load-balancing and multi-master replication. It also supports asynchronous replication to off-site NEO databases for further disaster resistance without affecting local operation latency.

For more information, see https://lab.nexedi.com/nexedi/neoppod.

RelStorage

RelStorage stores data in relational databases. This is especially useful when you have requirements or existing infrastructure for storing data in relational databases.

For more information, see http://relstorage.readthedocs.io/en/latest/.

ZEO

ZEO is a client-server database implementation for ZODB. To use ZEO, you run a ZEO server, and use ZEO clients in your application.

For more information, see https://github.com/zopefoundation/ZEO.

Optional layers 

ZRS

ZRS provides replication from one database to another. It’s most commonly used with ZEO. With ZRS, you create a ZRS primary database around a FileStorage and in a separate process, you create a ZRS secondary storage around any storage. As transactions are committed on the primary, they’re copied asynchronously to secondaries.

For more information, see https://github.com/zc/zrs.

zlibstorage

zlibstorage compresses database records using the compression algorithm used by gzip.

For more information, see https://pypi.org/project/zc.zlibstorage/.

beforestorage

beforestorage provides a point-in-time view of a database that might be changing. This can be useful to provide a non-changing view of a production database for use with a DemoStorage.

For more information, see https://pypi.org/project/zc.beforestorage/.

cipher.encryptingstorage

cipher.encryptingstorage provided compression and encryption of database records.

For more information, see https://pypi.org/project/cipher.encryptingstorage/.