============================ Writing persistent objects ============================ In the :ref:`Tutorial `, we discussed the basics of implementing persistent objects by subclassing ``persistent.Persistent``. This is probably enough for 80% of persistent-object classes you write, but there are some other aspects of writing persistent classes you should be aware of. Access and modification ======================= Two of the main jobs of the ``Persistent`` base class are to detect when an object has been accessed and when it has been modified. When an object is accessed, its state may need to be loaded from the database. When an object is modified, the modification needs to be saved if a transaction is committed. ``Persistent`` detects object accesses by hooking into object attribute access and update. In the case of object update, there may be other ways of modifying state that we need to make provision for. Rules of persistence ==================== When implementing persistent objects, be aware that an object's attributes should be : - immutable (such as strings or integers), - persistent (subclass Persistent), or - You need to take special precautions. If you modify a non-persistent mutable value of a persistent-object attribute, you need to mark the persistent object as changed yourself by setting ``_p_changed`` to True:: import persistent class Book(persistent.Persistent): def __init__(self, title): self.title = title self.authors = [] def add_author(self, author): self.authors.append(author) self._p_changed = True .. -> src >>> exec(src) >>> db = ZODB.DB(None) >>> with db.transaction() as conn: ... conn.root.book = Book("ZODB") >>> conn = db.open() >>> book = conn.root.book >>> bool(book._p_changed) False >>> book.authors.append('Jim') >>> bool(book._p_changed) False >>> book.add_author('Carlos') >>> bool(book._p_changed) True >>> db.close() In this example, ``Book`` objects have an ``authors`` object that's a regular Python list, so it's mutable and non-persistent. When we add an author, we append it to the ``authors`` attribute's value. Because we didn't set an attribute on the book, it's not marked as changed, so we set ``_p_changed`` ourselves. Using standard Python lists, dicts, or sets is a common thing to do, so this pattern of setting ``_p_changed`` is common. Let's look at some alternatives. Using tuples for small sequences instead of lists ------------------------------------------------- If objects contain sequences that are small or that don't change often, you can use tuples instead of lists:: import persistent class Book(persistent.Persistent): def __init__(self, title): self.title = title self.authors = () def add_author(self, author): self.authors += (author, ) .. -> src >>> exec(src) >>> db = ZODB.DB(None) >>> with db.transaction() as conn: ... conn.root.book = Book("ZODB") >>> conn = db.open() >>> book = conn.root.book >>> bool(book._p_changed) False >>> book.add_author('Carlos') >>> bool(book._p_changed) True >>> db.close() Because tuples are immutable, they satisfy the rules of persistence without any special handling. Using persistent data structures -------------------------------- The ``persistent`` package provides persistent versions of ``list`` and ``dict``, namely ``persistent.list.PersistentList`` and ``persistent.mapping.PersistentMapping``. We can update our example to use ``PersistentList``:: import persistent import persistent.list class Book(persistent.Persistent): def __init__(self, title): self.title = title self.authors = persistent.list.PersistentList() def add_author(self, author): self.authors.append(author) .. -> src >>> exec(src) >>> db = ZODB.DB(None) >>> with db.transaction() as conn: ... conn.root.book = Book("ZODB") >>> conn = db.open() >>> book = conn.root.book >>> bool(book._p_changed) False >>> book.add_author('Carlos') >>> bool(book._p_changed) False >>> bool(book.authors._p_changed) True >>> db.close() Note that in this example, when we added an author, the book itself didn't change, but the ``authors`` attribute value did. Because ``authors`` is a persistent object, it's stored in a separate database record from the book record and is managed by ZODB independent of the management of the book. In addition to ``PersistentList`` and ``PersistentMapping``, general persistent data structures are provided by the BTrees_ package, most notably ``BTree`` and ``TreeSet`` objects. Unlike ``PersistentList`` and ``PersistentMapping``, ``BTree`` and ``TreeSet`` objects are scalable and can easily hold millions of objects, because their data are spread over many subobjects. It's generally better to use ``BTree`` objects than ``PersistentMapping`` objects, because they're scalable and because they handle :ref:`conflicts ` better. ``TreeSet`` objects are the only ZODB-provided persistent set implementation. ``BTree`` and ``TreeSets`` come in a number of families provided via different modules and differ in their internal implementations: =============== =============== ================ Module Key type Value Type =============== =============== ================ BTrees.OOBTree object object BTrees.IOBTree integer Object BTrees.OIBTree object integer BTrees.IIBTree integer integer BTrees.IFBTree integer float BTrees.LOBTree 64-bit integer Object BTrees.OLBTree object 64-bit integer BTrees.LLBTree 64-bit integer 64-bit integer BTrees.LFBTree 64-bit integer float =============== =============== ================ Here's a version of the example that uses a ``TreeSet``:: import persistent from BTrees.OOBTree import TreeSet class Book(persistent.Persistent): def __init__(self, title): self.title = title self.authors = TreeSet() def add_author(self, author): self.authors.add(author) .. -> src >>> exec(src) >>> db = ZODB.DB(None) >>> with db.transaction() as conn: ... conn.root.book = Book("ZODB") >>> conn = db.open() >>> book = conn.root.book >>> bool(book._p_changed) False >>> book.add_author('Carlos') >>> bool(book._p_changed) False >>> bool(book.authors._p_changed) True >>> db.close() If you're going to use custom classes as keys in a ``BTree`` or entries in a ``TreeSet``, they must provide a `total ordering `_. The builtin python `str` class is always safe to use as BTree key. You can use `zope.keyreference `_ to treat arbitrary persistent objects as totally orderable based on their persistent object identity. Scalable sequences are a bit more challenging. The `zc.blist `_ package provides a scalable list implementation that works well for some sequence use cases. Properties ========== If you implement some attributes using Python properties (or other types of descriptors), they are treated just like any other attributes by the persistence machinery. When you set an attribute through a property, the object is considered changed, even if the property didn't actually modify the object state. Special attributes ================== There are some attributes that are treated specially. Attributes with names starting with ``_p_`` are reserved for use by the persistence machinery and by ZODB. These include (but aren't limited to): _p_changed The ``_p_changed`` attribute has the value ``None`` if the object is a :ref:`ghost `, True if it's changed, and False if it's not a ghost and not changed. _p_oid The object's unique id in the database. _p_serial The object's revision identifier also know as the object serial number, also known as the object transaction id. It's a timestamp and if not set has the value 0 encoded as string of 8 zero bytes. _p_jar The database connection the object was accessed through. This is commonly used by database-aware application code to get hold of an object's database connection. An object's ``__dict__`` attribute is treated specially in that getting it doesn't cause an object's state to be loaded. It may have the value ``None`` rather than a dictionary for :ref:`ghosts `. Volatile Attributes ------------------- Attributes with names starting with ``_v_`` are volatile, they are never serialized and not saved to the database. They are useful for caching data that can be computed from other data[#cache]_. Volatile attributes are local to a specific active object in memory and thus to a specific connection. If an object is removed from the connection cache the volatile attribute is lost. Setting a volatile attribute does not cause an object to be considered to be modified. Object storage and management ============================= Every persistent object is stored in its own database record. Some storages maintain multiple object revisions, in which case each persistent object is stored in its own set of records. Data for different persistent objects are stored separately. The database manages each object separately, according to a :ref:`life cycle `. This is important when considering how to distribute data across your objects. If you use lots of small persistent objects, then more objects may need to be loaded or saved and you may incur more memory overhead. On the other hand, if objects are too big, you may load or save more data than would otherwise be needed. You can't change your mind in subclassing persistent ==================================================== Currently, you can't change your mind about whether a class is persistent (subclasses ``persistent.Persistent``) or not. If you save objects in a database who's classes subclass ``persistent.Persistent``, you can't change your mind later and make them non-persistent, and the other way around. This may be a `bug or misfeature `_. .. _schema-migration-label: Schema migration ================ Object requirements and implementations tend to evolve over time. This isn't a problem for objects that are short lived, but persistent objects may have lifetimes that extend for years. There needs to be some way of making sure that state for an older object schema can still be loaded into an object with the new schema. Adding attributes ----------------- Perhaps the commonest schema change is to add attributes. This is usually accomplished easily by adding a default value in a class definition:: class Book(persistent.Persistent): publisher = 'UNKNOWN' def __init__(self, title, publisher): self.title = title self.publisher = publisher self.authors = TreeSet() def add_author(self, author): self.authors.add(author) Removing attributes ------------------- Removing attributes generally doesn't require any action, assuming that their presence in older objects doesn't do any harm. Renaming/moving classes ----------------------- The easiest way to handle renaming or moving classes is to leave aliases for the old name. For example, if we have a class, ``library.Book``, and want to move it to ``catalog.Publication``, we can keep a ``library`` module that contains:: from catalog import Publication as Book # XXX deprecated name A downside of this approach is that it clutters code and may even cause us to keep modules solely to hold aliases. (`zope.deferredimport `_ can help with this by making these aliases a little more efficient and by generating deprecation warnings.) Migration scripts ----------------- If the simple approaches above aren't enough, then migration scripts can be used. How these scripts are written is usually application dependent, as the application usually determines where objects of a given type reside in the database. (There are also some low-level interfaces for iterating over all of the objects of a database, but these are usually impractical for large databases.) An improvement to running migration scripts manually is to use a generational framework like `zope.generations `_. With a generational framework, each migration is assigned a migration number and the number is recorded in the database as each migration is run. This is useful because remembering what migrations are needed is automated. Upgrading multiple clients without down time ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Production applications typically have multiple clients for availability and load balancing. This means an active application may be committing transactions using multiple software and schema versions. In this situation, you may need to plan schema migrations in multiple steps: #. Upgrade software on all clients to a version that works with the old and new version of the schema and that writes data using the old schema. #. Upgrade software on all clients to a version that works with the old and new version of the schema and that writes data using the new schema. #. Migrate objects written with the old schema to the new schema. #. Remove support for the old schema from the software. .. _object-life-cycle-label: Object life cycle states and special attributes (advanced) ========================================================== Persistent objects typically transition through a collection of states. Most of the time, you don't need to think too much about this. Unsaved When an object is created, it's said to be in an *unsaved* state until it's associated with a database. Added When an unsaved object is added to a database, but hasn't been saved by committing a transaction, it's in the *added* state. Note that most objects are added implicitly by being set as subobjects (attribute values or items) of objects already in the database. Saved When an object is added and saved through a transaction commit, the object is in the *saved* state. Changed When a saved object is updated, it enters the *changed* state to indicate that there are changes that need to be committed. It remains in this state until either: - The current transaction is committed, and the object transitions to the saved state, or - The current transaction is aborted, and the object transitions to the ghost state. .. _ghost-label: Ghost An object in the *ghost* state is an empty shell. It has no state. When it's accessed, its state will be loaded automatically, and it will enter the saved state. A saved object can become a ghost if it hasn't been accessed in a while and the database releases its state to make room for other objects. A changed object can also become a ghost if the transaction it's modified in is aborted. An object that's loaded from the database is loaded as a ghost. This typically happens when the object is a subobject of another object who's state is loaded. We can interrogate and control an object's state, although somewhat indirectly. To do this, we'll look at some special persistent-object attributes, described in `Special attributes`_, above. Let's look at some state transitions with an example. First, we create an unsaved book:: >>> book = Book("ZODB") >>> from ZODB.utils import z64 >>> book._p_changed, bool(book._p_oid) (False, False) We can tell that it's unsaved because it doesn't have an object id, ``_p_oid``. If we add it to a database:: >>> import ZODB >>> connection = ZODB.connection(None) >>> connection.add(book) >>> book._p_changed, bool(book._p_oid), book._p_serial == z64 (False, True, True) We know it's added because it has an oid, but its serial (object revision timestamp), ``_p_serial``, is the special zero value, and it's value for ``_p_changed`` is False. If we commit the transaction that added it:: >>> import transaction >>> transaction.commit() >>> book._p_changed, bool(book._p_oid), book._p_serial == z64 (False, True, False) We see that the object is in the saved state because it has an object id and serial, and is unchanged. Now if we modify the object, it enters the changed state: >>> book.title = "ZODB Explained" >>> book._p_changed, bool(book._p_oid), book._p_serial == z64 (True, True, False) If we abort the transaction, the object becomes a ghost: >>> transaction.abort() >>> book._p_changed, bool(book._p_oid) (None, True) We can see it's a ghost because ``_p_changed`` is None. (``_p_serial`` isn't meaningful for ghosts.) If we access the object, it will be loaded into the saved state, which is indicated by a false ``_p_changed`` and an object id and non-zero serial. >>> book.title 'ZODB' >>> book._p_changed, bool(book._p_oid), book._p_serial == z64 (False, True, False) Note that accessing ``_p_`` attributes didn't cause the object's state to be loaded. We've already seen how modifying ``_p_changed`` can cause an object to be marked as modified. We can also use it to make an object into a ghost: >>> book._p_changed = None >>> book._p_changed, bool(book._p_oid) (None, True) Things you can do, but need to carefully consider (advanced) ============================================================ While you can do anything with a persistent subclass that you can with a normal subclass, certain things have additional implications for persistent objects. These often show up as performance issues, or the result may become hard to maintain. Implement ``__eq__`` and ``__hash__`` ------------------------------------- When you store an entry into a Python ``dict`` (or the persistent variant ``PersistentMapping``, or a ``set`` or ``frozenset``), the key's ``__eq__`` and ``__hash__`` methods are used to determine where to store the value. Later they are used to look it up via ``in`` or ``__getitem__``. When that ``dict`` is later loaded from the database, the internal storage is rebuilt from scratch. This means that every key has its ``__hash__`` method called at least once, and may have its ``__eq__`` method called many times. By default, every object, including persistent objects, inherits an implementation of ``__eq__`` and ``__hash__`` from :class:`object`. These default implementations are based on the object's *identity*, that is, its unique identifier within the current Python process. Calling them, therefore, is very fast, even on :ref:`ghosts `, and doesn't cause a ghost to load its state. If you override ``__eq__`` and ``__hash__`` in a custom persistent subclass, however, when you use instances of that class as a key in a ``dict``, then the instance will have to be unghosted before it can be put in the dictionary. If you're building a large dictionary with many such keys that are ghosts, you may find that loading all the object states takes a considerable amount of time. If you were to store that dictionary in the database and load it later, *all* the keys will have to be unghosted at the same time before the dictionary can be accessed, again, possibly taking a long time. For example, a class that defines ``__eq__`` and ``__hash__`` like this:: class BookEq(persistent.Persistent): def __init__(self, title): self.title = title self.authors = () def add_author(self, author): self.authors += (author, ) def __eq__(self, other): return self.title == other.title and self.authors == other.authors def __hash__(self): return hash((self.title, self.authors)) .. -> src >>> exec(src) is going to be much slower to use as a key in a persistent dictionary, or in a new dictionary when the key is a ghost, than the class that inherits identity-based ``__eq__`` and ``__hash__``. .. Example of the above. Here's what that class would look like:: >>> class Book(persistent.Persistent): ... def __init__(self, title): ... self.title = title ... self.authors = () ... ... def add_author(self, author): ... self.authors += (author, ) Lets see an example of how these classes behave when stored in a dictionary. First, lets store some dictionaries:: >>> import ZODB >>> db = ZODB.DB(None) >>> conn1 = db.open() >>> conn1.root.with_hashes = {BookEq(str(i)) for i in range(5000)} >>> conn1.root.with_ident = {Book(str(i)) for i in range(5000)} >>> transaction.commit() Now, in a new connection (so we don't have any objects cached), lets load the dictionaries:: >>> conn2 = db.open() >>> all((book._p_status == 'ghost' for book in conn2.root.with_ident)) True >>> all((book._p_status == 'ghost' for book in conn2.root.with_hashes)) False We can see that all the objects that did have a custom ``__eq__`` and ``__hash__`` were loaded into memory, while those that did weren't. There are some alternatives: - Avoiding the use of persistent objects as keys in dictionaries or entries in sets sidesteps the issue. - If your application can tolerate identity based comparisons, simply don't implement the two methods. This means that objects will be compared only by identity, but because persistent objects are persistent, the same object will have the same identity in each connection, so that often works out. It is safe to remove ``__eq__`` and ``__hash__`` methods from a class even if you already have dictionaries in a database using instances of those classes as keys. - Make your classes `orderable `_ and use them as keys in a BTree or entries in a TreeSet instead of a dictionary or set. Even though your custom comparison methods will have to unghost the objects, the nature of a BTree means that only a small number of objects will have to be loaded in most cases. - Any persistent object can be wrapped in a ``zope.keyreferenece`` to make it orderable and hashable based on persistent identity. This can be an alternative for some dictionaries if you can't alter the class definition but can accept identity comparisons in some dictionaries or sets. You must remember to wrap all keys, though. Implement ``__getstate__`` and ``__setstate__`` ----------------------------------------------- When an object is saved in a database, its ``__getstate__`` method is called without arguments to get the object's state. The default implementation simply returns a copy of an object's instance dictionary. (It's a little more complicated for objects with slots.) An object's state is loaded by loading the state from the database and passing it to the object's ``__setstate__`` method. The default implementation expects a dictionary, which it uses to populate the object's instance dictionary. Early on, we thought that overriding these methods would be useful for tasks like providing more efficient state representations or for :ref:`schema migration `, but we found that the result was to make object implementations brittle and/or complex and the benefit usually wasn't worth it. Implement ``__getattr__``, ``__getattribute__``, or ``__setattribute__`` ------------------------------------------------------------------------ This is something extremely clever people might attempt, but it's probably never worth the bother. It's possible, but it requires such deep understanding of persistence and internals that we're not even going to document it. :) Links ===== `persistent.Persistent `_ provides additional documentation on the ``Persistent`` base class. The `zc.blist `_ package provides a scalable sequence implementation for many use cases. The `zope.cachedescriptors `_ package provides descriptor implementations that facilitate implementing caching attributes, especially ``_v_`` volatile attributes. The `zope.deferredimport `_ package provides lazy import and support for deprecating import location, which is helpful when moving classes, especially persistent classes. The `zope.generations `_ package provides a framework for managing schema-migration scripts. .. [#cache] The `zope.cachedescriptors `_ package provides some descriptors that help implement attributes that cache data. .. _BTrees: https://pythonhosted.org/BTrees/