| OVDB(5) | InterNetNews Documentation | OVDB(5) |
ovdb - Overview storage method for INN
The ovdb overview is a storage method that uses the Berkeley DB library to store overview data. It requires version 4.4 or later of the Berkeley DB library (4.7+ is recommended because older versions suffer from various issues).
The ovdb overview method makes use of the full transaction/logging/locking functionality of the Berkeley DB environment. Berkeley DB may be downloaded from <http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html> and is needed to build the ovdb backend.
There are several versions of the ovdb storage method:
If you have a database created with a previous version of ovdb, your database will need to be upgraded using ovdb_init. See the ovdb_init(8) man page for upgrade instructions, as well as the COMPRESSION section below.
Note that when the Berkeley DB library is updated to a newer version, the ovdb database also needs being upgraded.
If the Berkeley DB library is found at configure time, INN will be built with Berkeley DB support unless the --without-bdb flag is explicitly passed to configure. By default, configure will search for Berkeley DB in standard locations; there will be a message in the configure output indicating the pathname that will be used.
You can override this pathname by adding a path to the option, for instance --with-bdb=/usr/BerkeleyDB.4.4. This directory is expected to have subdirectories include and lib (lib32 and lib64 are also checked), containing respectively db.h, and the library itself. In case non-standard paths to the Berkeley DB libraries are used, one or both of the options --with-bdb-include and --with-bdb-lib can be given to configure with a path.
The ovdb database may take up more disk space for a given spool than the other overview methods. Plan on needing at least 1.1 KB for every article in your spool (not counting crossposts). So, if you have 5 million articles, you'll need at least 5.5 GB of disk space for ovdb. With compression enabled, this estimate changes to 0.9 KB per article, so you'll need at least 4.5 GB of disk space for 5 million articles. See the COMPRESSION section below. Plus, you'll need additional space for transaction logs: at least 100 MB. By default, the transaction logs go in the same directory as the database. To improve performance, they can be placed on a different disk -- see the DB_CONFIG section.
To enable the ovdb overview method, set the ovmethod parameter in inn.conf to "ovdb". The ovdb database is stored in the directory specified by the pathoverview parameter in inn.conf. This is the "DB_HOME" directory. To start out, this directory should be empty (other than an optional DB_CONFIG file; see DB_CONFIG for details), and innd (or makehistory) will create the files as necessary in that directory. Also, make sure the directory is owned by the news user.
Other parameters for configuring ovdb are in the ovdb.conf configuration file. The following parameters can be set in that file:
default_minkey = MAX(2, pagesize / 2600) if compress is false
default_minkey = MAX(2, pagesize / 1500) if compress is true
The lowest allowed minkey is 2. Setting minkey higher than the default is not recommended, as it will cause the databases to have a lot of overflow pages. Changing this parameter has no effect on an already-established database.
If this parameter is set to 0, expireover will compact all newsgroups; if set to 1, expireover will not compact any newsgroups; and if set to a value greater than one, expireover will only compact groups that have less than that number of articles.
Experience has shown that compacting has minimal effect (other than making expireover take longer) so the default is 1. This parameter will probably be removed in the future.
When the readserver parameter is set to true, the nnrpd processes will access overview via a helper server (ovdb_server -- which is started by ovdb_init). All ovdb reads will then be funnelled through a single process with a cleaner interface to the underlying Berkeley DB database. This will result in cleaner shutdowns for the database, improving stability and avoiding deadlocks, timing issues and corrupted databases. That's why you should try to set this parameter to true if you are experiencing any instability in the ovdb overview method.
Default value is true.
The ovdb storage method has the ability to compress overview data before it is stored into the database. In addition to consuming less disk space, compression keeps the average size of the database keys smaller. This in turn increases the average number of keys per page, which can significantly improve performance and also helps keep the database more compact. This feature requires that INN be built with zlib. Only records larger than 600 bytes get compressed, because that is the point at which compression starts to become significant.
If compression is not enabled (either from the compress option in ovdb.conf or INN was not built with zlib support), the database will be backward compatible with older versions of ovdb. However, if compression is enabled, the database is marked with a newer version that will prevent older versions of ovdb from opening the database.
You can upgrade an existing database to use compression simply by setting compress to true in ovdb.conf. Note that existing records in the database will remain uncompressed; only new records added after enabling compression will be compressed.
If you disable compression on a database that previously had it enabled, new records will be stored uncompressed, but the database will still be incompatible with older versions of ovdb (and will also be incompatible with this version of ovdb if INN was not built with zlib support). So to downgrade to a completely uncompressed database, you will have to rebuild the database using makehistory.
A file called DB_CONFIG may be placed in the database directory (pathoverview in inn.conf) to customize where the various database files and transaction logs are written. By default, all of the files are written in the "DB_HOME" directory. One way to improve performance is to put the transaction logs on a different disk. To do this, put:
DB_LOG_DIR /path/to/logs
in the DB_CONFIG file. If the pathname you give starts with a "/", it is treated as an absolute path; otherwise, it is relative to the "DB_HOME" directory. Make sure that any directories you specify exist and have proper ownership/mode before starting INN, because they won't be created automatically. Also, don't change the DB_CONFIG file while anything that uses ovdb is running.
Another thing that you can do with this file is to split the overview database across multiple disks. In the DB_CONFIG file, you can list directories that Berkeley DB will search when it goes to open a database.
For example, let's say that you have pathoverview set to /mnt/overview and you have four additional file systems created on /mnt/ovX. You would create a file /mnt/overview/DB_CONFIG containing the following lines:
set_data_dir /mnt/overview
set_data_dir /mnt/ov1
set_data_dir /mnt/ov2
set_data_dir /mnt/ov3
set_data_dir /mnt/ov4
Distribute your ovNNNNN files into the four filesystems (say, 8 each). When called upon to open a database file, the db library will look for it in each of the specified directories (in order). If said file is not found, one will be created in the first of those directories.
Whenever you change DB_CONFIG or move database files around, make sure all news processes that use the database are shut down first (including nnrpd processes).
The DB_CONFIG functionality is part of Berkeley DB itself, rather than something provided by ovdb. See the Berkeley DB documentation for complete details for the version of Berkeley DB that you're running.
When starting the news system, rc.news will invoke the ovdb_init program. See the ovdb_init(8) man page for information about the tasks it performs. ovdb_init must be run before using the database.
And when stopping INN, rc.news kills the ovdb_monitor processes after the other INN processes have been shut down.
Problems relating to ovdb are logged to news.err with "OVDB" in the error message.
INN programs that use overview will fail to start up if the ovdb_monitor processes aren't running. Be sure to run ovdb_init before running anything that accesses overview.
Also, INN programs that use overview will fail to start up if the user running them is not the news user.
If a program accessing the database crashes, or otherwise exits uncleanly, it might leave a stale lock in the database. This lock could cause other processes to deadlock on that stale lock. To fix this, shut down all news processes (using "kill -9" if necessary) and then restart. ovdb_init should perform a recovery operation which will remove the locks and repair damage caused by killing the deadlocked processes.
Implement a way to limit how many databases can be open at once (to reduce file descriptor usage); maybe using something similar to the cache code in legacy ov3.c file.
Written by Heath Kehoe <hakehoe@avalon.net> for InterNetNews.
$Id: ovdb.pod 10525 2021-01-20 11:51:15Z iulius $
inn.conf(5), innd(8), makehistory(8), nnrpd(8), ovdb_init(8), ovdb_monitor(8), ovdb_stat(8).
Berkeley DB documentation: in the docs directory of the Berkeley DB source distribution, or on the Oracle Berkeley DB web page (<http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html>).
| 2021-01-21 | INN 2.6.4 |