| PMLOGEXTRACT(1) | General Commands Manual | PMLOGEXTRACT(1) |
pmlogextract - reduce, extract, concatenate and merge Performance Co-Pilot archives
pmlogextract [-dfmwz] [-c configfile] [-S starttime] [-s samples] [-T endtime] [-v volsamples] [-Z timezone] input [...] output
pmlogextract reads one or more Performance Co-Pilot (PCP) archive logs identified by input and creates a temporally merged and/or reduced PCP archive log in output. input is a comma-separated list of names, each of which may be the base name of an archive or the name of a directory containing one or more archives. The nature of merging is controlled by the number of input archive logs, while the nature of data reduction is controlled by the command line arguments. The input(s) must be sets of PCP archive logs created by pmlogger(1) with performance data collected from the same host, but usually over different time periods and possibly (although not usually) with different performance metrics being logged.
If only one input is specified, then the default behavior simply copies the input set of PCP archive logs, into the output PCP archive log. When two or more sets of PCP archive logs are specified as input, the sets of logs are merged (or concatenated) and written to output.
In the output archive log a <mark> record may be inserted at a time just past the end of each of the input archive logs to indicate a possible temporal discontinuity between the end of one input archive log and the start of the next input archive log. See the MARK RECORDS section below for more information. There is no <mark> record after the end of the last (in temporal order) of the input archive logs.
The command line options for pmlogextract are as follows:
Independent of any -v option, each volume of an archive is limited to no more than 2^31 bytes, so pmlogextract will automatically create a new volume for the archive before this limit is reached.
The configfile contains metrics of interest - only those metrics (or instances) mentioned explicitly or implicitly in the configuration file will be included in the output archive. Each specifications must begin on a new line, and may span multiple lines in the configuration file. Instances may also be specified, but they are optional. The format for each specification is
metric [[instance[,instance...]]]
where metric may be a leaf or a non-leaf name in the Performance Metrics Name Space (PMNS, see pmns(5)). If a metric refers to a non-leaf node in the PMNS, pmlogextract will recursively descend the PMNS and include all metrics corresponding to descendent leaf nodes.
Instances are optional, and may be specified as a list of one or more space (or comma) separated names, numbers or strings (enclosed in single or double quotes). Elements in the list that are numbers are assumed to be internal instance identifiers - see pmGetInDom(3) for more information. If no instances are given, then all instances of the associated metric(s) will be extracted.
Any additional white space is ignored and comments may be added with a `#' prefix.
This is an example of a valid configfile:
#
# config file for pmlogextract
#
kernel.all.cpu
kernel.percpu.cpu.sys ["cpu0","cpu1"]
disk.dev ["dks0d1"]
When more than one input archive log contributes performance data to the output archive log, then <mark> records may be inserted to indicate a possible discontinuity in the performance data.
A <mark> record contains a timestamp and no performance data and is used to indicate that there is a time period in the PCP archive log where we do not know the values of any performance metrics, because there was no pmlogger(1) collecting performance data during this period. Since these periods are often associated with the restart of a service or pmcd(1) or a system, there may be considerable doubt as to the continuity of performance data across this time period.
Most current archives are created with a prologue record at the beginning and an epilogue record at the end. These records identify the state of pmcd(1) at the time, and may be used by pmlogextract to determine that there is no discontinuity between the end of one archive and the next output record, and as a consequence the <mark> record can safely be omitted from the output archive.
The rationale behind <mark> records may be demonstrated with an example. Consider one input archive log that starts at 00:10 and ends at 09:15 on the same day, and another input archive log that starts at 09:20 on the same day and ends at 00:10 the following morning. This would be a very common case for archives managed and rotated by pmlogger_check(1) and pmlogger_daily(1).
The output archive log created by pmlogextract would
contain:
00:10.000 first record from first input archive log
...
09:15.000 last record from first input archive log
09:15.001 <mark> record
09:20.000 first record from second input archive log
...
01:10.000 last record from second input archive log
The time period where the performance data is missing starts just after 09:15 and ends just before 09:20. When the output archive log is processed with any of the PCP reporting tools, the <mark> record is used to indicate a period of missing data. For example using the output archive above, imagine one was reporting the average I/O rate at 30 minute intervals aligned on the hour and half-hour. The I/O count metric is a counter, so the average I/O rate requires two valid values from consecutive sample times. There would be values for all the intervals ending at 09:00, then no values at 09:30 because of the <mark> record, then no values at 10:00 because the ``prior'' value at 09:30 is not available, then the rate would be reported again at 10:30 and continue every 30 minutes until the last reported value at 01:00.
The presence of <mark> records in a PCP archive log can be established using pmdumplog(1) where a timestamp and the annotation <mark> is used to indicate a <mark> record.
When more than one input archive set is specified, pmlogextract performs a number of checks to ensure the metadata is consistent for metrics appearing in more than one of the input archive sets. These checks include:
If any of these checks fail, pmlogextract reports the details and aborts without creating the output archive.
To address these semantic issues, use pmlogrewrite(1) to translate the input archives into equivalent archives with consistent metdadata before using pmlogextract.
For each of the input and output archive logs, several physical files are used.
Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).
PCPIntro(1), pmdumplog(1), pmlc(1), pmlogger(1), pmlogreduce(1), pmlogrewrite(1), pcp.conf(5) and pcp.env(5).
All error conditions detected by pmlogextract are reported on stderr with textual (if sometimes terse) explanation.
Should one of the input archive logs be corrupted (this can happen if the pmlogger instance writing the log suddenly dies), then pmlogextract will detect and report the position of the corruption in the file, and any subsequent information from that archive log will not be processed.
If any error is detected, pmlogextract will exit with a non-zero status.
The preamble metrics (pmcd.pmlogger.archive, pmcd.pmlogger.host, and pmcd.pmlogger.port), which are automatically recorded by pmlogger at the start of the archive, may not be present in the archive output by pmlogextract. These metrics are only relevant while the archive is being created, and have no significance once recording has finished.
| PCP | Performance Co-Pilot |