duct - lightweight wrapper that collects execution data for a
command
usage: duct [-h] [--version] [-p OUTPUT_PREFIX]
- [--summary-format SUMMARY_FORMAT] [--colors] [--clobber] [-l
{NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}] [-q] [--sample-interval
SAMPLE_INTERVAL] [--report-interval REPORT_INTERVAL] [--fail-time
FAIL_TIME] [-c {all,none,stdout,stderr}] [-o {all,none,stdout,stderr}] [-t
{all,system-summary,processes-samples}] command [command_args ...]
...
duct is a lightweight wrapper that collects execution data for an
arbitrary command. Execution data includes execution time, system
information, and resource usage statistics of the command and all its child
processes. It is intended to simplify the problem of recording the resources
necessary to execute a command, particularly in an HPC environment.
Resource usage is determined by polling (at a sample-interval).
During execution, duct produces a JSON lines (see https://jsonlines.org)
file with one data point recorded for each report (at a
report-interval).
- Many duct options can be configured by environment variables (which are
overridden by command line options).
- DUCT_LOG_LEVEL: see --log-level DUCT_OUTPUT_PREFIX: see
--output-prefix DUCT_SUMMARY_FORMAT: see --summary-format
DUCT_SAMPLE_INTERVAL: see --sample-interval DUCT_REPORT_INTERVAL:
see --report-interval DUCT_CAPTURE_OUTPUTS: see
--capture-outputs
- -h, --help
- show this help message and exit
- --version
- show program's version number and exit
- -p, --output-prefix
OUTPUT_PREFIX
- File string format to be used as a prefix for the files -- the
captured stdout and stderr and the resource usage logs. The understood
variables are {datetime}, {datetime_filesafe}, and {pid}. Leading
directories will be created if they do not exist. You can also provide
value via DUCT_OUTPUT_PREFIX env variable. (default:
.duct/logs/{datetime_filesafe}-{pid}_)
- --summary-format
SUMMARY_FORMAT
- Output template to use when printing the summary following execution.
Accepts custom conversion flags: !S: Converts filesizes to human readable
units, green if measured, red if None. !E: Colors exit code, green if
falsey, red if truthy, and red if None. !X: Colors green if truthy, red if
falsey. !N: Colors green if not None, red if None (default: Summary: Exit
Code: {exit_code!E} Command: {command} Log files location: {logs_prefix}
Wall Clock Time: {wall_clock_time:.3f} sec Memory Peak Usage (RSS):
{peak_rss!S} Memory Average Usage (RSS): {average_rss!S} Virtual Memory
Peak Usage (VSZ): {peak_vsz!S} Virtual Memory Average Usage (VSZ):
{average_vsz!S} Memory Peak Percentage: {peak_pmem:.2f!N}% Memory Average
Percentage: {average_pmem:.2f!N}% CPU Peak Usage: {peak_pcpu:.2f!N}%
Average CPU Usage: {average_pcpu:.2f!N}% )
- --colors
- Use colors in duct output. (default: False)
- --clobber
- Replace log files if they already exist. (default: False)
- -l, --log-level
{NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}
- Level of log output to stderr, use NONE to entirely disable. (default:
INFO)
- -q, --quiet
- [deprecated, use log level NONE] Disable duct logging output (to stderr)
(default: False)
- --sample-interval,
--s-i SAMPLE_INTERVAL
- Interval in seconds between status checks of the running process. Sample
interval must be less than or equal to report interval, and it achieves
the best results when sample is significantly less than the runtime of the
process. (default: 1.0)
- --report-interval,
--r-i REPORT_INTERVAL
- Interval in seconds at which to report aggregated data. (default:
60.0)
- --fail-time,
--f-t FAIL_TIME
- If command fails in less than this specified time (seconds), duct would
remove logs. Set to 0 if you would like to keep logs for a failing command
regardless of its run time. Set to negative (e.g. -1) if you would
like to not keep logs for any failing command. (default: 3.0)
- -c, --capture-outputs
{all,none,stdout,stderr}
- Record stdout, stderr, all, or none to log files. You can also provide
value via DUCT_CAPTURE_OUTPUTS env variable. (default: all)
- -o, --outputs
{all,none,stdout,stderr}
- Print stdout, stderr, all, or none to stdout/stderr respectively.
(default: all)
- -t, --record-types
{all,system-summary,processes-samples}
- Record system-summary, processes-samples, or all (default: all)