autopkgtest - test an installed binary package using the source
package's tests
autopkgtest [options...] [testbinary...]
testsrc -- virt-server [virt-server-arg...]
autopkgtest runs tests on binary Debian packages, as installed on
a system (called "testbed"). The tests are supplied in the source
package.
autopkgtest runs each test supplied by a particular package and
reports the results. It drives the specified virtualisation regime as
appropriate, parses the test description metadata, and arranges for data to
be copied to and from the testbed as required.
See /usr/share/doc/autopkgtest/README.running-tests.rst.gz for an
introduction about how to use autopkgtest.
Positional (non-option) arguments specify exactly one source
package (containing the test code) and optionally some binary packages to
test.
testsrc can be one of:
- .dsc file
- Run tests from Debian .dsc source package. By default the package
will also be built and the resulting binaries will be used to satisfy test
dependencies; to disable that, specify the
-B/--no-built-binaries option.
- source package
directory
- Run tests from a Debian source tree directory. If that is an unbuilt tree,
this is very similar to specifying a .dsc. If that is a built tree,
all test dependencies get satisfied by archive packages, unless you
explicitly specify locally built .debs as well.
Attention: If you just specify a bare directory name
which is a legal Debian source package name, it will be interpreted as
the latter (see below). In this case, prefix the directory name with
./.
- current
directory
- If no source package is specified on the command line and the current
directory is a Debian source package, this will be tested.
- source package
name
- Downloads the given source package name with apt-get source in the
testbed and run its tests. This is similar to specifying a .dsc but avoids
copying the source from the host to the testbed. Possibly built binaries
(if the test specifies build-needed) will not be used to
satisfy dependencies, as usually in this mode you want to test binaries
from a real archive.
- git URL or
URL#branch
- Git-clones the given URL (which must contain an unbuilt Debian source
tree) and runs the tests from that. If branch is given, this branch
will be checked out instead of the default (usually "master").
This can also be a more general refspec such as #refs/pull/123/head"
for a GitHub pull request.
This is very similar to cloning manually and specifying the
checkout directory as test; i. e. this is commonly used with
--no-built-binaries. The git package will be installed if
necessary.
- .changes file
- Run tests from the .dsc source package in the given .changes file. If the
.changes contains .deb packages, they will be used for the test. Acts as
if you had specified the .debs and .dsc from the .changes file as explicit
arguments. Note that if the .changes contains only debs, the
corresponding .dsc still needs to be specified alongside, or the current
directory must be the source package.
All other positional arguments must be .deb binary packages. They
will be used for both build and test dependencies of the source package. If
any binary package is given, then --no-built-binaries is implied.
- -a ARCH |
--test-architecture=ARCH
-
Make the testbed a multiarch system, set up the environment
for cross-building (possibly installing extra build dependencies), and
run tests for the specified foreign architecture rather than for the
testbed native architecture. Assumes that ARCH binaries can be run on
the testbed, possibly via qemu-user-binfmt or qemu-user-static.
- -B |
--no-built-binaries
- Binaries from unbuilt source packages (see above) will not be built or
ignored, and dependencies are satisfied with packages from the archive.
Note that the source package still gets built if a test requires
build-needed.
- --override-control=PATH
- Read the test metadata from PATH instead of
debian/tests/control.
- --test-name=TEST
- Run only the given test name (from test control file). If this option is
used more than once, all the named tests are run.
- --skip-test=TEST
- Skip the given test name (from test control file). If this option is used
more than once, all the named tests are skipped.
If you don't specify any option, autopkgtest only writes
its output/results to stderr.
- -o dir |
--output-dir=dir
- Specifies that test artifacts (stderr and stdout from the tests, the log
file, built binary packages etc.) should be placed in the given directory.
dir must not exist yet or be empty, otherwise autopkgtest
will refuse to use it.
- -l logfile |
--log-file=logfile
- Specifies that the trace log should be written to logfile instead
of to output-dir.
- --summary=summary
- Specifies that a summary of the outcome should be written to
summary. The events in the summary are written to the log in any
case.
- -q | --quiet
- Do not send a copy of autopkgtest's trace logstream to stderr. This
option does not affect the copy sent to logfile or
output-dir. Note that without the trace logstream it can be very
hard to diagnose problems.
- --setup-commands=commands
- Run commands before running any test, but after adding and updating
APT sources specified via --add-apt-source,
--add-apt-release, or --apt-pocket. This can be used to do
anything that isn't supported by an existing autopkgtest option. If
commands is an existing file name, the commands are read from that;
otherwise it is a string with the actual commands that gets run as-is.
File names without directory will be searched in both the current
directory and in /usr/share/autopkgtest/setup-commands/ so you do
not need to give the full path for setup scripts shipped with autopkgtest.
Normally, if the setup commands fail, autopkgtest will
consider this a transient testbed error (exit code 16). However, if the
setup commands exit with code 100, autopkgtest will consider this an
"erroneous package" (exit code 12) instead, so this can be
used to e. g. detect upgrade errors to a new version. Note that apt
exits with exit code 100 in these cases.
This option can be specified multiple times.
If --user is given or the test bed provides a
suggested-normal-user capability, the
$AUTOPKGTEST_NORMAL_USER environment variable will be set to that
user.
If the setup commands affect anything in boot directories
(like /boot or /lib/systemd/system) and the testbed supports rebooting,
the testbed will be rebooted after the setup commands. This can be
suppressed by creating a file
/run/autopkgtest_no_reboot.stamp.
Warning: The way that autopkgtest uses this user means
it will be completely trusted by root inside the testbed.
- --setup-commands-boot=commands
- Run commands after the --setup-commands, and after every
reboot. For example, these commands could be used to add files in a tmpfs.
These commands never cause the testbed to be rebooted (because
that could lead to an infinite loop). Otherwise, they are just like the
--setup-commands.
This option can be specified multiple times.
- --add-apt-source='deb|deb-src
[ options ] MIRROR SUITE COMPONENT...'
- --add-apt-source=ppa:[USER:TOKEN@]OWNER/PPA[:FINGERPRINT]
- --add-apt-source=FILENAME
-
Add the given apt source to the APT sources.list. Specifying
this option triggers an apt-get update. Multiple sources can be
specified at once by separating them with a newline, but specifying this
option multiple times is to be preferred instead.
When the ppa: syntax is used, both binary
("deb") and source ("deb-src") entries are added.
Also, when USER and TOKEN are specified, then the PPA in
assumed to be private and FINGERPRINT becomes a mandatory field.
Note that autopkgtest can leak TOKEN in many ways, notably when
running in debug mode. If credentials are private then autopkgtest
should be run on a trusted platform.
When the FILENAME syntax is used, the contents of the
specified file are configured as sources on the testbed. FILENAME
should end with .list or .sources.
- --add-apt-release=RELEASE
- Add the given RELEASE to the APT sources.list. Mirrors, components
and options are taken from the release appearing first in the existing APT
sources.list; any "deb" sources.list entry for this release will
used as a template for adding RELEASE. Both binary
("deb") and source ("deb-src") entries are added.
Specifying this option triggers an apt-get update. This option can
be specified multiple times.
Note that autopkgtest ignores the "NotAutomatic:
yes" setting that added releases may declare, e.g. Debian
"experimental", the Debian "backports" pocket, or
the Ubuntu "proposed" pocket since 24.04 LTS (Noble
Numbat).
- --apt-pocket=POCKET[=PACKAGELIST]
- Add the given POCKET to the APT sources.list. The pocket
"base" release assumed to be the first configured release in the
APT sources.list. This is equivalent to specifying
--add-apt-release=BASE-POCKET where BASE is the
pocket base release.
Specifying PACKAGELIST allows to pull only specific
packages from the pocket. It is equivalent to
--pin-packages=BASE-POCKET=PACKAGELIST,
where BASE is the pocket base release as defined above. Refer to
--pin-packages for more information on the PACKAGELIST
syntax.
- --pin-packages=RELEASE=PACKAGE[,PACKAGE2,...]
- Set up apt pinning to use only specified packages from RELEASE.
This can be used for minimizing dependencies taken from RELEASE so
that package updates in that release can be tested independently from each
other for better isolation. If PACKAGE is specified as
src:SRCPACKAGE, pinning will cover all binary packages built
by source package SRCPACKAGE.
Attention: This does not currently resolve some
situations where dependencies of the given packages can only be resolved
in RELEASE. In this case the apt pinning will be removed and
package installation will be retried with the entirety of
RELEASE, unless --no-apt-fallback is specified.
- --no-apt-fallback
- Disable the apt-get fallback which is used with --pin-packages or
--apt-pocket in case installation of dependencies fails due to
strict pinning.
- --apt-upgrade |
-U
- Run apt-get update, upgrade, autopurge in the testbed
before running the tests. Any --add-apt-source or
--apt-pocket options take effect first, so this will upgrade
packages from those sources if appropriate.
- --apt-default-release=SUITE
- Select the release that should provide the test dependencies that are not
explicitly pinned via --apt-pocket or --pin-packages.
Specifying this option is normally not necessary: suitable pinning it
configured automatically. This option is equivalent to setting
APT::Default-Release.
- --copy=HOSTPATH:TESTBEDPATH
- Copy file or directory from host into testbed after opening. This happens
before --setup-commands thus you can use these files in the setup
commands.
- --env=VAR=value
- Set arbitrary environment variable in the build and test. Can be specified
multiple times.
- --ignore-restrictions=RESTRICTION,RESTRICTION...
- If a test would normally be skipped because it has Restrictions:
RESTRICTION, run it anyway. Can be specified multiple times.
For example, you might ignore the restriction
isolation-machine when using the null virtualization
server if you know that autopkgtest itself is running on an
expendable virtual machine. These options also work for unknown
restrictions, so they can be used when experimenting with new
restrictions.
- -u user |
--user=user
- Run builds and tests as user on the testbed. This needs root on the
testbed; if root on the testbed is not available then builds and tests run
as whatever user is provided.
- --gain-root=gain-root
- Prefixes debian/rules binary with gain-root. The default is
not to use anything, except that if --user is supplied or root on
the testbed is not available the default is fakeroot.
- --insecure
- Relaxes file permissions etc., to try to make things not fail with
permission errors.
Passing this option might allow tests to corrupt the testbed
(if it doesn't support resetting), or to escape from it, or for
untrusted processes on the host or the testbed to take over the account
running autopkgtest.
It is safe to pass --insecure on a dedicated throwaway
host.
This partially restores the behaviour of autopkgtest 5.47 and
earlier, which had several world-writable directories.
- --debug|-d
- Include additional debugging information in the trace log. Each additional
-d increases the debugging level; the current maximum is
-ddd. If you like to see what's going on, -d or
-dd is recommended.
- --shell-fail|-s
- Run an interactive shell in the testbed after a failed build, test, or
dependency installation.
- --shell
- Run an interactive shell in the testbed after every test.
- --timeout=seconds
- Define the global timeout, limiting the time allowed to perform the build
(if one is required) and run the tests. If the global timeout is reached
during the build, the run will be aborted in the same way as if the build
had failed. If the global timeout is exceeded while running a test, the
test is aborted and any remaining tests are skipped. (default: 0s, meaning
a global timeout is not in effect). The value must be specified as an
integer number of seconds.
- --timeout-which=seconds
- Use a different timeout for operations on or with the testbed. There are
five timeouts affected by five values of which: short:
supposedly short operations like setting up the testbed's apt and checking
the state (default: 100s); install: installation of packages
including dependencies (default: 3,000s); test: test runs (default:
10,000s); copy: copy files/directories between host and testbed
(default: 300s); and build: builds (default: 100,000s). The value
must be specified as an integer number of seconds.
- --timeout-factor=double
- Multiply all of the default timeouts by the specified factor (see
--timeout-which above). Only the defaults are affected;
explicit timeout settings are used exactly as specified.
- --set-lang=langval
- When running commands on the testbed, sets the LANG environment
variable to langval. The default in autopkgtest is to set it
to C.UTF-8.
- --no-auto-control
- Disable automatic test generation with autodep8, even if it is installed.
In that case, packages without tests will exit with code 8 ("No tests
in this package") just like without autodep8.
- --build-parallel=N
- Set parallel=N DEB_BUILD_OPTION for building packages. By default
this is the number of available processors. This is mostly useful in
containers where you can restrict the available RAM, but not restrict the
number of CPUs.
- --needs-internet=run|try|skip
- Define how to handle the needs-internet restriction. With "try"
tests with needs-internet restrictions will be run, but if they fail they
will be treated as flaky tests. With "skip" these tests will
skipped immediately and will not be run. With "run" the
restriction is basically ignored, this is the default.
- -V|--validate
- Validate the test control file and exit without running any tests.
- -h|--help
- Show command line help and exit.
- -- virt-server
virt-server-arg...
- Specifies the virtualisation regime server, as a command and arguments to
invoke.
In the autopkgtest invocation, all the arguments and options
after -- are passed to the virtualisation server.
The autopkgtest package provides several virtualisation servers,
named autopkgtest-virt-* (e.g. autopkgtest-virt-qemu,
autopkgtest-virt-unshare, ...), each coming with its own manpage.
When one if these virtualisation servers is to be used, the
autopkgtest-virt- prefix in the command can be omitted.
The virtualisation server choice depends on several factors, such
as the desired level of isolation of the testbed and the user's preference
towards specific tools. If unsure, the qemu virt server offers VM
isolation (thus allowing tests with the isolation-machine restriction) and
only relies on QEMU being available. See autopkgtest-virt-qemu(1) for more
information.
During a normal test run, one line is printed for each test. This
consists of a short string identifying the test, some horizontal whitespace,
and one of PASS, PASS details, FAIL
reason, SKIP reason, or FLAKY reason
where the pass/fail indication is separated by any reason or details by some
horizontal whitespace.
The string to identify the test consists of a short alphanumeric
string invented by autopkgtest to distinguish different command-line
arguments, the argid, followed by a hyphen and the test name.
SKIP indicates that a test was not run, or that the test
code was started but detected that the test could not complete, for instance
because a required resource was not available.
FLAKY indicates that a test would ordinarily have failed,
but because this particular test is known to be unreliable, the failure was
ignored.
Sometimes a SKIP will be reported when the name of the test
is not known or not applicable: for example, when there are no tests in the
package, or a there is a test stanza which contains features not understood
by this version of autopkgtest. In this case * will appear
where the name of the test should be.
If autopkgtest detects that erroneous package(s) are
involved, it will print the two lines blame:
blamed-thing... and badpkg: message. Here
each whitespace-separated blamed-thing is one of
arg:argument (representing a pathname found in a command line
argument), dsc:package (a source package name),
deb:package (a binary package name) or possibly other strings
to be determined. This indicates which arguments and/or packages might have
contributed to the problem; the ones which were processed most recently and
which are therefore most likely to be the cause of a problem are listed
last.
Other possible errors are: erroneous package:
message, testbed failure: message, or quitting:
message.
If you use lots of options or nontrivial virt server arguments,
you can put any part of the command line into a text file, with one line per
option. E. g. you can create a file sid.cfg with contents like
-s
--output-dir=/tmp/testout
--apt-upgrade
--
qemu
/path/to/sid.img
and then run
autopkgtest foo_1_amd64.changes @sid.cfg
The contents of the configuration file will be expanded in-place
as if you would have given its contents on the command line. Please ensure
that you don't place spaces between short options and their values,
they would become a part of the argument value.
0 all tests passed
2 at least one test was skipped (or at least one flaky test failed)
4 at least one test failed
6 at least one test failed and at least one test skipped
8 no tests in this package, or all non-superficial tests were skipped
12 erroneous package
14 erroneous package and at least one test skipped
16 testbed failure
20 other unexpected failures including bad usage
/usr/share/doc/autopkgtest/README.running-tests.rst.gz
/usr/share/doc/autopkgtest/README.package-tests.rst.gz
This manpage is part of autopkgtest, a tool for testing Debian
binary packages. autopkgtest is Copyright (C) 2006-2014 Canonical Ltd.
See /usr/share/doc/autopkgtest/CREDITS for the list of
contributors and full copying conditions.