autopkgtest - test an installed binary package using the source
package's tests
autopkgtest [options...] [testbinary...]
testsrc -- virt-server [virt-server-arg...]
autopkgtest runs tests on binary Debian or Click packages, as
installed on a system (called "testbed"). The tests are supplied
in the source package.
autopkgtest runs each test supplied by a particular package and
reports the results. It drives the specified virtualisation regime as
appropriate, parses the test description metadata, and arranges for data to
be copied to and from the testbed as required.
See /usr/share/doc/autopkgtest/README.running-tests.rst.gz for an
introduction about how to use autopkgtest.
Positional (non-option) arguments specify exactly one source
package (containing the test code) and optionally some binary packages to
test.
testsrc can be one of:
- .dsc file
- Run tests from Debian .dsc source package. By default the package
will also be built and the resulting binaries will be used to satisfy test
dependencies; to disable that, specify the
-B/--no-built-binaries option.
- source package
directory
- Run tests from a Debian source tree directory. If that is an unbuilt tree,
this is very similar to specifying a .dsc. If that is a built tree,
all test dependencies get satisfied by archive packages, unless you
explicitly specify locally built .debs as well.
Attention: If you just specify a bare directory name
which is a legal Debian source package name, it will be interpreted as
the latter (see below). In this case, prefix the directory name with
./.
- current
directory
- If no source package is specified on the command line and the current
directory is a Debian source package, this will be tested.
- source package
name
- Downloads the given source package name with apt-get source in the
testbed and run its tests. This is similar to specifying a .dsc but avoids
copying the source from the host to the testbed. Possibly built binaries
(if the test specifies needs-build) will not be used to
satisfy dependencies, as usually in this mode you want to test binaries
from a real archive.
- git URL or
URL#branch
- Git-clones the given URL (which must contain an unbuilt Debian source
tree) and runs the tests from that. If branch is given, this branch
will be checked out instead of the default (usually "master").
This can also be a more general refspec such as #refs/pull/123/head"
for a GitHub pull request.
This is very similar to cloning manually and specifying the
checkout directory as test; i. e. this is commonly used with
--no-built-binaries. The git package will be installed if
necessary.
- .changes file
- Run tests from the .dsc source package in the given .changes file. If the
.changes contains .deb packages, they will be used for the test. Acts as
if you had specified the .debs and .dsc from the .changes file as explicit
arguments. Note that if the .changes contains only debs, the
corresponding .dsc still needs to be specified alongside, or the current
directory must be the source package.
All other positional arguments must be .deb binary packages. They
will be used for both build and test dependencies of the source package. If
any binary package is given, then --no-built-binaries is implied.
This needs exactly one .click package. If its manifest specifies
an x-source URL, the corresponding source package will be downloaded
and tests run from there; then a source directory does not need to be given.
Otherwise specifying a source directory as positional argument is
mandatory.
For testing a local .click package, give its path to
autopkgtest. For testing a preinstalled click, use
--installed-click com.example.myclick (this cannot be given as
positional argument as this cannot be disambiguated from Debian source
package names).
Examples with explicit click source directory:
autopkgtest src/myclick_0.1_all.click com.example.myclick
src/myclick/ -- [...]
autopkgtest --installed-click com.example.myclick src/myclick/ --
[...]
Examples without click source directory; click package needs to
have x-source manifest entry:
autopkgtest src/myclick_0.1_all.click -- [...]
autopkgtest --installed-click com.example.myclick -- [...]
- -B |
--no-built-binaries
- Binaries from unbuilt source packages (see above) will not be built or
ignored, and dependencies are satisfied with packages from the archive.
Note that the source package still gets built if a test requires
build-needed.
- --override-control=PATH
- Read the test metadata from PATH instead of
debian/tests/control (for Debian sources) or the Click manifest.
- --test-name=TEST
- Run only the given test name (from test control file). This replaces
--testname which is deprecated.
If you don't specify any option, autopkgtest only writes
its output/results to stderr.
- -o dir |
--output-dir=dir
- Specifies that test artifacts (stderr and stdout from the tests, the log
file, built binary packages etc.) should be placed in the given directory.
dir must not exist yet or be empty, otherwise autopkgtest
will refuse to use it.
- -l logfile |
--log-file=logfile
- Specifies that the trace log should be written to logfile instead
of to output-dir.
- --summary=summary
- Specifies that a summary of the outcome should be written to
summary. The events in the summary are written to the log in any
case.
- -q | --quiet
- Do not send a copy of autopkgtest's trace logstream to stderr. This
option does not affect the copy sent to logfile or
output-dir. Note that without the trace logstream it can be very
hard to diagnose problems.
- --setup-commands=commands
- Run commands after opening the testbed. This can be used to do
anything that isn't supported by an existing autopkgtest command. If
commands is an existing file name, the commands are read from that;
otherwise it is a string with the actual commands that gets run as-is.
File names without directory will be searched in both the current
directory and in /usr/share/autopkgtest/setup-commands/ so you do
not need to give the full path for setup scripts shipped with autopkgtest.
Normally, if the setup commands fail, autopkgtest will
consider this a transient testbed error (exit code 16). However, if the
setup commands exit with code 100, autopkgtest will consider this an
"erroneous package" (exit code 12) instead, so this can be
used to e. g. detect upgrade errors to a new version. Note that apt
exits with exit code 100 in these cases.
This option can be specified multiple times.
If --user is given or the test bed provides a
suggested-normal-user capability, the
$AUTOPKGTEST_NORMAL_USER environment variable will be set to that
user.
If the setup commands affect anything in boot directories
(like /boot or /lib/systemd/system) and the testbed supports rebooting,
the testbed will be rebooted after the setup commands. This can be
suppressed by creating a file
/run/autopkgtest_no_reboot.stamp.
- --setup-commands-boot=commands
- Run commands after the --setup-commands, and after every
reboot. For example, these commands could be used to add files in a tmpfs.
These commands never cause the testbed to be rebooted (because
that could lead to an infinite loop). Otherwise, they are just like the
--setup-commands.
This option can be specified multiple times.
- --add-apt-source='deb
http://MIRROR SUITE COMPONENT...'
- Add the given apt source to /etc/apt/sources.list.d and update it,
before running any --setup-commands.
This option can be specified multiple times.
- --add-apt-release='RELEASE'
- Add the given apt RELEASE to /etc/apt/sources.list.d and
update it, before running any --setup-commands. The mirror and
components to use are copied from the very first existing APT sources.list
entry. Both binary ("deb") and source ("deb-src")
entries are added.
This option can be specified multiple times.
- --apt-upgrade
| -U
- Run apt-get update and apt-get dist-upgrade -y in the
testbed before running the tests. Any --add-apt-source or
--apt-pocket options take effect first, so this will upgrade
packages from those sources if appropriate.
- --apt-default-release=SUITE
- Set's APT::Default-Release value to the provided value. For apt
pinning (related to --apt-pocket, and --pin-packages) to
work properly, APT::Default-Release must be set to the release that
should provide the packages that are not pinned. For Debian and Ubuntu,
this is normally automatically detected from the first entry in
/etc/apt/sources.list.
- --apt-pocket=pocket[=pkgname,src:srcname,...]
- Add apt sources for release-pocket. This finds the first
deb line in /etc/apt/sources.list which does not already
specify a pocket and adds a deb and deb-src line with that pocket to
/etc/apt/sources.list.d/pocket.list. This also calls
apt-get update for the new pocket (but not for anything else).
If a package list is given after =, set up apt pinning to use
only those packages from pocket. An entry "src:srcname"
expands to all binary packages built by that source. This can be used
for minimizing dependencies taken from pocket so that package
updates in that pocket can be tested independently from each other for
better isolation. Attention: This does not currently resolve some
situations where dependencies of the given packages can only be resolved
in the given pocket. In that case the apt pinning will be removed and
package installation will be retried with the entirety of
pocket.
- --copy=HOSTPATH:TESTBEDPATH
- Copy file or directory from host into testbed after opening. This happens
before --setup-commands thus you can use these files in the setup
commands.
- --env=VAR=value
- Set arbitrary environment variable in the build and test. Can be specified
multiple times.
- --pin-packages=RELEASE=PACKAGE[,PACKAGE2]
- Set up apt pinning to use only those packages from RELEASE. An
entry "src:srcname" expands to all binary packages built by that
source. This can be used for minimizing dependencies taken from
RELEASE so that package updates in that release can be tested
independently from each other for better isolation.
- --no-apt-fallback
- Disable the apt-get fallback which is used with --apt-pocket or
--pin-packages in case installation of dependencies fails due to
strict pinning.
- -u user |
--user=user
- Run builds and tests as user on the testbed. This needs root on the
testbed; if root on the testbed is not available then builds and tests run
as whatever user is provided.
- --gain-root=gain-root
- Prefixes debian/rules binary with gain-root. The default is
not to use anything, except that if --user is supplied or root on
the testbed is not available the default is fakeroot.
- --debug|-d
- Include additional debugging information in the trace log. Each additional
-d increases the debugging level; the current maximum is
-ddd. If you like to see what's going on, -d or
-dd is recommended.
- --shell-fail|-s
- Run an interactive shell in the testbed after a failed build, test, or
dependency installation.
- --shell
- Run an interactive shell in the testbed after every test.
- --timeout-which=seconds
- Use a different timeout for operations on or with the testbed. There are
five timeouts affected by five values of which: short:
supposedly short operations like setting up the testbed's apt and checking
the state (default: 100s); install: installation of packages
including dependencies (default: 3,000s); test: test runs (default:
10,000s); copy: copy files/directories between host and testbed
(default: 300s); and build: builds (default: 100,000s). The value
must be specified as an integer number of seconds.
- --timeout-factor=double
- Multiply all of the default timeouts by the specified factor (see
--timeout-which above). Only the defaults are affected;
explicit timeout settings are used exactly as specified.
- --set-lang=langval
- When running commands on the testbed, sets the LANG environment
variable to langval. The default in autopkgtest is to set it
to C.UTF-8.
- --no-auto-control
- Disable automatic test generation with autodep8, even if it is installed.
In that case, packages without tests will exit with code 8 ("No tests
in this package") just like without autodep8.
- --build-parallel=N
- Set parallel=N DEB_BUILD_OPTION for building packages. By default
this is the number of available processors. This is mostly useful in
containers where you can restrict the available RAM, but not restrict the
number of CPUs.
- -h|--help
- Show command line help and exit.
- --
virt-server virt-server-arg...
- Specifies the virtualisation regime server, as a command and arguments to
invoke. virt-server must be an existing autopkgtest virtualization
server such as schroot or qemu.
All the remaining arguments and options after -- are
passed to the virtualisation server program. See the manpages of the
individual servers for how to use them.
During a normal test run, one line is printed for each test. This
consists of a short string identifying the test, some horizontal whitespace,
and one of PASS, PASS details, FAIL
reason, SKIP reason, or FLAKY reason
where the pass/fail indication is separated by any reason or details by some
horizontal whitespace.
The string to identify the test consists of a short alphanumeric
string invented by autopkgtest to distinguish different command-line
arguments, the argid, followed by a hyphen and the test name.
SKIP indicates that a test was not run, or that the test
code was started but detected that the test could not complete, for instance
because a required resource was not available.
FLAKY indicates that a test would ordinarily have failed,
but because this particular test is known to be unreliable, the failure was
ignored.
Sometimes a SKIP will be reported when the name of the test
is not known or not applicable: for example, when there are no tests in the
package, or a there is a test stanza which contains features not understood
by this version of autopkgtest. In this case * will appear
where the name of the test should be.
If autopkgtest detects that erroneous package(s) are
involved, it will print the two lines blame:
blamed-thing... and badpkg: message. Here
each whitespace-separated blamed-thing is one of
arg:argument (representing a pathname found in a command line
argument), dsc:package (a source package name),
deb:package (a binary package name) or possibly other strings
to be determined. This indicates which arguments and/or packages might have
contributed to the problem; the ones which were processed most recently and
which are therefore most likely to be the cause of a problem are listed
last.
If you use lots of options or nontrivial virt server arguments,
you can put any part of the command line into a text file, with one line per
option. E. g. you can create a file sid.cfg with contents like
-s
--output-dir=/tmp/testout
--apt-upgrade
--
schroot
sid
and then run
autopkgtest foo_1_amd64.changes @sid.cfg
The contents of the configuration file will be expanded in-place
as if you would have given its contents on the command line. Please ensure
that you don't place spaces between short options and their values,
they would become a part of the argument value.
0 all tests passed
2 at least one test was skipped (or at least one flaky test failed)
4 at least one test failed
6 at least one test failed and at least one test skipped
8 no tests in this package, or all non-superficial tests were skipped
12 erroneous package
14 erroneous package and at least one test skipped
16 testbed failure
20 other unexpected failures including bad usage
/usr/share/doc/autopkgtest/README.running-tests.rst.gz
/usr/share/doc/autopkgtest/README.package-tests.rst.gz
This manpage is part of autopkgtest, a tool for testing Debian
binary packages. autopkgtest is Copyright (C) 2006-2014 Canonical Ltd.
See /usr/share/doc/autopkgtest/CREDITS for the list of
contributors and full copying conditions.