| Test::Cmd(3pm) | User Contributed Perl Documentation | Test::Cmd(3pm) |
Test::Cmd - Perl module for portable testing of commands and scripts
An example using Test::More with this module to run a command and then test the exit code, standard out, and standard error:
use Test::Cmd; use Test::More tests => 3; my $test = Test::Cmd->new( prog => 'outerr', workdir => '' ); $test->run(); is( $test->stdout, "out\n", 'standard out' ); is( $test->stderr, "err\n", 'standard error' ); is( $? >> 8, 1, 'exit status' );
Where "outerr" is the shell script:
$ cat outerr #!/bin/sh echo out echo >&2 err exit 1 $ chmod +x outerr
See below for other examples. Otherwise, the full list of available methods is:
use Test::Cmd;
$test = Test::Cmd->new(prog => 'program_or_script_to_test',
interpreter => 'script_interpreter',
string => 'identifier_string',
workdir => '',
subdir => 'dir',
match_sub => $code_ref,
verbose => 1);
$test->verbose(1);
$test->prog('program_or_script_to_test');
$test->basename(@suffixlist);
$test->interpreter('script_interpreter');
$test->string('identifier string');
$test->workdir('prefix');
$test->workpath('subdir', 'file');
$test->subdir('subdir', ...);
$test->subdir(['sub', 'dir'], ...);
$test->write('file', <<'EOF');
contents of file
EOF
$test->write(['subdir', 'file'], <<'EOF');
contents of file
EOF
$test->read(\$contents, 'file');
$test->read(\@lines, 'file');
$test->read(\$contents, ['subdir', 'file']);
$test->read(\@lines, ['subdir', 'file']);
$test->writable('dir');
$test->writable('dir', $rwflag);
$test->writable('dir', $rwflag, \%errors);
$test->preserve(condition, ...);
$test->cleanup(condition);
$test->run(prog => 'program_or_script_to_test',
interpreter => 'script_interpreter',
chdir => 'dir', args => 'arguments', stdin => <<'EOF');
input to program
EOF
$test->pass(condition);
$test->pass(condition, \&func);
$test->fail(condition);
$test->fail(condition, \&func);
$test->fail(condition, \&func, $caller);
$test->no_result(condition);
$test->no_result(condition, \&func);
$test->no_result(condition, \&func, $caller);
$test->stdout;
$test->stdout($run_number);
$test->stderr;
$test->stderr($run_number);
$test->match(\@lines, \@matches);
$test->match($lines, $matches);
$test->match_exact(\@lines, \@matches);
$test->match_exact($lines, $matches);
$test->match_regex(\@lines, \@regexes);
$test->match_regex($lines, $regexes);
$test->diff_exact(\@lines, \@matches, \@output);
$test->diff_exact($lines, $matches, \@output);
$test->diff_regex(\@lines, \@regexes, \@output);
$test->diff_regex($lines, $regexes, \@output);
sub func {
my ($self, $lines, $matches) = @_;
# code to match $lines and $matches
}
$test->match_sub(\&func);
$test->match_sub(sub { code to match $_[1] and $_[2] });
$test->here;
The "Test::Cmd" module provides a low-level framework for portable automated testing of executable commands and scripts (in any language, not just Perl), especially commands and scripts that interact with the file system.
The "Test::Cmd" module makes no assumptions about what constitutes a successful or failed test. Attempting to read a file that doesn't exist, for example, may or may not be an error, depending on the software being tested.
Consequently, no "Test::Cmd" methods (including the "new()" method) exit, die or throw any other sorts of exceptions (but they all do return useful error indications). Exceptions or other error status should be handled by a higher layer: a subclass of Test::Cmd, or another testing framework such as the Test or Test::Simple Perl modules, or by the test itself.
(That said, see the Test::Cmd::Common module if you want a similar module that provides exception handling, either to use directly in your own tests, or as an example of how to use "Test::Cmd".)
In addition to running tests and evaluating conditions, the "Test::Cmd" module manages and cleans up one or more temporary workspace directories, and provides methods for creating files and directories in those workspace directories from in-line data (that is, here-documents), allowing tests to be completely self-contained. When used in conjunction with another testing framework, the "Test::Cmd" module can function as a fixture (common startup code for multiple tests) for simple management of command execution and temporary workspaces.
The "Test::Cmd" module inherits File::Spec methods ("file_name_is_absolute()", "catfile()", etc.) to support writing tests portably across a variety of operating and file systems.
A "Test::Cmd" environment object is created via the usual invocation:
$test = Test::Cmd->new();
Arguments to the "Test::Cmd::new" method are keyword-value pairs that may be used to initialize the object, typically by invoking the same-named method as the keyword.
As mentioned, because the "Test::Cmd" module makes no assumptions about what constitutes success or failure of a test, it can be used to provide temporary workspaces, other file system interaction, or command execution for a variety of testing frameworks. This section describes how to use the "Test::Cmd" with several different higher-layer testing frameworks.
Note that you should not intermix multiple testing frameworks in a single testing script.
The "Test::Cmd" module may be used in tests that print results in a format suitable for the standard Perl Test::Harness module:
use Test::Cmd;
print "1..5\n";
$test = Test::Cmd->new(prog => 'test_program', workdir => '');
if ($test) { print "ok 1\n"; } else { print "not ok 1\n"; }
$input = <<_EOF;
test_program should process this input
and exit successfully (status 0).
_EOF_
$wrote_file = $test->write('input_file', $input);
if ($wrote_file) { print "ok 2\n"; } else { print "not ok 2\n"; }
$test->run(args => '-x input_file');
if ($? == 0) { print "ok 3\n"; } else { print "not ok 3\n"; }
$wrote_file = $test->write('input_file', $input);
if ($wrote_file) { print "ok 4\n"; } else { print "not ok 4\n"; }
$test->run(args => '-y input_file');
if ($? == 0) { print "ok 5\n"; } else { print "not ok 5\n"; }
Several other Perl modules simplify the use of Test::Harness by eliminating the need to hand-code the "print" statements and test numbers. The Test module, the Test::Simple module, and the Test::More module all export an "ok()" subroutine to test conditions. Here is how the above example would look rewritten to use Test::Simple:
use Test::Simple tests => 5;
use Test::Cmd;
$test = Test::Cmd->new(prog => 'test_program', workdir => '');
ok($test, "creating Test::Cmd object");
$input = <<_EOF;
test_program should process this input
and exit successfully (status 0).
_EOF_
$wrote_file = $test->write('input_file', $input);
ok($wrote_file, "writing input_file");
$test->run(args => '-x input_file');
ok($? == 0, "executing test_program -x input_file");
$wrote_file = $test->write('input_file', $input);
ok($wrote_file, "writing input_file");
$test->run(args => '-y input_file');
ok($? == 0, "executing test_program -y input_file");
The Perl Test::Unit package provides a procedural testing interface modeled after a testing framework widely used in the eXtreme Programming development methodology. The "Test::Cmd" module can function as part of a Test::Unit fixture that can set up workspaces as needed for a set of tests. This avoids having to repeat code to re-initialize an input file multiple times:
use Test::Unit;
use Test::Cmd;
my $test;
$input = <<'EOF';
test_program should process this input
and exit successfully (status 0).
EOF
sub set_up {
$test = Test::Cmd->new(prog => 'test_program', workdir => '');
$test->write('input_file', $input);
}
sub test_x {
my $result = $test->run(args => '-x input_file');
assert($result == 0, "failed test_x\n");
}
sub test_y {
my $result = $test->run(args => '-y input_file');
assert($result == 0, "failed test_y\n");
}
create_suite();
run_suite;
Note that, because the "Test::Cmd" module takes care of cleaning up temporary workspaces on exit, there is no need to remove explicitly the workspace in a "tear_down" subroutine. (There may, of course, be other things in the test that need a "tear_down" subroutine.)
Alternatively, the "Test::Cmd" module provides "pass()", "fail()", and "no_result()" methods that can be used to provide an appropriate exit status and simple printed indication for a test. These methods terminate the test immediately, reporting "PASSED", "FAILED", or "NO RESULT" respectively, and exiting with status 0 (success), 1 or 2 respectively.
The separate "fail()" and "no_result()" methods allow for a distinction between an actual failed test and a test that could not be properly evaluated because of an external condition (such as a full file system or incorrect permissions).
The exit status values happen to match the requirements of the Aegis change management system, and the printed strings are based on existing Aegis conventions. They are not really Aegis-specific, however, and provide a simple, useful starting point if you don't already have another testing framework:
use Test::Cmd;
$test = Test::Cmd->new(prog => 'test_program', workdir => '');
Test::Cmd->no_result(! $test);
$input = <<EOF;
test_program should process this input
and exit successfully (status 0).
EOF
$wrote_file = $test->write('input_file', $input);
$test->no_result(! $wrote_file);
$test->run(args => '-x input_file');
$test->fail($? != 0);
$wrote_file = $test->write('input_file', $input);
$test->no_result(! $wrote_file);
$test->run(args => '-y input_file');
$test->fail($? != 0);
$test->pass;
Note that the separate Test::Cmd::Common wrapper module can simplify the above example even further by taking care of common exception handling cases within the testing object itself.
use Test::Cmd::Common;
$test = Test::Cmd::Common->new(prog => 'test_program', workdir => '');
$input = <<EOF;
test_program should process this input
and exit successfully (status 0).
EOF
$wrote_file = $test->write('input_file', $input);
$test->run(args => '-x input_file');
$wrote_file = $test->write('input_file', $input);
$test->run(args => '-y input_file');
$test->pass;
See the Test::Cmd::Common module for details.
Methods supported by the "Test::Cmd" module include:
Returns the absolute pathname to the temporary working directory, or FALSE if the directory could not be created.
$test->subdir('sub', ['sub', 'dir'], [qw(sub dir ectory)]);
Returns the number of subdirectories actually created.
Returns TRUE on successfully opening and reading the file, FALSE otherwise.
Typically, this method is not called directly, but is used when the script exits to clean up temporary working directories as appropriate for the exit status.
Arguments are supplied as keyword-value pairs:
$test->run(args => 'arg1 arg2');
$test->run(chdir => 'xyzzy');
If the specified path is not an absolute path name (begins with '/' on Unix systems), then the subdirectory is relative to the temporary working directory for the environment ("$test-&"workdir>). Note that, by default, the "Test::Cmd" module does NOT chdir to the temporary working directory, so to execute the test under the temporary working directory, you must specify an explicit "chdir" to the current directory:
$test->run(chdir => '.'); # Unix-specific
$test->run(chdir => $test->curdir); # portable
$test->run(stdin => <<_EOF_);
input to the program under test
_EOF_
Returns the exit status of the program or script.
Returns TRUE if each line matched its corresponding line in the other array, FALSE otherwise.
Returns TRUE if each line matched each regular expression, FALSE otherwise.
If the Algorithm::DiffOld package is installed on the local system, output describing the differences between the input lines and the matching lines, in diff(1) format, is saved to the $output array reference. In the diff output, the expected output lines are considered the "old" (left-hand) file, and the actual output is considered the "new" (right-hand) file.
If the Algorithm::DiffOld package is not installed on the local system, the Expected and Actual contents are saved as-is to the $output array reference.
The "lines" and "matches" arguments are passed in as either scalars, in which case each is split on newline boundaries, or as array references. Trailing newlines are stripped from each line and regular expression.
Returns TRUE if each line matched its corresponding line in the expected matches, FALSE otherwise, in order to conform to the conventions of the "match" method.
Typical invocation:
if (! $test->diff_exact($test->stdout,
\@expected_lines,
\@diff)) {
print @diff;
}
If the Algorithm::DiffOld package is installed on the local system, output describing the differences between the input lines and the matching lines, in diff(1) format, is saved to the $output array reference. In the diff output, the expected output lines are considered the "old" (left-hand) file, and the actual output is considered the "new" (right-hand) file.
If the Algorithm::DiffOld package is not installed on the local system, the Expected and Actual contents are saved as-is to the $output array reference.
The "lines" and "regexes" arguments are passed in as either scalars, in which case each is split on newline boundaries, or as array references. Trailing newlines are stripped from each line and regular expression. Comparison is performed for each entire line, that is, with each regular expression anchored at both the start of line (^) and end of line ($).
Returns TRUE if each line matched each regular expression, FALSE otherwise, in order to conform to the conventions of the "match" method.
Typical invocation:
if (! $test->diff_regex($test->stdout,
\@expected_lines,
\@diff)) {
print @diff;
}
$test->match_sub(\&Test::Cmd::match_exact);
$test->match_sub(\&Test::Cmd::match_regex);
$test->match_sub(\&Test::Cmd::diff_exact);
$test->match_sub(\&Test::Cmd::diff_regex);
The "match_exact", "match_regex", "diff_exact" and "diff_regex" subroutine names are exportable from the "Test::Cmd" module, and may be specified at object initialization:
use Test::Cmd qw(match_exact match_regex diff_exact diff_regex);
$test_exact = Test::Cmd->new(match_sub => \&match_exact);
$test_regex = Test::Cmd->new(match_sub => \&match_regex);
$test_exact = Test::Cmd->new(match_sub => \&diff_exact);
$test_regex = Test::Cmd->new(match_sub => \&diff_regex);
Several environment variables affect the default values in a newly created "Test::Cmd" environment object. These environment variables must be set when the module is loaded, not when the object is created.
Although the "Test::Cmd" module is intended to make it easier to write portable tests for portable utilities that interact with file systems, it is still very easy to write non-portable tests if you're not careful.
The best and most comprehensive set of portability guidelines is the standard "Writing portable Perl" document at:
http://www.perl.com/pub/doc/manual/html/pod/perlport.html
To reiterate one important point from the "WpP" document: Not all Perl programs have to be portable. If the program or script you're testing is UNIX-specific, you can (and should) use the "Test::Cmd" module to write UNIX-specific tests.
That having been said, here are some hints that may help keep your tests portable, if that's a requirement.
Passing in a file name that has had its directory separators altered, however, may confuse the command or script under test, or make it difficult to compare output from the command or script with an expected result. The "Test::Cmd::here" method returns the absolute path name of the current working directory, like "Cwd::cwd", but does not manipulate the returned path in any way.
if (! Test::Cmd->file_name_is_absolute($prog)) {
my $prog = Test::Cmd->catfile(Test::Cmd->here, $prog);
}
For details about the available methods and their use, see the documentation for the File::Spec module and its sub-modules, especially the File::Spec::Unix modules.
$foo_exe = "foo$Config{_exe}";
ok(-f $foo_exe);
(Unfortunately, there is no existing $Config value that specifies the suffix for a directly-executable Perl script.)
If your test somehow requires executing a script that you generate from the test itself, the best way is to generate the script in Perl and then explicitly feed it to the Perl executable on the local system. To be maximally portable, use the $^X variable instead of hard-coding "perl" into the string you execute:
$line = "This is output from the generated perl script.";
$test->write('script', <<EOF);
print STDOUT "$line\\n";
EOF
$output = `$^X script`;
ok($output eq "$line\n");
This completely avoids having to make the "script" file itself executable. (Since you're writing your test in Perl, it's safe to assume that Perl itself is executable.)
If you must generate a directly-executable script, then use the $Config{'startperl'} variable at the start of the script to generate the appropriate magic that will execute it as a Perl script:
use Config;
$line = "This is output from the generated perl script.";
$test->write('script', <<EOF);
$Config{'startperl'};
print STDOUT "$line\\n";
EOF
chdir($test->workdir);
chmod(0755, 'script'); # POSIX-SPECIFIC
$output = `script`;
ok($output eq "$line\n");
Addtional hints on writing portable tests are welcome.
perl(1), Algorithm::DiffOld, File::Find, File::Spec, Test, Test::Cmd::Common, Test::Harness, Test::More, Test::Simple, Test::Unit.
Alternative command-testing modules include:
Test::Exit, Test::Output, or using Capture::Tiny with one of the above test modules, for example Test::More.
A rudimentary page for the "Test::Cmd" module is available at:
http://www.baldmt.com/Test-Cmd/
The most involved example of using the "Test::Cmd" package to test a real-world application is the "cons-test" testing suite for the Cons software construction utility. The suite uses a sub-class of Test::Cmd::Common (which in turn is a sub-class of "Test::Cmd") to provide common, application-specific infrastructure across a large number of end-to-end application tests. The suite, and other information about Cons, is available at:
http://www.dsmit.com/cons
<https://github.com/neilb/Test-Cmd>
Steven Knight, knight@baldmt.com
This module is now being maintained by Neil Bowers <neilb@cpan.org>.
Copyright 1999-2001 Steven Knight. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Thanks to Greg Spencer for the inspiration to create this package and the initial draft of its implementation as a specific testing package for the Cons software construction utility. Information about Cons is available at:
http://www.dsmit.com/cons/
The general idea of managing temporary working directories in this way, as well as the test reporting of the "pass", "fail" and "no_result" methods, come from the testing framework invented by Peter Miller for his Aegis project change supervisor. Aegis is an excellent bit of work which integrates creation and execution of regression tests into the software development process. Information about Aegis is available at:
http://www.tip.net.au/~millerp/aegis.html
Thanks to Michael Schwern for all of the thoughtful work he's put into Perl's standard testing methodology, including the Test::Simple and Test::More modules, and enhancement and maintenance of the Test and Test::Harness modules. Thanks also to Christian Lemburg for the impressively complete Test::Unit framework of modules. Ideas from both have helped keep "Test::Cmd" flexible enough to be useful in multiple testing frameworks.
| 2022-10-13 | perl v5.34.0 |