Test::Cmd - Perl module for portable testing of commands and scripts

  use Test::Cmd;

  $test = Test::Cmd->new(prog => 'program_or_script_to_test',
                        interpreter => 'script_interpreter',
                        string => 'identifier_string',
                        workdir => '',
                        subdir => 'dir',
                        match_sub => $code_ref,
                        verbose => 1);

  $test->verbose(1);

  $test->prog('program_or_script_to_test');

  $test->basename(@suffixlist);

  $test->interpreter('script_interpreter');

  $test->string('identifier string');

  $test->workdir('prefix');

  $test->workpath('subdir', 'file');

  $test->subdir('subdir', ...);
  $test->subdir(['sub', 'dir'], ...);

  $test->write('file', <<'EOF');
  contents of file
  EOF
  $test->write(['subdir', 'file'], <<'EOF');
  contents of file
  EOF

  $test->read(\$contents, 'file');
  $test->read(\@lines, 'file');
  $test->read(\$contents, ['subdir', 'file']);
  $test->read(\@lines, ['subdir', 'file']);

  $test->writable('dir');
  $test->writable('dir', $rwflag);
  $test->writable('dir', $rwflag, \%errors);

  $test->preserve(condition, ...);

  $test->cleanup(condition);

  $test->run(prog => 'program_or_script_to_test',
                interpreter => 'script_interpreter',
                chdir => 'dir', args => 'arguments', stdin => <<'EOF');
  input to program
  EOF

  $test->pass(condition);
  $test->pass(condition, \&func);

  $test->fail(condition);
  $test->fail(condition, \&func);
  $test->fail(condition, \&func, $caller);

  $test->no_result(condition);
  $test->no_result(condition, \&func);
  $test->no_result(condition, \&func, $caller);

  $test->stdout;
  $test->stdout($run_number);

  $test->stderr;
  $test->stderr($run_number);

  $test->match(\@lines, \@matches);
  $test->match($lines, $matches);

  $test->match_exact(\@lines, \@matches);
  $test->match_exact($lines, $matches);

  $test->match_regex(\@lines, \@regexes);
  $test->match_regex($lines, $regexes);

  $test->diff_exact(\@lines, \@matches, \@output);
  $test->diff_exact($lines, $matches, \@output);

  $test->diff_regex(\@lines, \@regexes, \@output);
  $test->diff_regex($lines, $regexes, \@output);

  sub func {
        my ($self, $lines, $matches) = @_;
        # code to match $lines and $matches
  }
  $test->match_sub(\&func);
  $test->match_sub(sub { code to match $_[1] and $_[2] });

  $test->here;

The CWTest::Cmd module provides a low-level framework for portable automated testing of executable commands and scripts (in any language, not just Perl), especially commands and scripts that interact with the file system.

The CWTest::Cmd module makes no assumptions about what constitutes a successful or failed test. Attempting to read a file that doesn't exist, for example, may or may not be an error, depending on the software being tested.

Consequently, no CWTest::Cmd methods (including the CWnew() method) exit, die or throw any other sorts of exceptions (but they all do return useful error indications). Exceptions or other error status should be handled by a higher layer: a subclass of CWTest::Cmd, or another testing framework such as the CWTest or CWTest::Simple Perl modules, or by the test itself.

(That said, see the CWTest::Cmd::Common module if you want a similar module that provides exception handling, either to use directly in your own tests, or as an example of how to use CWTest::Cmd.)

In addition to running tests and evaluating conditions, the CWTest::Cmd module manages and cleans up one or more temporary workspace directories, and provides methods for creating files and directories in those workspace directories from in-line data (that is, here-documents), allowing tests to be completely self-contained. When used in conjunction with another testing framework, the CWTest::Cmd module can function as a fixture (common startup code for multiple tests) for simple management of command execution and temporary workspaces.

The CWTest::Cmd module inherits CWFile::Spec methods (CWfile_name_is_absolute(), CWcatfile(), etc.) to support writing tests portably across a variety of operating and file systems.

A CWTest::Cmd environment object is created via the usual invocation:

    $test = Test::Cmd->new();

Arguments to the CWTest::Cmd::new method are keyword-value pairs that may be used to initialize the object, typically by invoking the same-named method as the keyword.

As mentioned, because the CWTest::Cmd module makes no assumptions about what constitutes success or failure of a test, it can be used to provide temporary workspaces, other file system interaction, or command execution for a variety of testing frameworks. This section describes how to use the CWTest::Cmd with several different higher-layer testing frameworks.

Note that you should not intermix multiple testing frameworks in a single testing script. The CWTest::Cmd module may be used in tests that print results in a format suitable for the standard Perl CWTest::Harness module:

    use Test::Cmd;

    print "1..5\n";

    $test = Test::Cmd->new(prog => 'test_program', workdir => '');
    if ($test) { print "ok 1\n"; } else { print "not ok 1\n"; }

    $input = <<_EOF;
    test_program should process this input
    and exit successfully (status 0).
    _EOF_

    $wrote_file = $test->write('input_file', $input);
    if ($wrote_file) { print "ok 2\n"; } else { print "not ok 2\n"; }

    $test->run(args => '-x input_file');
    if ($? == 0) { print "ok 3\n"; } else { print "not ok 3\n"; }

    $wrote_file = $test->write('input_file', $input);
    if ($wrote_file) { print "ok 4\n"; } else { print "not ok 4\n"; }

    $test->run(args => '-y input_file');
    if ($? == 0) { print "ok 5\n"; } else { print "not ok 5\n"; }

Several other Perl modules simplify the use of CWTest::Harness by eliminating the need to hand-code the CWprint statements and test numbers. The CWTest module, the CWTest::Simple module, and the CWTest::More module all export an CWok() subroutine to test conditions. Here is how the above example would look rewritten to use CWTest::Simple:

    use Test::Simple tests => 5;
    use Test::Cmd;

    $test = Test::Cmd->new(prog => 'test_program', workdir => '');
    ok($test, "creating Test::Cmd object");

    $input = <<_EOF;
    test_program should process this input
    and exit successfully (status 0).
    _EOF_

    $wrote_file = $test->write('input_file', $input);
    ok($wrote_file, "writing input_file");

    $test->run(args => '-x input_file');
    ok($? == 0, "executing test_program -x input_file");

    $wrote_file = $test->write('input_file', $input);
    ok($wrote_file, "writing input_file");

    $test->run(args => '-y input_file');
    ok($? == 0, "executing test_program -y input_file");

    use Test::Unit;
    use Test::Cmd;

    my $test;

    $input = <<'EOF';
    test_program should process this input
    and exit successfully (status 0).
    EOF

    sub set_up {
        $test = Test::Cmd->new(prog => 'test_program', workdir => '');
        $test->write('input_file', $input);
    }

    sub test_x {
        my $result = $test->run(args => '-x input_file');
        assert($result == 0, "failed test_x\n");
    }

    sub test_y {
        my $result = $test->run(args => '-y input_file');
        assert($result == 0, "failed test_y\n");
    }

    create_suite();
    run_suite;

Note that, because the CWTest::Cmd module takes care of cleaning up temporary workspaces on exit, there is no need to remove explicitly the workspace in a CWtear_down subroutine. (There may, of course, be other things in the test that need a CWtear_down subroutine.)

Alternatively, the CWTest::Cmd module provides CWpass(), CWfail(), and CWno_result() methods that can be used to provide an appropriate exit status and simple printed indication for a test. These methods terminate the test immediately, reporting CWPASSED, CWFAILED, or CWNO RESULT respectively, and exiting with status 0 (success), 1 or 2 respectively.

The separate CWfail() and CWno_result() methods allow for a distinction between an actual failed test and a test that could not be properly evaluated because of an external condition (such as a full file system or incorrect permissions).

The exit status values happen to match the requirements of the Aegis change management system, and the printed strings are based on existing Aegis conventions. They are not really Aegis-specific, however, and provide a simple, useful starting point if you don't already have another testing framework:

    use Test::Cmd;

    $test = Test::Cmd->new(prog => 'test_program', workdir => '');
    Test::Cmd->no_result(! $test);

    $input = <<EOF;
    test_program should process this input
    and exit successfully (status 0).
    EOF

    $wrote_file = $test->write('input_file', $input);
    $test->no_result(! $wrote_file);

    $test->run(args => '-x input_file');
    $test->fail($? != 0);

    $wrote_file = $test->write('input_file', $input);
    $test->no_result(! $wrote_file);

    $test->run(args => '-y input_file');
    $test->fail($? != 0);

    $test->pass;

Note that the separate CWTest::Cmd::Common wrapper module can simplify the above example even further by taking care of common exception handling cases within the testing object itself.

    use Test::Cmd::Common;

    $test = Test::Cmd::Common->new(prog => 'test_program', workdir => '');

    $input = <<EOF;
    test_program should process this input
    and exit successfully (status 0).
    EOF

    $wrote_file = $test->write('input_file', $input);

    $test->run(args => '-x input_file');

    $wrote_file = $test->write('input_file', $input);

    $test->run(args => '-y input_file');

    $test->pass;

See the CWTest::Cmd::Common module for details.

Methods supported by the CWTest::Cmd module include: Create a new CWTest::Cmd environment. Arguments with which to initialize the environment are passed in as keyword-value pairs. Fails if a specified temporary working directory or subdirectory cannot be created. Does NOT die or exit on failure, but returns FALSE if the test environment object cannot be created. Sets the verbose level for the environment object to the specified value. Specifies the executable program or script to be tested. Returns the absolute path name of the current program or script. Returns the basename of the current program or script. Any specified arguments are a list of file suffixes that may be stripped from the basename. Specifies the program to be used to interpret CWprog as a script. Returns the current value of CWinterpreter. Specifies an identifier string for the functionality being tested to be printed on failure or no result. When an argument is specified, creates a temporary working directory with the specified name. If the argument is a NULL string (''), the directory is named CWtestcmd by default, followed by the unique ID of the executing process. Returns the absolute pathname to the temporary working directory, or FALSE if the directory could not be created. Returns the absolute path name to a subdirectory or file under the current temporary working directory by concatenating the temporary working directory name with the specified arguments. Creates new subdirectories under the temporary working dir, one for each argument. An argument may be an array reference, in which case the array elements are concatenated together using the CWFile::Spec-&catfile> method. Subdirectories multiple levels deep must be created via a separate argument for each level:

    $test->subdir('sub', ['sub', 'dir'], [qw(sub dir ectory)]);

        $test->run(args => 'arg1 arg2');

        $test->run(chdir => 'xyzzy');

        $test->run(chdir => '.');               # Unix-specific

        $test->run(chdir => $test->curdir);     # portable

        $test->run(stdin => <<_EOF_);
        input to the program under test
        _EOF_

        if (! $test->diff_exact($test->stdout,
                                \@expected_lines,
                                \@diff)) {
                print @diff;
        }

        if (! $test->diff_regex($test->stdout,
                                \@expected_lines,
                                \@diff)) {
                print @diff;
        }

        $test->match_sub(\&Test::Cmd::match_exact);

        $test->match_sub(\&Test::Cmd::match_regex);

        $test->match_sub(\&Test::Cmd::diff_exact);

        $test->match_sub(\&Test::Cmd::diff_regex);

        use Test::Cmd qw(match_exact match_regex diff_exact diff_regex);
        $test_exact = Test::Cmd->new(match_sub => \&match_exact);
        $test_regex = Test::Cmd->new(match_sub => \&match_regex);
        $test_exact = Test::Cmd->new(match_sub => \&diff_exact);
        $test_regex = Test::Cmd->new(match_sub => \&diff_regex);

Several environment variables affect the default values in a newly created CWTest::Cmd environment object. These environment variables must be set when the module is loaded, not when the object is created. If set to a true value, all temporary working directories will be preserved on exit, regardless of success or failure of the test. The full path names of all temporary working directories will be reported on error output. If set to a true value, all temporary working directories will be preserved on exit from a failed test. The full path names of all temporary working directories will be reported on error output. If set to a true value, all temporary working directories will be preserved on exit from a test for which there is no result. The full path names of all temporary working directories will be reported on error output. If set to a true value, all temporary working directories will be preserved on exit from a successful test. The full path names of all temporary working directories will be reported on error output. When set to a true value, enables verbose reporting of various internal things (path names, exact command line being executed, etc.).

Although the CWTest::Cmd module is intended to make it easier to write portable tests for portable utilities that interact with file systems, it is still very easy to write non-portable tests if you're not careful.

The best and most comprehensive set of portability guidelines is the standard Writing portable Perl document at:

        http://www.perl.com/pub/doc/manual/html/pod/perlport.html

To reiterate one important point from the WpP document: Not all Perl programs have to be portable. If the program or script you're testing is UNIX-specific, you can (and should) use the CWTest::Cmd module to write UNIX-specific tests.

That having been said, here are some hints that may help keep your tests portable, if that's a requirement. The normal Perl way to fetch the current working directory is to use the CWCwd::cwd method. Unfortunately, the CWCwd::cwd method canonicalizes the path name it returns, changing the native directory separators into the forward slashes favored by Perl and UNIX. For most Perl scripts, this makes a great deal of sense and keeps code uncluttered. Passing in a file name that has had its directory separators altered, however, may confuse the command or script under test, or make it difficult to compare output from the command or script with an expected result. The CWTest::Cmd::here method returns the absolute path name of the current working directory, like CWCwd::cwd, but does not manipulate the returned path in any way. The CWFile::Spec module provides a system-independent interface for manipulating path names. Because the CWTest::Cmd class is a sub-class of the CWFile::Spec class, you can use these methods directly as follows:

        if (! Test::Cmd->file_name_is_absolute($prog)) {
                my $prog = Test::Cmd->catfile(Test::Cmd->here, $prog);
        }

        $foo_exe = "foo$Config{_exe}";
        ok(-f $foo_exe);

Avoid generating executable programs or scripts.

How to make a file or script executable varies widely from system to system, some systems using file name extensions to indicate executability, others using a file permission bit. The differences are complicated to accomodate in a portable test script. The easiest way to deal with this complexity is to avoid it if you can. If your test somehow requires executing a script that you generate from the test itself, the best way is to generate the script in Perl and then explicitly feed it to the Perl executable on the local system. To be maximally portable, use the CW$^X variable instead of hard-coding perl into the string you execute:

        $line = "This is output from the generated perl script.";
        $test->write('script', <<EOF);
        print STDOUT "$line\\n";
        EOF
        $output = `$^X script`;
        ok($output eq "$line\n");

This completely avoids having to make the CWscript file itself executable. (Since you're writing your test in Perl, it's safe to assume that Perl itself is executable.) If you must generate a directly-executable script, then use the CW$Config{'startperl'} variable at the start of the script to generate the appropriate magic that will execute it as a Perl script:

        use Config;
        $line = "This is output from the generated perl script.";
        $test->write('script', <<EOF);
        $Config{'startperl'};
        print STDOUT "$line\\n";
        EOF
        chdir($test->workdir);
        chmod(0755, 'script');  # POSIX-SPECIFIC
        $output = `script`;
        ok($output eq "$line\n");

Addtional hints on writing portable tests are welcome.

perl(1), Algorithm::DiffOld(3), File::Find(3), File::Spec(3), Test(3), Test::Cmd::Common(3), Test::Harness(3), Test::More(3), Test::Simple(3), Test::Unit(3).

A rudimentary page for the CWTest::Cmd module is available at:

        http://www.baldmt.com/Test-Cmd/

The most involved example of using the CWTest::Cmd package to test a real-world application is the CWcons-test testing suite for the Cons software construction utility. The suite uses a sub-class of CWTest::Cmd::Common (which in turn is a sub-class of CWTest::Cmd) to provide common, application-specific infrastructure across a large number of end-to-end application tests. The suite, and other information about Cons, is available at:

        http://www.dsmit.com/cons

Steven Knight, knight@baldmt.com

Copyright 1999-2001 Steven Knight. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

Thanks to Greg Spencer for the inspiration to create this package and the initial draft of its implementation as a specific testing package for the Cons software construction utility. Information about Cons is available at:

        http://www.dsmit.com/cons/

The general idea of managing temporary working directories in this way, as well as the test reporting of the CWpass, CWfail and CWno_result methods, come from the testing framework invented by Peter Miller for his Aegis project change supervisor. Aegis is an excellent bit of work which integrates creation and execution of regression tests into the software development process. Information about Aegis is available at:

        http://www.tip.net.au/~millerp/aegis.html

Thanks to Michael Schwern for all of the thoughtful work he's put into Perl's standard testing methodology, including the CWTest::Simple and CWTest::More modules, and enhancement and maintenance of the CWTest and CWTest::Harness modules. Thanks also to Christian Lemburg for the impressively complete CWTest::Unit framework of modules. Ideas from both have helped keep CWTest::Cmd flexible enough to be useful in multiple testing frameworks.