.. _ctest:

-------------------
The ctest Framework
-------------------

The ``ctest`` infrastructure in Enzo-e is comprised of unit tests (which test individual functions and functionality) and integration tests which test a more holistic portion of the codebase. In this documentation, we detail the layout of the ctest infrastructure. All input files are located in the input directory and the tests themselves are defined in ``test/CMakeLists.txt``

How to Run Tests
================

To run all the tests currently in the testing infrastructure run ``ctest`` in the build directory.

Basic parameters to control ``ctest`` are:

* ``ctest -N`` to see all available tests
* ``ctest -V`` to run all tests with verbose output
* ``ctest -R someregex`` to run all tests matching ``someregex``, e.g., ``ctest -R heat``
* ``ctest -L somelabel`` to run all test with label matching ``somelabel``. For ``Enzo-E`` we currently only set ``serial`` and ``parallel`` for serial and parallel (i.e., using multiple PEs) tests, respectively

*Note*, you may need to adjust the parallel launch configuration to your environment for the parallel test.
By default, ``charmrun`` with 4 PEs will be used to launch parallel tests.
To use ``srun`` instead (e.g., for a pure Charm++ MPI build) with 8 ranks you have to set
``-DPARALLEL_LAUNCHER=srun -DPARALLEL_LAUNCHER_NPROC_ARG="-n" -DPARALLEL_LAUNCHER_NPROC=8``
when configuring your build.

In order to run a test separately from the main ``ctest`` infrastructure write ``bin/enzo-e`` followed be the location of the test to be run in the main enzo directory. For example, ``bin/enzo-e input/Cosmology/method_cosmology-1.in`` runs the method_cosmology-1 test. Some tests are intended to run on multiple processors, for these tests the number of processors must be specified like so ``charmrun +p4 bin/enzo-e input/Cosmology/method_cosmology-8.in``, this runs the method_cosmology-8 test on four processors, a test designed to be run on multiple processors in parallel. If charmrun command is not in your path charmrun's path must also be included:  ``~/Charm/bin/charmrun +p4 bin/enzo-e input/Cosmology/method_cosmology-8.in``.

In order to exclude the ``"shu_collapse"`` and ``"bb_test"`` tests (as is done when running
the tests on CircleCI, execute the following command: ``ctest -E "(shu_collapse)|(bb_test)"``.


How to Analyse the Test Results
===============================

By default all tests will be run and the output is stored in the `test` directory of the build.

``ctest`` will automatically tell you which tests passed and which tests failed. For a more verbose output of the test, you can call ``ctest`` with ``--output-on-failure``.

If an integration test fails in start-up this implies that there is something wrong with how the test is set up either as an error in the ``test/CMakeLists.txt`` file that calls the test or that the test has an undefined parameter. If a test fails while running this indicates that the feature being tested does not work.

In order to see what happened during the test, you can look at the output directory of the test, which is located in a subdirectory (named after the test) in the ``test`` directory of the build directory, for example ``method_cosmology-1.in`` the test results are stored in ``test/MethodCosmology/Cosmology-8``. This directory also contains all outputs (image and data files).