CTest

CTest
Name	CTest
Developer	Kitware
Initial release	1999
Programming language	C++
Operating system	Cross-platform
License	BSD-style

CTest is a test-driver program distributed with CMake that automates the execution and reporting of software tests. It is commonly used with CTestTestfile.cmake files and integrates with testing ecosystems such as CDash, Jenkins (software), Travis CI, GitHub Actions, and GitLab CI/CD to provide cross-platform test orchestration. CTest is maintained by Kitware and is widely adopted in projects including VTK, ParaView, ITK, Meshlab, and other scientific and engineering software stacks.

Overview

CTest acts as a bridge between build systems and test reporting infrastructures, invoking test executables, collecting exit codes, timing information, and producing XML or dashboard submissions consumable by tools like CDash and NUnit. It supports multiple test types including regular tests, memory checks via Valgrind, coverage runs with LCOV, sanitizer-assisted runs via AddressSanitizer, and performance tests similar to those used in Google Benchmark workflows. CTest interacts with generators and build tools such as Make (software), Ninja (build system), Visual Studio, and Xcode to discover test targets produced by CMake configuration.

Features and Functionality

CTest provides features for grouping tests, filtering by labels, rerunning failures, and parallel execution with configurable job counts. Labeling integrates with project conventions used by VTK, ITK, Eigen (software), and OpenCV to categorize tests as integration test, regression test, or performance test (note: these are proper nouns when used as names of specific test suites). Output formats include terse console summaries, XML compliant with CTest Test schema for CDash dashboards, and JUnit-style XML consumable by Jenkins (software) and TeamCity. Advanced functionality includes test fixtures, environment isolation using Docker, resource throttling for shared HPC clusters managed by systems like Slurm Workload Manager and PBS (software), and support for cross-compilation workflows employed by projects targeting Android (operating system), iOS, and embedded platforms from ARM Holdings.

Usage and Configuration

CTest is configured via CMakeLists.txt and auxiliary files such as CTestConfig.cmake and CTestTestfile.cmake. Users define tests with the add_test command in CMake specifying executable targets from projects like Boost (C++) Libraries or Google Test. Labels and properties are set to integrate with dashboard submission configured for CDash instances, which may be hosted by organizations such as Kitware or institutions like Lawrence Livermore National Laboratory. Invocation modes include ctest --verbose for detailed output, ctest -R for regex filtering, ctest --repeat-until-fail to exercise flakey scenarios, and ctest --submit for submitting results to CDash. Integration with continuous integration services often leverages preconfigured scripts in repositories hosted on GitHub, GitLab, or Bitbucket.

Integration with CMake and Continuous Integration

CTest is bundled with CMake and is referenced directly from CMake packaging workflows developed by teams at Kitware, Kitware, Inc. contributors, and open-source communities maintaining Paraview and VTK ecosystems. It pairs with CPack for packaging validation and with CTest/Test dashboards to furnish historical test trends. CI pipelines in Jenkins (software), Travis CI, CircleCI, Azure DevOps, and GitHub Actions run ctest as a primary step, often followed by code coverage collection via Codecov or Coveralls. For regulated or high-assurance projects managed by organizations like NASA or European Space Agency, CTest workflows are integrated into traceability matrices and release pipelines that use artifact repositories such as Artifactory.

Examples and Best Practices

Common practices include tagging long-running tests to exclude them from quick regressions, using fixtures and scripted setup/teardown commands to prepare test environments, and running memory checks with Valgrind or sanitizers from the LLVM Project for leak detection. Example patterns used in ITK and VTK repositories include separate nightly builds that invoke ctest --repeat-until-fail for flakiness detection and ctest -j to parallelize execution on multi-core Jenkins (software) agents. Projects targeting embedded targets often build with cross-toolchains from GNU Toolchain and execute tests in emulators like QEMU (software), submitting results to CDash for centralized analysis. Recommended practices also include version pinning of CMake to ensure consistent behavior across developer workstations and CI agents.

History and Development

CTest originated as part of the CMake ecosystem at Kitware in the late 1990s to provide a unified test runner for cross-platform projects such as VTK and ParaView. Over time, CTest gained integrations for dashboard submission via CDash and features supporting memory checking and sanitizer workflows influenced by tools from the GNU Project and the LLVM Project. The project has evolved alongside CMake releases, with community contributions from developers affiliated with institutions including Kitware, Sandia National Laboratories, Lawrence Berkeley National Laboratory, and various open-source projects like OpenCV and Boost (C++) Libraries.

Category:Software testing tools