DOCSEARCHbyAlgolia
Submit Docs

tmt (Test Management Tool)

repository·main·Indexed 41 days ago

https://github.com/teemtee/tmt

A system for managing test lifecycles, including creation, execution across environments, and result review. It utilizes a specialized metadata format (fmf) to store test data within version control systems like Git and features a hierarchical class structure (Run, Plan, Story, Test) for managing execution.

Tokens
40.3K
Snippets
159
Records
202
Agent score
99%

What's inside tmt

  1. Overview of tmt - Test Management Tool

    main

    The tmt tool is a test management system designed to simplify creating, running, and reviewing tests across different environments. It is available as both a Python module and a command-line tool.

    Key features include:

    • Metadata Specification: Allows storing test execution data directly within a git repository, facilitating easy sharing of test coverage across projects and distributions.
    • Flexible Metadata Format (fmf): A human and machine-readable format used to store data close to the source code. It uses inheritance and elasticity to organize metadata efficiently and prevent duplication.
    • CI Integration: Enables tests in CI environments using consistent and concise configurations.
    • Remote Referencing: Supports referencing remote repositories to share test data.

    For a quick start, you can use the tldr tmt command.

  2. Available provision plugins for environment setup

    main

    TMT supports several provision plugins to set up testing environments. Depending on your needs, you can use different methods such as virtual machines, containers, or local environments.

    Supported provision plugins include:

    • virtual.testcloud
    • container
    • connect
    • local
    • artemis

    During the provision step, you can also specify detailed hardware requirements as defined in the TMT hardware specification.

  3. What is a tmt recipe and how is it generated

    main

    A recipe is a YAML file that captures a complete, static snapshot of a tmt run after all dynamic evaluation has been resolved. It contains preprocessed information about plans, tests, and run configuration, along with a link to the original results.yaml file.

    tmt automatically generates a recipe at the end of every run and stores it as recipe.yaml in the run workdir (e.g., /var/tmp/tmt/run-001/recipe.yaml). Recipes allow you to reproduce a run with the exact same configuration without needing to re-select plans or tests via the command line.

  4. Configure arbitrary bash scripts via TMT_PLAN_SOURCE_SCRIPT

    main
    If you need to execute arbitrary bash commands or logic that cannot be expressed as simple NAME=VALUE pairs, use TMT_PLAN_SOURCE_SCRIPT. This file is sourced in all tests and prepare/shell phases. You must populate its content during a prepare step.

  5. Organize multiple tests using L1 and L2 metadata

    main

    For complex test suites, use a hierarchical structure:

    1. L2 Metadata (Plan file): A central plan file that uses discover to find tests and execute to run them.
    2. L1 Metadata (Test files): Individual .fmf files for each test. This allows each test to define its own environment variables and require specific packages without affecting other tests in the same plan.

    This pattern is highly recommended when managing a large number of tests in a single repository.

    # plans/example.fmf (L2 Metadata)
discover:
    how: fmf
execute:
    how: tmt

# tests/smoke27.fmf (L1 Metadata)
test: ./venv.sh
environment:
    VERSION: 2.7
    METHOD: virtualenv
require:
  - python27
  - python2-virtualenv
  - python2-devel

  6. Understand tmt machine roles: guest vs runner

    main

    In tmt, there is a distinction between the machine executing the tmt command and the machine where the actual tests are executed:

    • runner: The machine that is running the tmt command.
    • guest: The machine where the tests are being run.

  7. How the object hierarchy works in tmt

    main
    The execution hierarchy follows the path: Run -> Plans -> Steps -> Plugins. The Run object sits at the top of this hierarchy. Most objects in this tree maintain a parent attribute that points to the instance containing them.

  8. Use inheritance in FMF files

    main

    The FMF format allows you to reduce duplication by using inheritance. This is useful for grouping common metadata or sharing test logic.

    Common Patterns:

    • Global Metadata: Place a main.fmf file at the root of a directory tree (e.g., /tests/main.fmf) to define common attributes like framework: beakerlib or require: [tmt] for all sub-tests.
    • Virtual Tests: Define a base test script and create sub-tests that inherit from it while providing different environment variables or parameters.
    • Plan Inheritance: Use the + sign when defining sub-plans to extend parent data instead of replacing it. This allows you to inherit base discover or prepare steps and add specific filters or requirements.
    # Example: Virtual Tests (sharing a script with different environments)
test: ./test.sh
require: curl

/fast:
    summary: Quick smoke test
    environment:
        MODE: fast

/full:
    summary: Full test set
    environment:
        MODE: full

# Example: Inheriting and extending a plan
/basic:
    summary: Quick set of basic functionality tests
    discover+:
        filter: tier:1

  9. Distinguish between Commands and ShellScripts

    main

    In tmt, there is a strict distinction between executable commands and free-form shell scripts. This distinction ensures proper handling of user-provided scripts (like prepare or finish scripts) and test commands.

    • tmt.utils.Command: Represents an executable followed by its arguments as a list of "command elements". This is the only type accepted by the tmt.utils.Common.run() method and should never be used with shell=True.
    • tmt.utils.ShellScript: A free-form string containing a shell script (single commands or multiline scripts).

    Best Practices for Plugin Writers:

    • Avoid using bare str or List[str] when annotating inputs that are intended to be scripts. Use tmt.utils.ShellScript instead.
    • Avoid using shell=True. Instead, use ShellScript.to_shell_command() to convert a script into a command that can be executed via a shell.
    • Use Command and ShellScript types when passing commands between functions to leverage their built-in conversion helpers.
    from tmt.utils import Command, ShellScript

# Building a Command
command = Command('ls')
command += Command('-al')
command += ['/']
# str(command) -> 'ls -al /'

# Building a ShellScript
script = ShellScript('ls -al')
script += ShellScript('ls -al $HOME')
# str(script) -> 'ls -al; ls -al $HOME'

  10. Understand the Test Tree and Discover Plugin behavior

    main

    The Test Tree is a tests directory prepared by a discover plugin, containing the test scripts and files available for execution. The structure of the Test Tree depends on which discover plugin is used:

    1. fmf discover plugin: Copies the User Tree to the Test Tree (unless a url is provided, in which case it clones the git repository into it).
    2. shell discover plugin: Creates a symlink to the Work Tree inside the test directory structure.
    # Shell discover plugin structure example
├── discover
│   ├── default-0
│   │   └── tests -> ../../tree
│   ├── step.yaml
│   └── tests.yaml

  11. Use AGENTS.md for Code Assistants

    main

    The project uses AGENTS.md to provide instructions and prompts to AI code assistants.

    • Primary Entrypoint: agents/AGENTS.md is the main file. It is symlinked to AGENTS.md in the repository root so assistants can discover it automatically.
    • Claude Code: Since Claude Code does not natively support AGENTS.md, the instructions are symlinked to .claude/CLAUDE.md, which Claude Code loads automatically.
    • Custom Instructions: You can create personal instruction files (e.g., agents/my-custom-prompts.md) and reference them alongside the project instructions in your prompts.

  12. Hard reboot support in provision plugins

    main

    Hard reboot functionality is not universally supported by all provision plugins. If a plugin lacks this capability, the following features will not work:

    • The restart-with-reboot test key (from /spec/tests/restart)
    • The reboot action of the watchdog test check

    Plugins that fully implement hard reboot include:

    • connect (only when the hard-reboot key is explicitly defined)
    • beaker
    • container
    • virtual.testcloud
    • artemis