DOCSEARCHbyAlgolia
Submit Docs

Bombadil Documentation

repository·main·Indexed 20 days ago

https://github.com/antithesishq/bombadil

Bombadil is a property-based testing (PBT) framework for web and terminal user interfaces. It autonomously explores UI state spaces by executing random and systematic sequences of actions to identify bugs, edge cases, and unexpected timings. It supports TypeScript specifications for defining properties and action generators, and can be integrated into local developer environments, CI pipelines via GitHub Actions, and the Antithesis platform.

Tokens
36K
Snippets
68
Records
218
Agent score
79%

What's inside Bombadil

  1. What is Bombadil?

    main

    Bombadil is a property-based testing (PBT) framework designed for user interfaces, including web applications and terminal applications. Unlike example-based testing (like Playwright or Cypress) which relies on fixed test cases, Bombadil autonomously explores the state space of your application by executing random and systematic sequences of actions to find edge cases, unexpected timings, and strange inputs that manual tests might miss.

    Bombadil can be run in:

    • Local developer environments
    • CI (Continuous Integration)
    • Inside the Antithesis platform

  2. Overview of Bombadil

    main
    Bombadil is a property-based testing tool designed for web and terminal UIs. It autonomously explores and validates correctness properties to identify bugs. It is designed to run in local developer environments, CI pipelines, and within the Antithesis platform.

  3. Define properties using invariants and temporal operators

    main

    A property in Bombadil describes how a system should behave in general. To define a property, you must export it from your specification module. The most common type is an invariant, which is a condition that must always be true.

    Invariants are expressed using the always temporal operator. Because properties must be evaluated against a sequence of states, the argument passed to always must be a thunk (a function that takes no arguments) so it can be re-evaluated in every state.

    To access the current state of the system (like the DOM or terminal grid) within a property, you must use Extractors.

    export const hasTitle = always(() => 
    title.current !== ""
);

  4. Structure a Bombadil specification module

    main

    A specification is a standard ES module. While you can split your specification into multiple files and modules, the top-level module provided to the Bombadil CLI must only export properties and action generators.

    If using TypeScript, install the types from @antithesishq/bombadil to ensure type safety.

    // Properties and action generators are exposed as named exports
export const myProperty = ...; 

export const myAction = ...;

  5. How Bombadil works

    main

    Bombadil operates in a continuous loop to explore your system's state space. Instead of writing specific test steps, you define general properties that must always hold true. The execution loop follows these steps:

    1. State Extraction: Bombadil extracts the current state from the browser (DOM) or terminal (output bytes).
    2. Property Checking: It checks all defined properties against the current state. You can configure it to report all violations or exit on the first one found.
    3. Action Selection: It selects the next action to perform based on the current state.
    4. Event Waiting: It waits for an event to occur, such as:
      • Browser: Page navigation, DOM mutation, or a timeout.
      • Terminal: A chunk of output bytes or a timeout.
    5. Repeat: The loop returns to step 1.

    You provide the properties and actions; Bombadil manages the exploration and event handling.

  6. Compose complex conditions with Formulas and temporal operators

    main

    Formulas represent "conditions over time". Temporal operators take subformulas and return new formulas that evaluate how those conditions behave over a sequence of states.

    Temporal Operators

    • always(x): x holds in this and every future state.
    • next(x): x holds in the next state.
    • eventually(x): x holds in this or any future state.
    • now(thunk): Converts a thunk into a formula representing its value in the current state. This is useful for single-state preconditions.

    Logical Connectives

    Formulas support logical operations via methods:

    • x.and(y): Both x and y hold.
    • x.or(y): Either x or y holds.
    • x.implies(y): If x holds, then y must hold (equivalent to not(x) or y).
    • x.not() or not(x): Negation of the formula.
    // Example: A button press implies a spinner appears now and eventually disappears
const buttonPressed = extract(() => ...);
const spinnerVisible = extract(() => ...);

now(() => buttonPressed.current).implies(
    now(() => spinnerVisible.current)
        .and(eventually(() => !spinnerVisible.current))
)

  7. Configure Bombadil in GitHub Actions

    main

    Use the antithesishq/bombadil-action@v2 action to run tests in your CI pipeline. You must specify a driver (either browser or terminal).

    # Browser driver example
- uses: antithesishq/bombadil-action@v2
  with:
    driver: browser
    origin: https://your-app.example.com
    specification: ./bombadil/specification.ts
    time-limit: 5m
    exit-on-violation: true
    output-path: bombadil-output

# Terminal driver example
- uses: antithesishq/bombadil-action@v2
  with:
    driver: terminal
    command: your-cli --arg value
    specification: ./bombadil/specification.ts
    time-limit: 5m
    exit-on-violation: true
    output-path: bombadil-output

  9. Reproduce a test violation

    main

    To reproduce a bug found by Bombadil, use the --reproduce option pointing to the output directory of the original test run. This attempts to perform the exact same sequence of actions to reach the same state.

    Note: Reproductions are not guaranteed to succeed. For best results, use the same options as the original test. Bombadil prints the exact reproduction command after a test run finishes.

  10. Build the Bombadil documentation

    main

    The Bombadil manual can be built into various formats using the provided Makefile. You can build all formats at once or target specific formats like HTML, EPUB, PDF, or plain text.

    # Build all formats (HTML, EPUB, PDF, TXT)
make all

# Build specific formats
make html     # Chunked HTML site
make epub     # EPUB e-book
make pdf      # PDF document
make txt      # Plain text (for LLMs)

# Clean build artifacts
make clean

# Serve HTML locally (requires Python)
make serve

  11. Install Bombadil via npm

    main

    Install Bombadil as a development dependency in your project. This method automatically provides TypeScript type definitions for writing specifications.

    To use it in your project, add a script to your package.json for either browser or terminal testing.

    npm install --save-dev @antithesishq/bombadil

    Example package.json scripts:

    {
  "scripts": {
    "test": "bombadil browser test --time-limit=1m https://your-app.example.com"
  }
}
    {
  "scripts": {
    "test": "bombadil terminal test --time-limit=1m your-cli --arg value"
  }
}

  12. Testing terminal applications with Bombadil

    main

    Bombadil can drive any program that reads from stdin and writes to a terminal, such as traditional CLIs, interactive REPLs, or full TUIs. To test a terminal application, you write a TypeScript specification that exports:

    1. Properties: General rules defining how the system should behave.
    2. Action Generators: Logic for generating user interactions.

    These can be custom domain-specific implementations or imported from Bombadil's defaults.