DOCSEARCHbyAlgolia
Submit Docs

nbmake

repository·main·Indexed 42 days ago

https://github.com/treebeardtech/nbmake

A pytest plugin for testing and releasing Jupyter notebooks. It automates notebook execution to ensure documentation runs without errors, supporting parallel testing via pytest-xdist, custom cell timeouts, kernel overrides, and programmatic execution via the NotebookRun class. Features include the ability to mock variables, run post-cell execution logic, and specifically identify ModuleNotFoundErrors.

Tokens
2.9K
Snippets
16
Records
17
Agent score
53%

What's inside nbmake

  1. Find missing imports in failing notebooks

    main

    If you cannot get all notebooks to run perfectly, you can use nbmake specifically to check for ModuleNotFoundErrors. Use the --nbmake-find-import-errors flag to ignore all other errors, and set a low --nbmake-timeout to skip long-running cells.

    pytest \
  --nbmake \
  --nbmake-find-import-errors \
  --nbmake-timeout=20

  2. Install and Quick Start with nbmake

    main

    To test Jupyter notebooks automatically using pytest, install pytest and nbmake, then run pytest targeting your .ipynb files. This executes the notebooks using nbclient and allows for parallel testing.

    pip install pytest nbmake
pytest --nbmake **/*ipynb

  3. Parallelise Notebook Testing

    main

    For large repositories with many notebooks, use pytest-xdist to run notebooks in parallel. Use the -n=auto flag to automatically determine the number of workers based on your CPU cores.

    pip install pytest-xdist

pytest --nbmake -n=auto

  4. Allow Errors for an Entire Notebook

    main

    To prevent a notebook from failing the test suite due to errors, add an execution key to the notebook's top-level metadata. You can also specify a notebook-wide timeout here.

    {
  "cells": [ ... ],
  "metadata": {
    "kernelspec": { ... },
    "execution": {
      "allow_errors": true,
      "timeout": 300
    }
  }
}

  5. Ignore a Code Cell

    main

    To prevent nbmake from executing a specific cell, add the skip-execution tag to the cell's metadata within the custom.metadata structure.

    {
  "language": "python",
  "custom": {
    "metadata": {
      "tags": [
        "skip-execution"
      ]
    }
  }
}

  6. Mock variables to simplify testing

    main

    You can overwrite global variables after a cell executes by using the nbmake.mock key in the cell's metadata. This is useful for bypassing long-running processes (like training) by injecting pre-defined values.

    {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
    "nbmake": {
      "mock": {
        "epochs": 2,
        "config": "/test/config.json",
        "args": {
          "env": "test"
        }
      }
    }
  },
  "source": [
    "epochs = 10\n",
    "..."
  ]
}

  7. How nbmake integrates with pytest

    main
    nbmake provides a pytest plugin that automatically discovers and tests Jupyter Notebook files. When running pytest, the plugin identifies notebook files and treats each one as a NotebookItem. During the test execution, nbmake runs the notebook using the configured kernel and timeout settings. If a notebook execution fails (e.g., due to a cell error or an import error), it raises a NotebookFailedException, which is then caught and formatted by the plugin to provide a clear error summary and traceback in the pytest output.

  9. Run test logic after a cell executes

    main

    To run arbitrary scripts or assertions after a cell finishes (e.g., to fetch CI secrets or verify state), use the nbmake.post_cell_execute key in the cell's metadata. The provided strings will be executed in the notebook context.

    {
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
     "nbmake": {
      "post_cell_execute": [
        "y = 3",
        "z = x+y"
      ]
     }
   },
   "source": [
     "x = 1\n",
     "y = 2\n",
     "z = 0\n"
   ]
  }
 ]
}

  10. Build Jupyter Books Faster

    main

    To speed up the building of large Jupyter Book repositories, combine pytest-xdist parallelization with the --overwrite flag to write back executed notebooks, then run the build command.

    pytest --nbmake --overwrite -n=auto examples
jb build examples

  11. Configure notebook execution via metadata

    main

    You can control how nbmake treats specific notebooks by adding metadata to the .ipynb file. This is useful for overriding global settings for specific edge cases.

    Execution Settings

    Add an execution key to the notebook metadata to control timeouts and error tolerance:

    {
  "metadata": {
    "execution": {
      "timeout": 60,
      "allow_errors": true
    }
  }
}

    Cell-level Mocking and Post-Execution

    You can use the nbmake metadata key within individual cells to inject state or run extra commands.

    Mocking variables:

    {
  "metadata": {
    "nbmake": {
      "mock": {
        "API_KEY": "fake_key",
        "DEBUG_MODE": true
      }
    }
  }
}

    Note: String values are wrapped in quotes, while other types are passed as literals.

    Post-cell execution:

    {
  "metadata": {
    "nbmake": {
      "post_cell_execute": [
        "print('Cell finished!')",
        "import os"
      ]
    }
  }
}
    {
  "metadata": {
    "execution": {
      "timeout": 60,
      "allow_errors": true
    },
    "nbmake": {
      "mock": {
        "VAR": "val"
      },
      "post_cell_execute": ["print('done')"]
    }
  }
}