DOCSEARCHbyAlgolia
Submit Docs

nbclient

repository·main·Indexed 41 days ago

https://github.com/jupyter/nbclient

A client library for the programmatic execution of Jupyter Notebooks. It provides the NotebookClient class and the jupyter execute CLI to run notebooks across various contexts, allowing for custom execution hooks, kernel overrides, and error handling. Formerly part of nbconvert's ExecutePreprocessor, it supports Python 3.6+ and integrates with nbformat for loading and saving executed notebooks.

Tokens
8.1K
Snippets
31
Records
41
Agent score
51%

What's inside nbclient

  1. What is NBClient

    main
    NBClient is a client library designed for the programmatic execution of Jupyter Notebooks. It allows you to run notebooks in various execution contexts, including via the command line. It was originally part of nbconvert's ExecutePreprocessor and was extracted into this standalone library to facilitate easier updates and integration for downstream applications.

  2. What is nbclient and how does it relate to JupyterClient?

    main

    Overview

    nbclient is a client library designed for the programmatic execution of Jupyter Notebooks. It allows you to run notebooks across various execution contexts, including the command line.

    Relationship to JupyterClient

    It is important to distinguish nbclient from jupyter_client:

    • jupyter_client: A low-level client library for the Jupyter protocol. It provides the Python API for starting, managing, and communicating with Jupyter kernels.
    • nbclient: A higher-level tool that uses execution contexts to allow notebooks to be run. It was originally part of the nbconvert project (as ExecutePreprocessor) and was extracted to facilitate easier updates and integration for downstream applications.

  3. Configure NotebookClient with traitlets

    main

    The NotebookClient accepts configuration options via traitlets. Key configuration options include:

    • timeout: The maximum time (in seconds) each notebook cell is allowed to run before raising an exception. Default is 30. Set to None or -1 to disable the timeout.
    • kernel_name: Specifies the name of the kernel to use. If not provided, it defaults to the kernel specified in the notebook's metadata. This is useful for overriding kernels (e.g., testing a notebook on both python2 and python3).
    • allow_errors: A boolean (default False). If set to True, the client will continue executing the notebook even if a cell raises an error, collecting all errors in the output notebook.

  4. Use execution hooks for notebooks and cells

    main

    You can inject custom logic into the execution pipeline using hooks. These hooks are functions that are called at specific lifecycle stages.

    Notebook-level hooks

    These receive a single argument: notebook=NotebookNode.

    • on_notebook_start: Called when the client is initialized, before execution.
    • on_notebook_complete: Called after execution and kernel cleanup.
    • on_notebook_error: Called if an exception occurs before kernel cleanup.

    Cell-level hooks

    These receive at least two arguments: cell=NotebookNode and cell_index=int.

    • on_cell_start: Called for all cell types before execution.
    • on_cell_execute: Called right before a code cell is executed.
    • on_cell_complete: Called after successful execution (no errors).
    • on_cell_executed: Called right after a code cell is executed. Also receives execute_reply=dict.
    • on_cell_error: Called if a cell fails. Also receives execute_reply=dict.

  6. Perform a manual release of nbclient

    main

    If not using jupyter_releaser, follow these steps to manually release the package.

    Prerequisites

    1. Ensure CHANGELOG.md is updated for the upcoming version.
    2. Install the necessary packaging tools:
      pip install tbump build tomlkit==0.7.0

    1. Bump the version

    Set the target version as an environment variable and use tbump to update the versioning files without pushing to remote repositories:

    export version=<NEW_VERSION>
tbump ${version} --no-push

    2. Push to GitHub

    Push the changes and the new version tags to the upstream repository:

    git push upstream && git push upstream --tags

    3. Push to PyPI

    Clean previous build artifacts, build the new distribution, and upload using twine:

    rm -rf dist/*
rm -rf build/*
python -m build .
twine upload dist/*
    # Bump version
export version=<NEW_VERSION>
tbump ${version} --no-push

# Push to GitHub
git push upstream && git push upstream --tags

# Push to PyPI
rm -rf dist/*
rm -rf build/*
python -m build .
twine upload dist/*

  7. Execute notebooks using the Python API

    main

    You can programmatically execute .ipynb files using the NotebookClient class. This process involves loading a notebook with nbformat, configuring the client, calling execute(), and then saving the resulting notebook.

    Workflow:

    1. Load: Use nbformat.read() to load the notebook.
    2. Configure: Initialize NotebookClient with your desired settings (e.g., timeout, kernel_name).
    3. Execute: Call client.execute() to run the cells.
    4. Save: Use nbformat.write() to save the notebook with its new outputs.
    import nbformat
from nbclient import NotebookClient

# Load
nb = nbformat.read('notebook.ipynb', as_version=4)

# Configure
client = NotebookClient(nb, timeout=600, kernel_name='python3')

# Execute
client.execute()

# Save
nbformat.write(nb, 'executed_notebook.ipynb')

  9. Handle execution errors and exceptions

    main

    By default, nbclient stops execution at the first error and raises a CellExecutionError. The resulting notebook will contain outputs up to the failing cell, including the stack trace.

    To ensure the notebook is saved even when an error occurs, use a try...except...finally block:

    from nbclient.exceptions import CellExecutionError
import nbformat

try:
    client.execute()
except CellExecutionError:
    print(f'Error executing the notebook. See output for traceback.')
    raise
finally:
    nbformat.write(nb, 'output.ipynb')

    To execute the entire notebook regardless of errors, set allow_errors=True in the NotebookClient configuration.

    from nbclient.exceptions import CellExecutionError

try:
    client.execute()
except CellExecutionError:
    msg = 'Error executing the notebook.\n\n' 
    msg += 'See notebook for the traceback.'
    print(msg)
    raise
finally:
    nbformat.write(nb, 'executed_notebook.ipynb')

  11. Understand CellControlSignal and execution flow

    main

    In nbclient, certain exceptions are used as control signals rather than just error indicators. These inherit from CellControlSignal:

    • CellExecutionComplete: Raised when all execution requests are finished and no further messages are expected from the kernel over ZeroMQ channels.
    • CellExecutionError: Used to propagate execution failures.
    • CellTimeoutError: Used to signal that a cell has timed out.

    Developers can catch CellControlSignal to manage the lifecycle and flow of notebook execution processes.

  12. Execute notebooks via the nbclient CLI

    main

    The nbclient CLI (provided by the NbClientApp class) allows you to execute Jupyter Notebook (*.ipynb) files from the command line. You can specify one or more notebooks to run, control execution timeouts, choose specific kernels, and decide whether to overwrite the original files or save results to a new base name.

    Common usage patterns include:

    • Running a single notebook and overwriting it with results using --inplace.
    • Running multiple notebooks and saving results using a pattern like --output={notebook_name}_executed.
    • Continuing execution even if cells encounter errors using --allow-errors.
    # Example: Run a notebook and overwrite it with the results
# jupyter-execute my_notebook.ipynb --inplace

# Example: Run multiple notebooks and save them with a suffix
# jupyter-execute notebook1.ipynb notebook2.ipynb --output={notebook_name}_out