DOCSEARCHbyAlgolia
Submit Docs

manubot

repository·main·Indexed 19 days ago

https://github.com/manubot/manubot

A workflow and set of Python tools for scholarly publishing. It provides utilities to automate manuscript processing via `manubot process`, retrieve and format bibliographic metadata with `manubot cite`, and perform AI-assisted revisions using `manubot ai-revision`. The package includes the `pandoc-manubot-cite` filter for Pandoc workflows and a `manubot webpage` command for deploying HTML and PDF outputs to a website.

Tokens
2.4K
Snippets
8
Records
12
Agent score
17%

What's inside manubot

  1. Overview of manubot CLI commands

    main

    The manubot package provides several command-line interfaces for scholarly publishing workflows:

    • manubot cite: Retrieves and formats bibliographic metadata for persistent identifiers such as DOIs or PubMed IDs.
    • manubot process: Prepares scholarly manuscripts for Pandoc consumption. This command is used by Manubot manuscripts (based on the Rootstock template) to automate manuscript generation.
    • manubot ai-revision: Automatically revises a manuscript based on a set of AI-generated suggestions.

  2. Use manual references in Manubot

    main

    Instead of generating all bibliographic metadata automatically, you can provide your own reference metadata. manubot process searches the --content-directory for files matching the glob manual-references*.*.

    These files are then added to the Pandoc metadata bibliography field, allowing them to be processed by the pandoc-manubot-cite filter.

  3. Install the manubot Python package

    main

    You can install manubot via pip. If you are using it as part of a manuscript repository, installation is typically handled automatically via Rootstock's environment specification. For standalone use, use the following commands:

    To install the latest release from PyPI:

    pip install --upgrade manubot

    To install from the GitHub source using a specific commit hash:

    COMMIT=d2160151e52750895571079a6e257beb6e0b1278
pip install --upgrade git+https://github.com/manubot/manubot@$COMMIT
    pip install --upgrade manubot

  4. Generate GIF and SVG from asciinema recordings

    main

    If you have an asciinema recording (a .cast file or an online recording), you can convert it into a GIF or an SVG using asciicast2gif and svg-term-cli respectively.

    To create a GIF, use asciicast2gif with the -s flag to specify playback speed. To create an SVG, use svg-term with the --window and --cast flags.

    # Convert an asciinema cast to a GIF
# -s 2.0 sets the playback speed to 2x
asciicast2gif -s 2.0 https://asciinema.org/a/205085.cast manubot-cite-cast.gif

# Convert an asciinema cast to an SVG
# --window adds a terminal window frame
# --cast specifies the recording ID
svg-term --window --cast=205085 --out=manubot-cite-cast.svg

  5. Configure Pandoc and panflute dependencies

    main

    Several manubot functions require Pandoc to be installed on your system.

    Additionally, the pandoc-manubot-cite filter requires panflute. You must install a version of panflute that is compatible with your installed Pandoc version.

    Example: If you have Pandoc 2.9 installed, you should install a compatible panflute version like this:

    pip install panflute==1.12.5
    pip install panflute==1.12.5

  6. Run Manubot development commands

    main

    Common commands for testing, linting, and processing manuscripts during development. These assume you are in the repository root and the conda environment is active.

    Testing and Linting

    • pytest: Runs the test suite.
    • pre-commit install: Installs git hooks to run checks on commits.
    • pre-commit run --all-files: Runs all pre-commit checks manually (required for CI).
    • git commit --no-verify: Commits changes while skipping pre-commit checks (not recommended as it will fail CI).

    Documentation and Utilities

    • python manubot/tests/test_readme.py: Regenerates README codeblocks for --help messages.
    • portray as_html --overwrite --output_dir=docs: Generates documentation as HTML.

    Manuscript Processing

    Use the manubot process command to process a manuscript example.

    # run the test suite
pytest

# install pre-commit git hooks
pre-commit install

# run the pre-commit checks
pre-commit run --all-files

# regenerate the README codeblocks
python manubot/tests/test_readme.py

# generate the docs
portray as_html --overwrite --output_dir=docs

# process the example testing manuscript
manubot process \
  --content-directory=manubot/process/tests/manuscripts/example/content \
  --output-directory=manubot/process/tests/manuscripts/example/output \
  --skip-citations \
  --log-level=INFO

  7. Run the primary Manubot processing workflow

    main

    The manubot process command is the main entry point for transforming manuscript source files into outputs suitable for Pandoc. It performs bibliographic processing and templating.

    Required Arguments:

    • --content-directory: The path to the directory containing your manuscript source files.
    • --output-directory: The path where Manubot will save the generated files.

    Note on Citations: Citation and reference processing has been moved from manubot process to the pandoc-manubot-cite filter. Consequently, the --skip-citations flag is now required when running manubot process.

    manubot process \
  --skip-citations \
  --content-directory=content \
  --output-directory=output

  8. Set up a Manubot development environment

    main

    To develop on Manubot, create a dedicated Conda environment with the required Python and Pandoc versions, then install the package in editable mode with webpage and dev extras.

    Note: The pip install command uses .[webpage,dev] to include necessary dependencies for webpage generation and development tools.

    conda create --name manubot-dev --channel conda-forge \
  python=3.11 pandoc=2.11.3.1
conda activate manubot-dev
pip install --editable ".[webpage,dev]"

  9. Use the `pandoc-manubot-cite` filter

    main

    The pandoc-manubot-cite filter allows you to use Manubot's cite-by-ID functionality directly within a Pandoc workflow. It reads and writes a JSON-encoded abstract syntax tree (AST) for Pandoc.

    Usage: Run the filter as part of a Pandoc command using the --filter flag:

    pandoc input.md --filter=pandoc-manubot-cite -o output.pdf

    Manual Reference Loading:

    • Manual references are loaded from the references and bibliography Pandoc metadata fields.
    • Files ending in .json or .yaml are treated as CSL Data.
    • Other extensions are converted to CSL JSON using pandoc-citeproc --bib2json.
    • If no prefix (like doi:) is provided, a raw: prefix is automatically added.

  10. Revise manuscript content with AI

    main

    The manubot ai-revision command uses large language models (via OpenAI) to suggest text improvements and revise manuscript content.

    Usage:

    manubot ai-revision --content-directory content/

    Configuration and Debugging:

    • --model-type: The model class to use. Defaults to GPT3CompletionModel. Can be any subclass of manubot_ai_editor.models.ManuscriptRevisionModel.
    • --model-kwargs: Keyword arguments for the revision model (format key=value). Useful for debugging paragraph detection.
    • --config-directory: Directory for custom AI revision configuration files.
    manubot ai-revision \
  --content-directory content/ \
  --model-type DummyManuscriptRevisionModel \
  --model-kwargs add_paragraph_marks=true

  11. Generate bibliographic metadata with `manubot cite`

    main

    The manubot cite utility generates bibliographic metadata for specific citation keys. It can output raw CSL JSON items or rendered, formatted references using Pandoc.

    Citation Key Format: Keys should follow the prefix:accession format (e.g., doi:10.1098/rsif.2017.0387).

    Common Options:

    • --format: Specifies the output format. Options include csljson, cslyaml, plain, markdown, docx, html, and jats. If not specified, it attempts to infer from the --output extension, defaulting to csljson.
    • --md: Shortcut for --format=markdown.
    • --yml: Shortcut for --format=cslyaml.
    • --txt: Shortcut for --format=plain.
    • --output: Specifies the output file path (defaults to stdout).
    • --csl: Path or URL to a CSL XML file for styling (defaults to Manubot's style).
    manubot cite --format=markdown \
  doi:10.1098/rsif.2017.0387 pubmed:29424689 pmc:PMC5640425 arxiv:1806.05726

  12. Deploy Manubot outputs to a webpage

    main

    The manubot webpage command populates a webpage directory tree with Manubot output files (HTML and PDF) for website hosting. This command should be run from the root of a Manubot manuscript following the Rootstock layout.

    Key Options:

    • --version: Creates a webpage/v/{version} directory. Defaults to the commit hash on CI or 'local' elsewhere.
    • --checkout: Specifies a branch to checkout for /v directory contents (e.g., --checkout=gh-pages).
    • --timestamp: Uses OpenTimestamps to timestamp versioned manuscripts in webpage/v.