DOCSEARCHbyAlgolia
Submit Docs

D2L-Book Toolkit

repository·master·Indexed 42 days ago

https://github.com/d2l-ai/d2l-book

A specialized toolkit for creating and publishing interactive, code-heavy documentation, such as books or Python package tutorials. It features the d2lbook CLI to manage a build pipeline that transforms markdown and Jupyter notebooks into hosted document sites, HTML, and PDF outputs. Key capabilities include notebook evaluation with GPU acceleration, multi-tab code blocks, and integration with Sphinx and Pandoc.

Tokens
6.2K
Snippets
28
Records
33
Agent score
53%

What's inside D2L-Book

  1. Overview of D2L-Book

    master

    D2L-Book is a toolkit designed for building and publishing interactive documentation. It is specifically optimized for two use cases:

    1. Building and publishing a full book that contains embedded Python code.
    2. Creating documentation for Python packages that includes integrated tutorials.

    The output of the toolkit is a hosted document site, such as the one found at https://book.d2l.ai.

  2. User Guide Overview

    master

    The D2L-Book user guide provides instructions for managing hands-on book projects. Key workflows include:

    • Project Creation: How to initialize a new book project.
    • Building and Deployment: Processes for generating document builds and deploying them to hosting services.
    • Configuration: Managing various project settings and options.
    • Content Authoring: Guidelines for writing Markdown and integrating code examples, including specialized components like code_tabs, slides, and colab integration.

  3. Understand the D2L-Book build pipeline

    master

    The D2L-Book project uses a multi-step pipeline to transform source markdown files into published artifacts (HTML, PDF, and ZIP files).

    Source Format: Source files are .md files. These are either standard markdown or Jupyter notebooks saved in markdown format with outputs removed.

    Pipeline Steps:

    1. Evaluation: .md files are converted to .ipynb files and executed to ensure they are runnable. To optimize this step, the pipeline uses:
      • Parallel execution via multiple processes.
      • GPU acceleration.
      • Caching (reusing existing .ipynb files if the source .md hasn't changed).
    2. Artifact Generation: The evaluated .ipynb files are uploaded to GitHub and zipped for user download.
    3. Sphinx Conversion: .ipynb files are converted to .rst (reStructuredText) files, including preprocessing for images, tables, and citations.
    4. Documentation Building: Sphinx is used to generate the final .html and .pdf documentation.
    5. Publishing: Final artifacts are published to online storage (e.g., AWS S3).

  4. Manage build cache and notebook evaluation

    master

    To keep source files clean and avoid issues with non-deterministic code outputs, it is recommended to evaluate notebooks and save results rather than keeping outputs in the source .ipynb files. d2lbook uses an evaluation cache to optimize build times.

    How the Cache Works

    • Incremental Builds: d2lbook only re-evaluates notebooks that have been modified. Unchanged notebooks are reused from the previous build.
    • Triggering a Full Rebuild: To force a complete rebuild of all notebooks, you can:
      1. Remove the saved notebooks in _build/eval.
      2. Delete the _build directory.
      3. Modify a file listed in the dependencies configuration.

    Configuring Dependencies

    You can specify files in a config.ini file that, when modified, will invalidate the entire cache and trigger a build from scratch. For example, adding config.ini to the dependencies list ensures that any change to the config file forces a full rebuild:

    [build]
dependencies = config.ini

  5. Deploy to GitHub Pages

    master

    You can deploy your d2lbook build results to GitHub Pages.

    Prerequisites

    1. Create a GitHub repository.
    2. Enable GitHub Pages by navigating to Settings -> GitHub Pages and selecting the master branch as the source.
    3. Ensure your machine's SSH key is imported to GitHub to allow automated deployment without manual password entry.

    Configuration

    In your config.ini file, specify the github_repo under the [deploy] section using the format username/repo-name.

    Workflow

    1. Build your project (HTML and/or PDF): d2lbook build html pdf
    2. Deploy the results: d2lbook deploy html pdf
    [deploy]
github_repo = d2l-ai/d2l-book-deploy-demo
    # Build the project
d2lbook build html pdf

# Deploy the project
d2lbook deploy html pdf

  6. Install the d2lbook CLI

    master

    Install the d2lbook command-line interface using pip. While a package exists on PyPI, it is recommended to install directly from the GitHub repository to ensure you have the latest features, as the project is under active development. The package is tested on macOS and Linux.

    pip install git+https://github.com/d2l-ai/d2l-book

  7. Build HTML and PDF outputs

    master

    Use the d2lbook build command to generate different formats from your project directory.

    • HTML: Run d2lbook build html. The output is located at _build/html/index.html within your project folder.
    • PDF: Run d2lbook build pdf. The output filename is determined by the name key in the [project] section of config.ini.
    # Build HTML
cd mybook && d2lbook build html

# Build PDF
cd mybook && d2lbook build pdf

  8. Insert tables with captions

    master

    To insert a table with a caption, start the caption line with a colon (:). You must leave an empty line between the caption and the table itself.

    : The number is computed by $z_{ij} = \sum_{k}x_{ik}y_{kj}.

| Year | Number | Comment |
| ---  | --- | --- |
| 2018 | 100 | Good year |
| 2019 | 200 | Even better |

  9. Best practices for editing source files

    master

    To ensure easier code review and version control, it is recommended to save all source files (including Jupyter notebooks) as markdown files.

    If you are converting a notebook to markdown:

    1. Clear the cell outputs before saving.
    2. Note that Jupyter code cells are represented in markdown using a code block with the {.python .input} tag. For example:
    ```{.python .input}
print('this is a Jupyter code cell')

  10. Deploy to AWS (S3, CloudFront, and Route 53)

    master

    For more flexible hosting, you can deploy to AWS. The recommended architecture is to use S3 for storage, CloudFront as a CDN to reduce latency, and Route 53 for domain management.

    AWS Setup Steps

    1. S3: Create a bucket (ideally named after your domain). Disable 'Block all public access' via ACLs and enable 'Static website hosting'.
    2. CloudFront: Create a distribution.
      • Origin Domain Name: Use your S3 website URL (without http://).
      • Default Root Object: Set to index.html.
      • CNAMEs: Add your target domain name.
      • SSL: Use an SSL certificate from AWS Certificate Manager.
    3. Route 53: Create a 'Hosted Zone' and a 'Type-A Record Set' that uses your CloudFront domain name as the alias target.

    Configuration

    In your config.ini file, specify the S3 bucket URI under the [deploy] section.

    Deployment

    Run d2lbook deploy to synchronize all results to the specified S3 bucket. Note that due to CDN caching, changes may take a few minutes to appear on your custom domain, though they will appear immediately on the S3 website URL.

    [deploy]
s3_bucket = s3://book.d2l.ai
    d2lbook deploy

  12. Hide code source or outputs in Jupyter cells

    master

    You can control the visibility of code cells in the rendered book using specific magic comments at the top of the cell:

    • To hide the source code: Add # Hide code
    • To hide the cell outputs: Add # Hide outputs
    # Hide code
1+2+3
    # Hide outputs
1+2+3