DOCSEARCHbyAlgolia
Submit Docs

docs.rs Documentation

repository·main·Indexed 22 days ago

https://github.com/rust-lang/docs.rs

An open source project that automatically builds and hosts documentation for Rust crates released on crates.io using the nightly Rust compiler. This repository contains the source code for the docs.rs service, including the web server (docs_rs_web), the build system (docs_rs_builder), and administration tools (docs_rs_admin).

Tokens
60.5K
Snippets
187
Records
290
Agent score
78%

What's inside docs.rs

  2. Overview of Docs.rs

    main

    Docs.rs is an open source service that hosts documentation for Rust crates. It automatically builds documentation for every crate released on crates.io using the nightly release of the Rust compiler.

    Note: This repository is for developing the docs.rs service itself. For information on how to use the hosted documentation, visit the docs.rs about page.

  4. How docs.rs archive storage and indexing works

    main

    docs.rs uses a specialized storage model to serve individual files (like HTML, CSS, or source files) from large crate archives without downloading the entire archive.

    Storage Model

    For every crate version, two ZIP archives are stored:

    1. rustdoc output archive: Contains the generated documentation.
    2. source files archive: Contains the crate's source code.

    The Index Model

    To enable fast random access, docs.rs generates an SQLite database index for each archive. This index acts like a ZIP central directory and maps a logical file path to its specific location within the ZIP file. Each entry in the index contains:

    • The filename/path within the archive.
    • The byte range from (inclusive start).
    • The byte range to (inclusive end).
    • The compression algorithm used for that specific entry.

    Request Flow

    When a single file is requested:

    1. The system identifies the correct archive and index for the crate/version.
    2. The SQLite index is checked (and downloaded to a local cache if missing).
    3. The index is queried for the requested path.
    4. An HTTP Range request is issued to S3 to fetch only the bytes between from and to.
    5. The returned byte range is decompressed using the algorithm specified in the index and served to the user.

  6. Understand compression layers in archive storage

    main

    There are two distinct compression layers to distinguish when working with docs.rs archives:

    1. ZIP entry compression: This is the compression applied to individual files inside the rustdoc or source archives. The SQLite index tracks this per-file. Currently, the archive index creation supports bzip2 (Bz2) ZIP entries to ensure wide compatibility for offline downloads.
    2. Object storage compression: This refers to the compression applied to the entire blob during upload to S3, represented by the blob's Content-Encoding or storage metadata.

    When serving files via the archive/index system, the ZIP entry compression is the layer used for decompressing the byte ranges fetched from S3.

  8. Set up a local development environment for docs.rs

    main

    The recommended development workflow uses cargo run for the main binary and docker-compose for external services (DB and S3).

    Prerequisites

    Ensure you have docker and docker-compose installed. You will need approximately 10GB of disk space for the initial data download.

    Native Dependencies

    You must install the following C dependencies:

    • gcc, g++
    • pkg-config
    • git
    • make
    • cmake
    • zlib
    • openssl (e.g., sudo apt install libssl-dev on Ubuntu)

    Quickstart Steps

    1. Clone and initialize submodules:
      git clone https://github.com/rust-lang/docs.rs.git docs.rs
cd docs.rs
git submodule update --init
    2. Configure environment and directories:
      cp .env.sample .env
mkdir -p ignored/cratesfyi-prefix/crates.io-index
    3. Build the binary (using offline SQLX mode):
      SQLX_OFFLINE=1 cargo build
    4. Start external services:
      docker compose up --wait db s3
    5. Apply environment variables and database migrations:
      . ./.env
# Allow S3 downloads
mcli policy set download docsrs/rust-docs-rs
# Run migrations
cargo run --bin docs_rs_admin -- database migrate
    6. Initialize the build environment:
      cargo run --bin docs_rs_builder -- build update-toolchain
    git clone https://github.com/rust-lang/docs.rs.git docs.rs
cd docs.rs
git submodule update --init
cp .env.sample .env
mkdir -p ignored/cratesfyi-prefix/crates.io-index
SQLX_OFFLINE=1 cargo build
docker compose up --wait db s3
. ./.env
mcli policy set download docsrs/rust-docs-rs
cargo run --bin docs_rs_admin -- database migrate
cargo run --bin docs_rs_builder -- build update-toolchain

  9. Run tests (Unit and GUI)

    main

    Unit Tests

    Run standard Rust tests using:

    cargo test

    GUI Tests

    GUI tests use the browser-ui-test framework. You can run them via just or manually using node.

    Manual Setup:

    1. Install the tester: npm install browser-ui-test
    2. Run the tester: node gui-tests/tester.js
    # Standard tests
cargo test

# GUI tests via just
just run-gui-tests

# Manual GUI tests
node gui-tests/tester.js

  10. Guidelines for submitting Pull Requests

    main

    When submitting pull requests to this repository, follow these requirements:

    1. Naming: Start the PR title with the package name in square brackets, e.g., [XML].
    2. Scope: Keep changes as small and self-contained as possible.
    3. Syntax Testing: If changing a .sublime-syntax file, you must include a new or enhanced syntax test.
    4. Performance: For significant changes, include multiple syntax tests and a set of performance measurements comparing the old version vs. the new version.

    Note on New Packages: Pull requests for entirely new packages are currently not accepted to avoid naming conflicts with Package Control. Complex plugins (linters, autocomplete) should be maintained as standalone packages rather than included here.

  11. Run the docs.rs web server

    main

    To start the web interface (typically on http://localhost:3000), use the docs_rs_web binary.

    Note: The web server does not automatically run database migrations; you must run cargo run --bin docs_rs_admin -- database migrate manually before starting the server.

    For development with automatic restarts on code or template changes, use cargo-watch.

    # Standard start
cargo run --bin docs_rs_web

# With auto-reload (requires cargo-watch)
cargo watch -x "run --bin docs_rs_web"

# Explicit webserver command
cargo run --bin docs_rs_webserver start-web-server