DOCSEARCHbyAlgolia
Submit Docs

attic

repository·main·Indexed 24 days ago

https://github.com/zhaofengli/attic

A self-hostable Nix Binary Cache server backed by S3-compatible storage. It features multi-tenancy, global deduplication, and managed signing. The project includes a client CLI (attic), a daemon CLI (atticd), an administrative CLI (atticadm), and high-level Rust bindings to libnixstore for interacting with Nix stores using async/await semantics.

Tokens
21.1K
Snippets
29
Records
130
Agent score
84%

What's inside attic

  1. Overview of Attic

    main
    Attic is a self-hostable Nix Binary Cache server that uses S3-compatible storage. It is designed to provide a scalable, multi-tenant Nix cache with features like global deduplication and managed signing. It is suitable for both single-machine setups and serverless deployments (e.g., fly.io).

  2. What is Attic

    main
    Attic is a self-hostable Nix Binary Cache server designed to work with S3-compatible storage providers. It provides a way to host Nix store paths with features like global deduplication and garbage collection. It is designed for scalability, supporting both single-machine setups and deployment to serverless platforms like fly.io.

  3. Access Attic CLI, daemon, and admin CLI references

    main

    The Attic project provides three primary command-line interfaces for interaction. Detailed documentation for each is available in their respective reference files:

    • attic CLI: The main client interface for interacting with the Attic service.
    • atticd CLI: The command-line interface for managing the Attic daemon.
    • atticadm CLI: The administrative CLI for performing administrative tasks.

  4. Understand the NAR chunking and compression pipeline

    main

    Uploaded NARs follow a specific pipeline on the server: they are chunked, then compressed, and finally streamed to the storage backend (e.g., S3).

    Crucially, global deduplication at the chunk level is performed using the hash of the uncompressed chunk.

    Data Flow Diagram:

                            ┌───────────────────────────────────►Chunk Hash
                        │
                        ├───────────────────────────────────►Chunk Size
                        │
                ┌───────┴────┐  ┌──────────┐  ┌───────────┐
 Chunk Stream──►│Chunk Hasher├─►│Compressor├─►│File Hasher├─►File Stream─►S3
                └────────────┘  └──────────┘  └─────┬─────┘
                                                    │
                                                    ├───────►File Hash
                                                    │
                                                    └───────►File Size

  6. How chunking works in Attic

    main

    Attic uses the FastCDC algorithm to split uploaded NARs (Nix Archive files) into smaller chunks. This process enables data deduplication by identifying and storing unique segments of data across different archives.

    Chunking behavior is controlled by four primary parameters that determine when a file is chunked and the size characteristics of the resulting chunks. Note that changing these parameters will change the 'cutpoints' (the boundaries where chunks are split), which means existing chunks cannot be easily reused for new NARs, temporarily reducing the deduplication ratio.

  7. Core features of Attic

    main

    Attic provides several key capabilities for managing Nix binary caches:

    • Multi-Tenancy: Supports creating isolated private caches for different users or groups. Tenants are mutually untrusting and cannot access or pollute each other's views.
    • Global Deduplication: Uses a content-addressed NAR Store and Chunk Store. Individual tenant caches act as restricted views of this global store; uploading a path creates a mapping to the global NAR.
    • Managed Signing: The server performs signing on-the-fly when store paths are fetched. This ensures that users pushing paths do not need access to the private signing keys.
    • Scalability: Designed for easy replication and deployment to serverless platforms.
    • Garbage Collection: Supports LRU (Least Recently Used) garbage collection to remove unused store paths.

  8. How deduplication and chunking work in Attic

    main

    Attic performs global deduplication at two levels: NAR files and chunks.

    1. NAR-level deduplication: If an identical NAR exists in the Global NAR Store, chunking is skipped and the NAR is directly deduplicated.
    2. Chunk-level deduplication: If the NAR is new, it is split into chunks using the FastCDC algorithm. Identical chunks are stored only once in the storage backend.

    Key behaviors:

    • Chunking threshold: Data chunking is optional and can be disabled for NARs smaller than a specific threshold. When disabled, the NAR is uploaded as a single chunk, but NAR-level deduplication still applies.
    • Reassembly: During a download, atticd reassembles the entire NAR from its constituent chunks by streaming them from the storage backend.
    • Chunking Strategy: Attic chunks the entire uncompressed NAR file rather than individual constituent files. This allows for larger average chunk sizes and more efficient handling of NARs containing many small files (e.g., VSCode or Zoom).

  10. Quickstart: Spin up Attic in 15 minutes

    main

    To quickly set up a local development environment, use nix shell to get the Attic binaries and run atticd in monolithic mode. This configuration uses a SQLite database and local storage.

    1. Install Attic via Nix:
    nix shell github:zhaofengli/attic
    1. Start the server:
    atticd

    When you run atticd, it will output a configuration path (usually ~/.config/attic/server.toml) and a root login command containing a JWT token. Copy this token to log in via the attic client.

    nix shell github:zhaofengli/attic
atticd

  11. Log in to an Attic server

    main

    To authenticate the attic client with a server, use the attic login command provided by your administrator. The command follows this format:

    attic login <server-name> <url> <token>

    The client supports multiple servers simultaneously. You can reference a specific cache using either its name (if the server is set as the default) or the server:cache syntax.

    To set a specific server as the default, update the default-server key in ~/.config/attic/config.toml.

    attic login central https://attic.domain.tld/ eyJ...

  12. Configure Access Control with atticadm tokens

    main

    Attic uses stateless JWT tokens for authentication. You can use atticadm to generate tokens with specific permissions for users.

    Restricted Access Token

    To create a token that only allows a user to pull and push to a specific cache:

    atticadm make-token --sub <username> --validity '<duration>' --pull <cache_name> --push <cache_name>

    Pattern-based Access Token

    To allow a user to create and manage their own caches using a prefix (e.g., alice-*), use the --create-cache flag:

    atticadm make-token --sub <username> --validity '<duration>' --pull '<prefix>*' --push '<prefix>*' --create-cache '<prefix>*'

    Use the --dump-claims flag to inspect the JWT claims in plain text without encoding the token.