DOCSEARCHbyAlgolia
Submit Docs

Lore Documentation

repository·main·Indexed 27 days ago

https://github.com/epicgames/lore

Lore is an open-source, high-scalability version control system optimized for projects combining code with large binary assets, such as games and entertainment. This documentation covers AWS deployment using Terraform and Graviton instances, server container image builds, and comprehensive development standards including code conventions, architectural decision records (ADRs), and documentation style guides.

Tokens
95.1K
Snippets
119
Records
611
Agent score
91%

What's inside Lore

  1. Overview of Lore Code Standards

    main
    Lore maintains specific coding conventions for error handling, logging, task spawning, testing, and documentation. These standards are organized by language and area to ensure consistency across the repository. Each standard document provides imperative rules, the rationale behind them, code examples, and reference tables.

  2. Overview of the Low-Level Memory-Based Revision Control API

    main

    The Low-Level Memory-Based Revision Control API is a new Lore API surface designed for automated services and pipelines (e.g., importers, mirrors, indexers, and search backends) that need to interact with repository structure without using a disk-backed working tree.

    While the storage system API (lore::storage) handles raw bytes via content addresses, and the file-system-based API (the lore CLI/crate) handles revisions via a working tree on disk, this new API bridges the gap by allowing users to manipulate the merkle tree structure directly in memory.

    Key capabilities include:

    • Resolving paths to opaque node IDs.
    • Listing directory children.
    • Reading leaf content addresses.
    • Mutating revision state via node-level edits.
    • Atomically committing new revisions and advancing branch tips.

  3. Overview of urc-aws

    main

    The urc-aws package provides wrappers and convenience tools designed to simplify working with Amazon Web Services (AWS) within various URC (Universal Resource Controller) components.

    Currently, the package provides support for:

    • S3
    • DynamoDB

  4. Overview of lore-proto

    main
    The lore-proto crate provides the Protocol Buffer (proto) files and the necessary build code to generate and expose them. It utilizes prost and tonic for code generation. Additionally, it includes a package.json to allow the proto files to be published to Artifactory.

  5. Understand Lore JavaScript Bindings implementation

    main

    The Lore library provides JavaScript bindings implemented using napi-rs. This approach allows JavaScript applications (such as VS Code plugins or Electron apps) to call Lore functionality directly without the overhead of spawning CLI processes.

    Key features of the bindings include:

    • Automatic Type Conversion: Native Rust data types are automatically converted to corresponding JavaScript data types.
    • Native Async Support: Async functions tagged in Rust are automatically exposed as JavaScript functions returning Promises, running on the Tokio runtime.
    • TypeScript Support: TypeScript interfaces are autogenerated from the Rust code, ensuring type safety and reducing maintenance errors.
    • Multiplatform: The bindings support cross-compiled multiplatform code that selects the correct binary implementation based on the system architecture.

  6. Understand the Lore Mutable Store and Partitioning

    main

    The mutable store is partition-scoped. Every mutable key exists in exactly one partition. A session bound to a specific partition cannot read or write keys in a different partition. The conditional-put primitive is enforced at the partition level, making partitions the primary access boundary for the storage subsystem.

    Key characteristics:

    • Partition Isolation: Keys with the same name in different partitions are distinct entries.
    • Scope: Partitions apply to the entire storage subsystem, including content-addressed components.
    • Contents: The mutable store contains only data that cannot be expressed via content addressing (e.g., name-to-ID mappings, latest pointers, metadata pointers).

  7. Understand Lore's Copy-on-Read Forking Mechanism

    main

    Lore uses a copy-on-read mechanism across partition boundaries to handle forks. When a session in a fork partition requests a fragment not yet registered in that fork, the server performs a read-through to the source partition. The fragment is then registered in the fork partition via a deduplication copy (the bytes are not re-transferred, only the registration is created in the storage layer's blob store).

    Subsequent accesses to that fragment are served directly from the fork partition without read-through. This allows forks to start empty and fill lazily based on the user's working set, rather than the full size of the source repository. Access control is maintained by requiring authorization for both the fork and the source partitions during read-through.

  8. Understand the Lore Storage API and Store Handles

    main

    Lore provides a standalone, content-addressed, deduplicated storage API that is independent of its version control subsystem. Applications can use Lore purely as a blob storage service (e.g., for build caches or asset pipelines) without using version control features like branches or merges.

    Access is managed through a store handle, which can be opened in two modes:

    • disk-backed: Persists against a local path.
    • in-memory: An ephemeral, transient store (useful for caches, scratch stores, or test fixtures).

    Both modes can optionally include a remote endpoint for fetching or pushing content to a peer storage service.

  9. Understand Lore's Filesystem-as-Truth model

    main

    Lore treats the filesystem as the authoritative source of truth for file content. Unlike Git (which uses an explicit index) or Perforce (which requires reconciliation), Lore does not maintain an intermediary index or a separate canonical copy of file content to keep in sync with the disk.

    Key Concepts:

    • Filesystem: The ground truth. Lore reads files directly when needed.
    • Staging: Recorded intent. It defines which paths from the filesystem should be included in the next revision.
    • Dirty Flag: A local-only flag on Merkle tree nodes indicating that a file differs from the committed revision. This flag is orthogonal to the staged flag.

  10. Forks and Isolated Partitions

    main
    Lore's data model supports forks as separate partitions with independent access control. A fork shares a source repository's initial content but evolves independently using copy-on-write. Future work aims to expose forks as a first-class operation, allowing teams to experiment in an isolated, access-controlled copy and eventually merge changes back to the source.

  11. Understand Lore's Design Goals

    main

    Lore is a version control system designed with the following core principles:

    • Binary-first: Content is treated as opaque byte streams; text-aware features are optional layers.
    • Centralized but offline-capable: The remote is the source of truth, but staging, committing, branching, switching, and diffing do not require a network round-trip.
    • Sparse by construction: Clients fetch only the specific fragments they need, minimizing the working set.
    • Atomic state: Operations are all-or-nothing; no partially applied revisions or half-pushed branches.
    • Cryptographically verifiable: Uses BLAKE3 hashes for fragments and hash-chained revision graphs.
    • API-first: The C library and public header are the primary artifacts. The CLI, language bindings, and server are thin layers over these APIs.
    • Multi-tenant safe: Designed to prevent cross-tenant content leakage in shared backend deployments.
    • Performance-oriented: Optimized for high-throughput, low-latency, and zero-copy deserialization.
    • Lifecycle-complete: Supports adding, deduplicating, and removing content without rewriting Merkle history.
    • Replaceable backends: Storage and transport are abstracted behind documented interfaces.

  12. Understand Lore documentation type families

    main

    Lore documentation is divided into two distinct families based on the topic covered, rather than the intended reader. Route your documentation by topic:

    1. Product docs (located in docs/): Describe Lore as a product (how to use it, its surfaces, and how it works). This includes tutorials, how-tos, references, and explanations. Even if a contributor reads them, if it documents the product (like a Rust API reference), it belongs here.
    2. Contributing docs (located in docs/developing/): Describe how to work on the Lore project (internals, architectural decisions, coding conventions, and environment setup).

    If a topic spans both families with significant depth, split it into two linked pages.