DOCSEARCHbyAlgolia
Submit Docs

indigo

repository·main·Indexed 23 days ago

https://github.com/bluesky-social/indigo

A collection of Go-based libraries and services for the AT Protocol (atproto), featuring implementations for synchronization, relays, firehose splitting, Lexicon handling, and DID resolution. It includes specialized components such as the automod rules engine, Carstore for PDS-like repository data, bluepages identity directory, and the collectiondir microservice.

Tokens
35.6K
Snippets
51
Records
222
Agent score
79%

What's inside indigo

  2. Overview of Go services in indigo

    main

    The indigo repository contains several Go-based services for the AT Protocol:

    • tap: A synchronization and backfill tool for atproto applications.
    • relay: A reference implementation of a relay.
    • rainbow: A firehose "splitter" or "fan-out" service.
    • hepa: An auto-moderation bot designed for use with Ozone.

  3. Overview of hepa (indigo edition)

    main

    hepa is an auto-moderation daemon that wraps the automod package. It is designed to keep the local atmosphere of an instance clean by processing events from the Relay firehose.

    Key characteristics:

    • State Management: All state (counters) and caches are stored in Redis.
    • Data Source: It consumes events from the Relay firehose (note: backfill functionality is not yet supported).
    • Configuration: The specific rules included in the daemon are configured at compile time.
    • Privacy & Access: Admin access to fetch private account metadata or persist moderation actions is optional; anyone can run a hepa instance without these permissions.
    • Performance: Initial startup performance may be slow as the daemon fetches and caches account-level metadata for every firehose event. Processing speed increases once caches are "warmed up."
    • Scope: It is not a labeling service; it does not push labels to existing moderation services, nor does it provide API endpoints or label streams.

  4. Overview of the rainbow service

    main

    The rainbow service is an atproto Firehose Fanout Service. It consumes events from a firehose (such as a relay or PDS) and fans them out to multiple subscribers.

    Key characteristics:

    • Functionality: Serves the com.atproto.sync.subscribeRepos WebSocket endpoint and proxies public/administrative API requests to a backing host.
    • Data Handling: Retains upstream firehose "sequence numbers" and maintains a "backfill window" on local disk using pebble.
    • Pass-through: It does not validate events (signatures, repo trees, hashes, etc.) and does not archive or mirror individual records or repositories.
    • Performance Requirements: The service is disk I/O intensive; fast NVMe disks and sufficient RAM for caching are recommended.
    • Deployment: Distributed as a single Go binary and supports observability via logging, Prometheus metrics, and OTEL traces.

  5. Choose a Carstore implementation

    main

    Carstore is designed to store massive amounts of data from PDS-like repositories, primarily focusing on firehose ingestion and egress. Depending on your scale and requirements, you can choose from three implementations:

    1. FileCarStore: The original production implementation (used through at least 2024-11). It stores 'car slices' from subscribeRepo firehose streams to the filesystem and uses GORM (PostgreSQL or SQLite3) for metadata. It supports periodic compaction of car slices into larger ones.
    2. ScyllaStore: A scalable implementation where blocks are stored in ScyllaDB, while user and PDS metadata are managed via GORM (PostgreSQL or SQLite3).
    3. SQLiteStore: An experimental/demo implementation where both blocks and metadata are stored in a local SQLite3 schema. This serves as a minimal reference implementation.

  6. What is Sonar and how does it work

    main

    Sonar is an AT Proto Firehose Monitoring tool designed to monitor event throughput on the network. It connects to an AT Proto Firehose (following com.atproto.sync.subscribeRepos semantics) from either a PDS or a Relay.

    Sonar performs two main monitoring tasks:

    1. Event Throughput: Produces Prometheus metrics based on the frequency of different event types.
    2. Repo Operations: Tracks the frequency of creation, update, and deletion operations across different record collections.

    It is primarily used to provide operational dashboards to visualize changes in event rates and understand network traffic patterns over time.

  7. OAuth implementation requirements and constraints

    main

    When working with the atproto/auth/oauth implementation, note the following technical requirements:

    • DPoP: Demonstrating Proof-of-Possession (DPoP) starts at the Pushed Authorization Request (PAR) stage.
    • Cryptography: The implementation requires ES256 (P-256) for both DPoP and client attestation private keys. While the API uses flexible interface types, the underlying requirement is ES256.
    • Scopes: Scopes are configured as part of the client metadata and are applied to each session.

  8. Filter Events by Collection

    main

    Once a network boundary is established, you can further filter record events by collection. Filters apply only to record events; identity events are always delivered for all tracked repositories.

    Syntax: Use TAP_COLLECTION_FILTERS with a comma-separated list. Wildcards are supported but only at period breaks in the NSID (e.g., app.bsky.graph.*).

    Example: To sync all records of a single type using a signal collection, you must specify that collection in both the signal and the filter settings:

    TAP_SIGNAL_COLLECTION=com.example.nsid TAP_COLLECTION_FILTERS=com.example.nsid

    To filter for specific collections:

    TAP_COLLECTION_FILTERS=app.bsky.feed.post,app.bsky.graph.*

  9. Choose a Tap Delivery Mode

    main

    Tap supports three delivery modes depending on your requirements for reliability and throughput:

    1. WebSocket with acks (Default): The client sends acknowledgments for each event after processing. This ensures no data loss and removes the need for the client to manage cursors. Recommended for most use cases using the @atproto/tap library.
    2. Fire-and-forget: Set TAP_DISABLE_ACKS=true. Events are sent and considered acknowledged immediately upon receipt. This is simpler but may result in data loss. Recommended for testing or non-critical data.
    3. Webhook: Set TAP_WEBHOOK_URL=http://.... Events are POSTed as JSON to the specified URL. An event is considered acknowledged once the webhook responds with a 200. Recommended for lower-throughput serverless environments.

  10. Understand the OAuth package structure in indigo

    main

    The atproto/auth/oauth package is organized around four primary abstractions for managing OAuth flows and sessions:

    • oauth.ClientApp: Represents the overall application or service. It manages client metadata, client attestation secrets (for confidential clients), and handles request and session storage. It is used to establish and manage oauth.ClientSession instances.
    • oauth.ClientSession: Represents an established user session. It wraps DPoP keys, tokens, and other metadata. It implements client.AuthMethod for use with an ApiClient, automates token refreshes (requiring a client secret for confidential clients), and triggers callbacks when session data (like nonces or tokens) are updated.
    • oauth.ClientAuthStore: An interface used to implement persistent storage systems for auth request and session metadata, including secrets and DPoP private keys.
    • oauth.Resolver: Responsible for resolving OAuth data, currently performing direct network resolutions.

  11. Understand relay operational behaviors

    main

    The relay has several specific behaviors regarding message handling, persistence, and connectivity that impact how it interacts with the atproto network:

    Message Handling

    • Field Pass-through: Records and commit objects are passed through verbatim (serialized in blocks fields on #commit and #sync messages). However, unknown or unexpected fields on overall firehose messages (e.g., #commit) are not passed through. It is critical to upgrade the relay when protocol changes occur.
    • Sequence Numbers: For an established upstream host WebSocket, sequence numbers must always increase; lower sequence messages are dropped. If the relay restarts, it attempts to reconnect at the last persisted sequence number. If a host is new, the relay connects at the "current" firehose offset rather than backfilling from the "oldest".
    • Account Revisions: Account-level #commit revisions must always increase. Repeated or lower revision messages, or messages with revisions too far in the "future" (beyond a few minutes), are dropped.
    • Host Validation: Messages for an account (DID) coming from a host that is not the current PDS host for that account are dropped. The relay will re-resolve the identity (DID document) to check for recent account migrations before dropping.

    Persistence and Configuration

    • Ephemeral Settings (configured via env vars):
      • new-hosts-per-day limit
      • requestCrawl (enable/disable)
    • Persisted Settings (stored in database):
      • Account takedowns
      • Domain bans
      • Host bans
      • Host account limit

    Connectivity and Security

    • SSRF Protection: When connecting to remote hosts, the relay includes protections against connecting to private, reserved, or local IP addresses, or ports other than 80/443 (unless explicitly connecting to localhost with a port).
    • Loop Prevention: The relay checks the Server header in HTTP responses. If it finds atproto-relay, it refuses the connection to prevent relay request loops.
    • Sibling Relays: If configured with "sibling" relay instances, the relay forwards requestCrawl and certain administrative requests. Forwarding only occurs if the request is processed successfully on the current instance. Auth headers are passed through, so sibling relays must share the same secret.

  12. Understand the collectiondir design and data flow

    main

    The collectiondir service is a single Go binary designed to maintain a directory of DIDs and their associated collections.

    Key Design Points:

    • Data Persistence: Uses a local pebble key/value database.
    • Data Ingestion: Consumes from the atproto firehose to stay updated with real-time record creation.
    • Bootstrapping: Can bootstrap the full network state by using com.atproto.sync.listRepos and com.atproto.repo.describeRepo.
    • Database Schema: The primary schema is (collection, seen time int64 milliseconds, did). This structure enables efficient cursor-based fetching of DIDs for a specific collection.

    Typical Workflow for New Services:

    1. A new service starts consuming the firehose for a specific collection (e.g., com.newservice.data.thing).
    2. The service calls listReposByCollection to find DIDs that have already created data in that collection.
    3. The service performs getRepo calls to those DIDs' PDSes to backfill historical data.
    4. Once backfilling is complete, the service relies on live firehose events for new data.