DOCSEARCHbyAlgolia
Submit Docs

ATProto Feed Generator Starter Kit

repository·main·Indexed 24 days ago

https://github.com/bluesky-social/feed-generator

A starter kit for building ATProto Feed Generators to implement custom algorithms that serve post skeletons to the Bluesky network. It provides implementations in both Go and TypeScript, including indexing logic, feed generation, JWT authentication for user-specific state, and pagination using cursors. The kit includes a sample algorithm called 'whats-alf' and supports SQLite by default.

Tokens
14.1K
Snippets
45
Records
81
Agent score
77%

What's inside bluesky-social-feed-generator

  1. How ATProto Feed Generators work

    main

    A Feed Generator is a service that provides custom algorithms to users via the AT Protocol.

    The Workflow:

    1. Request: A user's Personal Data Server (PDS) requests a feed using the at-uri of a declared feed record.
    2. Resolution: The PDS resolves the at-uri to find the Feed Generator's DID document.
    3. Requesting Skeleton: The PDS calls the app.bsky.feed.getFeedSkeleton endpoint on the Feed Generator. This request is authenticated via a JWT signed by the user's repo signing key.
    4. Response: The Feed Generator returns a 'skeleton' (a list of post URIs and optional metadata).
    5. Hydration: The PDS takes these URIs and 'hydrates' them (fetching full user info, post content, etc.) before sending the final feed to the client.

    Key Abstractions:

    • Service DID: Identifies the Feed Generator service itself.
    • Algorithm Declaration: Each algorithm hosted by the service is declared as a record in the creator's repository. The algorithm is identified by the at-uri of this declaration record.

  2. Implement Pagination with Cursors

    main

    The getFeedSkeleton method uses an opaque cursor for pagination. The Feed Generator returns a cursor in its response, and the PDS passes that same cursor back as an input in subsequent requests.

    Best Practices:

    • The cursor is entirely at the Feed Generator's discretion.
    • Crucial: Ensure the cursor is unique per feed item to prevent unexpected behavior during pagination.
    • Recommended Pattern: Use a compound cursor consisting of a timestamp and a CID.

    Example Compound Cursor: 1683654690921::bafyreia3tbsfxe3cc75xrxyyn6qc42oupi73fxiox76prlyi5bpx7hr72u

  3. Quickstart the Go Feed Generator

    main

    To run the Go implementation of the AT Protocol Feed Generator locally, copy the example environment file, configure your values, and run the server using go run .. You can then verify the feed by querying the getFeedSkeleton XRPC endpoint via curl.

    cp .env.example .env
# Edit .env with your values

go run .

# In another terminal, get some feed items with:
curl -s "http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf&limit=10" | jq .

  4. Getting Started with the Feed Generator Starter Kit

    main

    To build a custom feed using this starter kit, you need to implement two main components:

    1. Indexing Logic: Implement this in src/subscription.ts. This logic subscribes to the repository subscription stream on startup, parses events, and indexes them according to your algorithm's needs.
    2. Feed Generation Logic: Implement this in src/algos. You must return an array that satisfies the SkeletonFeedPost[] type. The starter kit includes a sample algorithm called whats-alf.

    Note on Database: The starter kit uses SQLite by default, but you can replace it with any database of your choice.

  6. Add Custom Feeds in Go

    main

    To implement a new feed algorithm, you must modify algos.go.

    1. Define a handler function with the following signature: func(ctx context.Context, db *Database, limit int, cursor string) (*bsky.FeedGetFeedSkeleton_Output, error)
    2. Register your new handler in the Algos map, using the feed's unique rkey as the map key.

  7. Deploy and Publish your Feed

    main

    Deployment Requirements

    • The service must be accessible via the hostname specified in the FEEDGEN_HOSTNAME environment variable.
    • The service must respond to HTTPS queries over port 443.

    Publishing the Feed

    To publish your feed generator and its metadata (name, avatar, description) to the ATProto network:

    1. Open scripts/publishFeedGen.ts and fill in the required variables at the top.
    2. Run the following command:
    yarn publishFeed

    Once published, the feed will be visible within the Bluesky app and can be shared via links.

  8. Run the Feed Generator Server

    main

    Using yarn

    Install dependencies and start the server:

    yarn
yarn start

    The server starts on port 3000 (or the port defined in your .env file). You can test the default whats-alf feed at: http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf

    Using Docker

    Build and run the server using Docker:

    docker build -t feed-generator . && docker run feed-generator

  9. Subscribe to repository event streams

    main

    The com.atproto.sync.subscribeRepos service provides a stream of events for a specific repository. Developers implementing a feed generator or indexer can use the Handler type to process these events. The stream emits various event types including commits, handle changes, migrations, and tombstones.

    Event Types

    • Commit: Contains repository changes, including a sequence number (seq), the commit CID, and a list of RepoOp operations.
    • Handle: Emitted when a user's handle changes.
    • Migrate: Emitted when a repository is migrating to a new DID.
    • Tombstone: Emitted when a repository is deleted.
    • Info: Can be used to signal an OutdatedCursor error.

    Handler Interface

    An implementation of Handler must accept a HandlerReqCtx and return an AsyncIterable<HandlerOutput>. The context provides access to authentication, query parameters (like cursor), the raw request, and an AbortSignal for cancellation.

    export type Handler<HA extends HandlerAuth = never> = (
  ctx: HandlerReqCtx<HA>,
) => AsyncIterable<HandlerOutput>

  10. View moderation status for repos, records, or blobs

    main

    The administrative API provides several 'View' types to inspect the moderation state of entities. These types are used to determine if an entity (like a repository, a specific record, or a blob) is currently subject to administrative actions or reports.

    Key View Types

    • RepoView / RepoViewDetail: Information about a repository, including its handle, DID, and moderation status.
    • RecordView / RecordViewDetail: Information about a specific record, including its URI, CID, and the associated RepoView.
    • BlobView: Metadata for a blob (image/video), including its CID, MIME type, and moderation status.
    • Moderation: A lightweight object containing the currentAction (of type ActionViewCurrent) applied to an entity.
    • ModerationDetail: A comprehensive view containing all actions (ActionView[]) and reports (ReportView[]) associated with an entity.

  11. Understand the Lexicon Namespace Hierarchy

    main

    The Server class organizes all available XRPC methods into a hierarchical namespace structure. This allows for discoverable and typed access to different protocol domains:

    1. server.com: Accesses the com.atproto namespace.

      • atproto.admin: Administrative actions (e.g., moderation, account management).
      • atproto.identity: Identity management (e.g., resolving handles).
      • atproto.label: Labeling services.
      • atproto.moderation: User-driven moderation reports.
      • atproto.repo: Repository operations (e.g., creating/deleting records, uploading blobs).
      • atproto.server: Server-level management (e.g., session management, account creation).
      • atproto.sync: Synchronization primitives (e.g., getting blobs, subscribing to repo updates).

    2. server.app: Accesses the app.bsky namespace.

      • bsky.actor: Actor/Profile operations (e.g., getting profiles, searching actors).
      • bsky.feed: Feed-specific operations (e.g., getting feeds, describing feed generators).
      • bsky.graph: Social graph operations (e-g., follows, mutes, lists).
      • bsky.notification: Notification management.
      • bsky.unspecced: Unspecified or experimental Bluesky features.

  12. How to process repository commits in a subscription

    main

    When consuming the firehose, you typically implement a handler for RepoCommit events. The process involves:

    1. Reading the Repository: Use repo.ReadRepoFromCar with the evt.Blocks from the commit event to access the actual record data.
    2. Iterating Operations: Loop through evt.Ops to identify create or delete actions.
    3. Constructing URIs: Build the record URI using the format at://<repo>/<path>.
    4. Handling Creates: For create actions, extract the record bytes from the CAR file using rr.GetRecordBytes(ctx, op.Path) and unmarshal them (e.g., into a bsky.FeedPost).
    5. Handling Deletes: Collect URIs for delete actions to perform bulk deletions in your local database.
    6. Updating Cursors: Periodically save the evt.Seq to your database using s.db.UpdateCursor to ensure you can resume after a restart.