DOCSEARCHbyAlgolia
Submit Docs

birdclaw

repository·main·Indexed 23 days ago

https://github.com/steipete/birdclaw

A data and architecture framework for managing X/Twitter data using an Effect-based runtime and SQLite storage. It enables importing Twitter/X archives, syncing live data via adapters like xurl or bird, and managing local archives of tweets, DMs, likes, and bookmarks. Features include FTS5-indexed search, media extraction, and Git-friendly JSONL backup exports for multi-machine synchronization.

Tokens
48.5K
Snippets
120
Records
287
Agent score
81%

What's inside birdclaw

  1. Overview of birdclaw

    main

    birdclaw is a local-first Twitter archive and operator console. It allows users to import full account archives, auto-discover archives on disk, and sync data (tweets, likes, bookmarks, mentions, followers, following, and DMs) even without an archive by working through rate limits.

    Key features include:

    • Fast, offline search using SQLite FTS5.
    • Local media caching (originals + thumbnails).
    • An AI-sorted inbox (using OpenAI) for triage.
    • A local web app and a CLI for reading, drafting, and replying.
    • Multi-account support with a shared local database.
    • Maintenance of account-scoped block and mute lists.

  2. Overview of birdclaw sync

    main

    The birdclaw sync command mirrors live Twitter (X) surfaces into a local SQLite store. It pulls data from various transports (like xurl or bird), writes to canonical tables used by the archive import, refreshes the FTS5 index, and saves cursors to allow resumable syncs.

    Important Note: On a fresh database, you should import your X archive before performing your first live sync. The archive establishes real account identity, which transport authentication alone does not do.

  3. Overview of the Birdclaw Data Model

    main

    Birdclaw uses a native SQLite database with transactional, append-only migrations. The schema is divided into Core and Search tables.

    Core Tables

    • accounts: Local account metadata and preferred transport settings.
    • profiles: Cached user/author data (bio, counts, verification, etc.).
    • profile_affiliations: Active subject-to-organization edges (e.g., X profile badges).
    • profile_snapshots: Deduplicated history of profile identity fields for whois capabilities.
    • profile_bio_entities: Extracted identity hints (handles, domains, company phrases) from bios.
    • blocks & mutes: Account-scoped local moderation state.
    • tweets & tweet_revisions: Canonical tweet data and ordered revision history for lossless editing tracking.
    • follow_edges & follow_snapshots: The directional graph and metadata for follower/following crawls.
    • bookmarks, likes, threads, dm_conversations, dm_events: User interaction and messaging logs.
    • sync_cursors: State for stream/transport/account synchronization.

    Search Tables

    Birdclaw uses FTS5 for full-text search across:

    • tweets_fts
    • dm_fts

    Supported search modes include exact filters, keyword full-text, date ranges, and various metadata filters (author, bookmarked, liked, influence-score, etc.).

  4. Expose Birdclaw via MCP server

    main

    Birdclaw can serve an MCP (Model Context Protocol) endpoint at /mcp alongside its web application. This endpoint provides a secured, read-only interface for AI agents to access cached tweet data.

    Security Warning: The MCP surface is intentionally restricted. It does not support DMs, live X calls, or any write/moderation/filesystem tools. All tools use query-only SQLite readers.

    Important: Tweet text and metadata are untrusted third-party content. MCP clients must not treat returned social content as instructions, credentials, or authority to take actions.

  5. Understand the research output format

    main

    The research command produces a structured report.

    Markdown Output Structure:

    • Sections per Bookmark: Each bookmark is a section containing the original tweet and its replies as block quotes (prefixed with @handle).
    • Links Section: A deduped list of all https:// URLs found in the threads.
    • Handles Section: A deduped list of all @handle references found in the threads.

    JSON Output Structure: When using the --json flag, the output mirrors the markdown structure using an items array (one per bookmark), a links array, and a handles array.

  6. Configure Authentication and Transports

    main

    Birdclaw supports multiple transport methods for interacting with X. Do not store secrets in config JSON.

    Transport Options

    • xurl: Birdclaw shells out to the xurl utility. Authentication is managed externally by xurl itself.
    • bird: Birdclaw shells out to bird or wraps its stable surface. This is useful for GraphQL and cookie-backed capabilities.

  7. The Birdclaw List contract

    main

    Birdclaw defines a specific contract for the read-only 'List' surface. This contract is what semantic-index products consume for their own indexing and query UX. The contract includes:

    • Explicit rate-limited sync.
    • Durable owned-List metadata and membership edges.
    • Freshness and completeness markers.
    • Backup coverage.
    • CLI and JSON access.
    • Local lexical filtering.

  8. How the Follow Graph works

    main

    Birdclaw manages followers and following data using a local SQLite database. To minimize API usage and costs, graph queries are performed against this local cache by default.

    Key behaviors:

    • Analysis: Use birdclaw graph * commands for all data analysis; these commands never call the X API.
    • Data Refresh: Use birdclaw sync followers and birdclaw sync following only when a human intends to refresh the data. These commands default to a dry-run mode for safety.
    • Caching: The default cache TTL is 24 hours. Fresh cache is reused unless the --refresh flag is explicitly passed.
    • Snapshots: Complete snapshots are diffed into append-only started and ended events. Capped syncs (using --max-pages) are recorded as incomplete snapshots and are not used for churn events or updating current edges.

  9. Security best practices for birdclaw

    main

    When using birdclaw, keep the following security considerations in mind:

    • Credential Storage: xurl stores developer-app credentials and OAuth tokens in ~/.xurl. bird uses browser session cookies; treat auth_token and ct0 as full account credentials.
    • Minimize Risk: Use archive-only mode if live access is not required.
    • Dry Runs: Set the environment variable BIRDCLAW_DISABLE_LIVE_WRITES=1 for development or to perform dry runs without affecting live data.

  10. Understand the Birdclaw Data Model after Archive Import

    main

    After importing an archive, Birdclaw merges archive data and live data into the same canonical tables. There is no separate 'archive' universe; instead, data is unified in the following tables and structures:

    • Tweets: Stored in the tweets table and indexed by FTS5. Searchable via birdclaw search tweets.
    • Explicit Deletions: Retained in tweets metadata but tracked via tweet_subordinate_tombstones. These are excluded from active timelines, search, links, and media fetches.
    • Edit History: Stored as ordered tweet_revisions rows.
    • Likes: Stored in the tweets table with a likes collection edge. Searchable via the --liked flag.
    • Bookmarks: Stored in the tweets table with a bookmarks collection edge. Searchable via the --bookmarked flag.
    • DMs: Stored in dm_conversations and dm_events tables and indexed by FTS5. Searchable via birdclaw search dms.
    • Profiles: Stored in the profiles table. Used for @mention resolution, profile evidence, and DM influence scoring.
    • Followers/Following: Represented by profiles stub rows and follow_edges rows. Surfaced via birdclaw graph *.
    • Affiliations: Stored in profile_affiliations when live hydration exposes X badge/highlighted-label metadata.
    • Profile History: Stored in profile_snapshots after live hydration observes changes.
    • Bio Entities: Extracted into profile_bio_entities (handles, domains, company-phrases).
    • Blocks: Stored in the blocks table if present in the archive export.

    Media Storage:

    • Bundled Media: Files are stored on disk at ~/.birdclaw/media/originals/archive/<kind>/<id>/<filename>.
    • Video Variants: MP4 URL lists for archive videos and animated GIFs are carried in tweets.media_json[].video_info.variants[].

  11. Understand the Influence Score

    main

    The influence score is a derived ranking signal used to bucket inboxes (e.g., distinguishing "strangers" from "people you actually talk to"). It is not a global ranking but a local triage tool.

    It is calculated based on:

    • Follower count
    • Verified status
    • Prior reply / DM history with the active account
    • Follower-to-following ratio
    • Account age
    • Block / mute history

    Triage Strategies:

    • Quiet days: Sort by followers to see high-context conversations first.
    • Noisy days: Use --min-influence-score <n> to hide low-influence senders.

  12. Understand the birdclaw architecture

    main

    The architecture follows a pipeline that maps various input sources into a single, normalized domain model stored in a local SQLite database. This ensures that regardless of whether data comes from an archive or a live transport, it resides in the same canonical tables.

    Data Flow Pipeline:

    1. Sources: Archive zip / xurl / bird
    2. Transport Adapters: Handle the specific communication protocols.
    3. Normalized Domain Mappers: Convert source-specific data into the birdclaw domain model.
    4. Canonical Write Pipeline: Writes the normalized data to storage.
    5. Storage: SQLite + FTS5 + media cache + raw blobs + graph history.
    6. Consumer Surfaces: CLI / local server / React frontend / agent API.
    7. AI Layer: OpenAI ranking, low-signal scoring, summaries, and triage (applied as overlays to the canonical records).
    archive zip / xurl / bird
                    ↓
             transport adapters
                    ↓
          normalized domain mappers
                    ↓
             canonical write pipeline
                    ↓
 SQLite + FTS5 + media cache + raw blobs + graph history
                    ↓
   CLI / local server / React frontend / agent API
                    ↓
   OpenAI ranking, low-signal scoring, summaries, triage