DOCSEARCHbyAlgolia
Submit Docs

discrawl

repository·main·Indexed 21 days ago

https://github.com/openclaw/discrawl

A tool for archiving Discord data, including guilds and local desktop cache, into local SQLite databases for offline search, SQL analysis, and terminal browsing. It features bot-driven synchronization, local cache extraction via wiretap, a TUI explorer, and the ability to share archives via Git snapshots. Supports full-text search (FTS5), semantic search via embeddings, and JSON-outputting commands for automation.

Tokens
53.7K
Snippets
182
Records
234
Agent score
61%

What's inside discrawl

  1. Overview of discrawl

    main
    discrawl is a Go-based CLI tool designed to mirror Discord guild data into a local SQLite database. It allows for one-shot backfills, long-running live synchronization via the Discord Gateway, and importing personal DMs from local Discord Desktop cache artifacts without requiring user tokens. The tool supports fast text search (FTS5) and optional semantic search using OpenAI embeddings stored locally via a vector extension.

  2. Choose your Discrawl usage path

    main

    Discrawl offers different workflows depending on your goal:

    • Full Mirroring: For users wanting to sync guilds into local SQLite, follow the Install guide and run discrawl init.
    • Existing Bot Users: Use the sync and search commands to manage and query existing data.
    • Reading Shared Archives: If you do not have a Discord token and only want to read an archive, use discrawl subscribe (for Git snapshots) or discrawl subscribe-cloud (for Worker-fronted archives).
    • DM Search: To import messages from your local Discord Desktop cache (including DMs), use the wiretap command.
    • Semantic Search: To enable semantic/hybrid search, configure Embeddings and then run the embed command.
    • Automation & Integration: For wiring Discrawl into an agent or launcher, use the JSON-outputting control surface commands (e.g., metadata --json, status --json, diagnostics --json, coverage --json, failures --json, remote status, and doctor --json).

  3. Choose a search mode for discrawl search

    main

    The discrawl search command supports three distinct modes. By default, it uses Full-Text Search (FTS).

    • fts (default): Searches the local SQLite FTS5 index. Results are returned with the newest matching messages first.
    • semantic: Embeds the search query and scores it against locally stored message vectors. This mode requires an active embeddings configuration and compatible vectors. It will error out cleanly if embeddings are disabled or no compatible vectors exist.
    • hybrid: Combines FTS and semantic search. It deduplicates results by message ID and automatically falls back to FTS if semantic search is unavailable.
    discrawl search "query"
# or explicitly specifying modes:
discrawl search --mode fts "query"
discrawl search --mode semantic "query"
discrawl search --mode hybrid "query"

  4. Identify Discrawl archive modes

    main

    Discrawl operates using four distinct data paths depending on your source and access level. Understanding which mode you are using is critical for knowing what data is available (e.g., whether DMs are included or if you are reading from a remote cloud source).

    ModeTypical commandWhat it provides
    Local bot archivediscrawl sync --source discordBot-visible guild metadata and message history from the Discord API
    Local desktop cachediscrawl sync --source wiretapClassifiable cache evidence, including proven DMs, without a bot or user token
    Git snapshot readerdiscrawl subscribe ... / discrawl updateA shared non-DM archive imported into local SQLite
    Cloud remote readerdiscrawl remote statusWorker-fronted read-only archive metadata and queries without local SQLite

    Note that bot sync and wiretap can both update the same local SQLite archive, while Git snapshots and cloud remote modes are read-only paths for data that has already been published.

  5. Compare local vs remote embedding providers

    main

    The privacy and data flow of your embeddings depend on the provider type:

    • Local Providers (e.g., Ollama): Both the message embeddings and the search query embeddings are processed on your local machine. Stored message text is not sent during local vector scoring.
    • Remote Providers (e.g., OpenAI):
      • During discrawl embed, message text is sent to the provider.
      • During discrawl search --mode semantic or --mode hybrid, the search query text is sent to the provider.

  6. How `update` interacts with `sync`

    main

    It is important to distinguish between importing snapshots and syncing live data:

    • discrawl update: Explicitly imports data from a Git snapshot into the local cache.
    • discrawl sync: Refreshes live data. By default, sync does not auto-import the share snapshot.

    To make sync perform an import, you must pass one of the following flags:

    • --update=auto: Performs a safe merge if the local cache is stale.
    • --update=force: Performs an exact replacement of the snapshot before applying live deltas.

  7. Understand `embed` job behavior and error handling

    main

    The embed command follows these operational rules:

    • Concurrency Control: It claims jobs using a short lock to prevent overlapping runs from processing the same batch.
    • Rate Limiting: If the provider returns a rate limit error, the batch is requeued and the current drain run stops cleanly.
    • Retries: Provider or validation failures trigger up to three retry attempts before the job is marked as failed.
    • Empty Content: Messages that contain no normalized text are marked as done, and any existing stale vectors for those messages are removed.

  8. How the embeddings two-phase pipeline works

    main

    The embedding process is split into two distinct phases to ensure stability and prevent provider overload:

    1. Queue Phase: Commands like sync --with-embeddings or tail --with-embeddings identify changed or new messages and write them as rows in the embedding_jobs table. The embedding provider is not contacted during this phase.
    2. Drain Phase: The discrawl embed command claims pending jobs from the queue using a short lock to prevent overlapping runs. It then calls the configured provider and writes the resulting vectors to the message_embeddings table, including metadata like provider, model, input_version, dimensions, and the binary vector data.

    Error Handling during Drain:

    • Rate limits: The batch is requeued and the current drain run stops cleanly.
    • Provider/Validation failures: The system retries up to three attempts before marking the job as failed.
    • Empty text: Messages with no normalized text are marked as done, and any stale vectors for that message are removed.

  9. Understand SQL query limitations and integrity warnings

    main

    When using discrawl sql, keep the following behaviors in mind:

    • Zero-row queries: A query returning zero rows only describes the current local snapshot; it cannot prove that data is absent from Discord itself.
    • Orphaned messages: If you use an inner join on channels, messages with incomplete metadata are dropped. discrawl sql will warn on zero-row output if it detects orphaned references.
    • undetermined status: If a diagnostic note says undetermined, it means the catalog probe failed or timed out (within its 2-second budget), not that referential integrity was verified.
    • Consistency: A catalog.state of consistent only guarantees that stored messages resolve to stored channels; it does not guarantee that the archive covers all source data from Discord.

  10. Understand FTS search behavior and limitations

    main

    Full-Text Search (FTS) in Discrawl is powered by SQLite FTS5 using the unicode61 tokenizer.

    Key behaviors:

    • Operator Escaping: User query terms are parameterized and quoted before the MATCH operation. This means FTS operators like AND, OR, NOT, NEAR, and * are treated as literal input terms rather than functional operators.
    • Punctuation: Follows standard FTS5 tokenization rules.
    • Content Filtering: By default, search skips rows with no searchable content. Searchable content includes attachment text, attachment filenames, embeds, and replies. To include rows that are otherwise considered empty, use the --include-empty flag.
    discrawl search --include-empty "GitHub"

  11. Untitled record

    main

    The wiretap component is designed to be local-only and does not interact with the Discord API as a user or selfbot. It reads local Discord Desktop cache files and adheres to the following security constraints:

    • It does not extract, store, or print Discord auth tokens.
    • It does not use a user token or call the Discord API as your user.
    • DM messages (stored under the synthetic guild ID @me) stay local and are never exported via publish, subscribe, or the --with-embeddings snapshot export.
    • A shared guild mirror refresh does not wipe local wiretap DM search; imports preserve existing local @me guilds, channels, messages, and attachments.

  12. Understand `diagnostics` report details and safety guarantees

    main

    When interpreting the output of discrawl diagnostics, keep the following distinctions in mind:

    Integrity vs. Consistency

    • safe_for_read_only_inspection: This reports only SQLite-byte integrity. It confirms the database file is not corrupted at the byte level, but it does not guarantee that every message has a corresponding channel record.
    • catalog.state: "consistent": This indicates that every stored message in your local archive has a corresponding stored channel record. It does not guarantee that your local archive contains every message or channel currently existing on Discord.

    Writer State and Locks

    • Active Writers: Discrawl identifies active operations (like sync, tail, wiretap, or import) by reading lock metadata. It does not scan all operating-system processes.
    • Stale Metadata: If a writer exits unexpectedly and leaves a lock file behind, diagnostics will report the leftover owner data as stale_metadata rather than incorrectly identifying it as an active writer.
    • Lock Detection: On platforms where the operating system does not support a specific file-lock probe, the JSON output will explicitly state detection: "unsupported" instead of attempting to guess the state.