DOCSEARCHbyAlgolia
Submit Docs

msgvault Documentation

repository·main·Indexed 24 days ago

https://github.com/kenn-io/msgvault

An offline-first archiving and search tool for personal communications, including Gmail, Google Calendar, Microsoft Teams, Discord, and various chat/email export formats. msgvault features high-performance full-text and vector search, DuckDB-powered analytics, and an MCP server for AI agent integration. It utilizes a three-tier data model (Account, Identity, and Collection) and a 'safety ladder' approach to deduplication to manage overlapping message sources.

Tokens
128.2K
Snippets
293
Records
681
Agent score
83%

What's inside msgvault

  1. Overview of msgvault features

    main

    msgvault is an offline archive for email, chat, and calendar data. Key capabilities include:

    • Email & Chat Sync: Supports Gmail (via History API), IMAP, Microsoft 365, Microsoft Teams (via Microsoft Graph), Discord (via read-only bot), Beeper Desktop, Granola, and Circleback.
    • Local Imports: Import PST, MBOX, Apple Mail (.emlx), WhatsApp, iMessage, Google Voice, Facebook Messenger, and SMS Backup & Restore.
    • Search: Provides SQLite FTS5-powered full-text search (with Gmail-like syntax) and opt-in semantic/hybrid search using local or self-hosted OpenAI-compatible embedding endpoints.
    • Interfaces: Access your data via a keyboard-driven TUI (powered by DuckDB over Parquet), an analytical Web UI, a REST API, or an MCP (Model Context Protocol) server for AI assistants.
    • Data Management: Features incremental/resumable sync, content-addressed attachment storage (deduplicated by hash), and verifiable backup snapshots.

  2. Archive Microsoft Teams chats and channels

    main
    msgvault can archive Microsoft Teams chats, group chats, meeting chats, and team channel messages. Teams messages are stored with message_type = teams, allowing them to be searched and queried alongside email and text messages without mixing into email-only workflows. The sync process is read-only via Microsoft Graph and does not modify Teams content or membership.

  3. Supported data sources and import types

    main

    msgvault is compatible with a wide range of communication and productivity services. Once imported, all messages are accessible via the same Web UI, search, TUI, MCP, REST API, and export surfaces.

    Supported Sync/Import types:

    • Email: Standard IMAP servers, Microsoft 365 (Mail and Teams), PST, MBOX, and Apple Mail.
    • Chat/Messaging: Discord guilds (channels, threads, forum posts, and attachments), Beeper Desktop, WhatsApp, iMessage, Facebook Messenger, and SMS Backup & Restore (Android).
    • Calendar/Meetings: Google Calendar and supported meeting note services.

  4. Configure server security and binding

    main

    The msgvault server is designed for local use and follows these security defaults:

    1. Loopback-only: By default, it binds to 127.0.0.1. Access is restricted to the local machine.
    2. API Key Requirement: If you change the bind_addr to a non-loopback address (e.g., 0.0.0.0), the server requires an api_key to be set in config.toml and will refuse to start without one.
    3. Insecure Binding: To bind to a non-loopback address without an api_key (not recommended), you must set allow_insecure = true in the [server] section.

    Warning: Exposing the server on a network without authentication allows anyone on that network access to your entire email archive.

  5. Locate msgvault data storage

    main

    By default, msgvault stores all data locally on your machine. The archive consists of a SQLite database and Parquet analytics files.

    • Default Directory: ~/.msgvault (controlled by the MSGVAULT_HOME environment variable).
    • Remote Deployment: If configured for remote deployment, the archive resides on your own server.

  6. Filter messages by `message_type`

    main

    Because msgvault archives mixed data types (email, calendar, Teams, Discord, SMS, etc.) in the same messages table, you should use the message_type column to scope your queries.

    Known message_type values include: email, calendar_event, meeting_transcript, beeper, teams, discord, sms, mms, whatsapp, imessage, fbmessenger, synctech_sms_call, google_voice_text, google_voice_call, and google_voice_voicemail.

    # Teams activity by month
msgvault query --format table "
  SELECT month, count(*) AS messages
  FROM messages
  WHERE message_type = 'teams'
  GROUP BY month
  ORDER BY month DESC
  LIMIT 12
"

  7. How Repack works in Kit

    main

    The Repack process is used to optimize storage by consolidating entries into packs. It follows a strict sequence to ensure data integrity:

    1. Pruning: Removes stale mappings and retires zero-live packs.
    2. Snapshotting: Takes an exact snapshot of the current catalog-member entry set for the source.
    3. Verification: Performs bounded-reads and SHA-verifies every entry through the production store.
    4. Replacement: Writes, seals, syncs, and publishes a new replacement pack.
    5. Atomic Swap: Uses a compare-and-swap (CAS) operation via the adapter to update the source's entry set to the new mappings.
    6. Cleanup: Retires cached readers and deletes the old file/record.

    If the compare-and-swap fails due to a race condition, the replacement is treated as an 'orphan' and is handled by the reconciliation process. If physical retirement fails, the old record is retained to allow for retries.

  8. How vector search works in msgvault

    main

    Vector search enables semantic search, allowing users to find messages by meaning rather than just keyword matching. For example, a query for "planning offsite agenda" can find a message titled "Q2 team kickoff" if the content is semantically related.

    Search Modes

    When vector search is enabled, the search command and the HTTP /api/v1/search endpoint support two modes:

    • mode=vector: Pure semantic search.
    • mode=hybrid: A combination of BM25 (keyword) and vector search, fused using Reciprocal Rank Fusion (RRF).

    Implementation Details

    • SQLite archives: Vectors are stored in a local vectors.db file.
    • PostgreSQL archives: Vectors are stored in pgvector tables within the same database as the message archive.
    • Embedding Process: msgvault sends message text to a configured embedding endpoint. The vectors and the archive remain local, but the embedding computation happens at the endpoint.
    • MCP Support: The Model Context Protocol (MCP) equivalent is semantic_search_messages. A separate tool, find_similar_messages, can be used to find nearest-neighbor messages for a given seed.

    Important Note on Chat Imports

    Vector indexing operates over the shared messages table. However, chat import commands do not automatically trigger the embed worker. If you import local files or chat/text data, you must manually run the build command to include them in the index.

    msgvault embeddings build --full-rebuild --yes

  9. Understand Read behavior and retries

    main

    Store.Open validates the requested hash and performs a logical lookup via the Resolver for membership and an optional pack mapping.

    Lookup Logic:

    1. Non-member: Returns fs.ErrNotExist immediately without checking physical storage.
    2. Member without mapping: Opens the canonical loose path.
    3. Member with mapping: Reads through the bounded pack-reader cache.

    Retry Mechanism: Because authority can move between lookup and file open (e.g., due to concurrent packing or repacking), the reader performs a single re-resolution if:

    • A loose open returns fs.ErrNotExist (because packing committed and removed the source).
    • The mapped pack file or footer entry is absent (because repack swapped mappings).

    Safety: Bounded reads validate the container, footer, entry-count, offset, length, flag, CRC, decompressed size, and SHA-256 constraints before returning content.

  10. Understand the Relationship List Index architecture

    main

    The Relationship List Index is a specialized Parquet-based indexing system designed to provide high-performance, low-latency responses for relationship-based queries in large archives (e.g., 2.5M+ messages).

    Instead of performing expensive real-time joins across message and participant tables in DuckDB, the system uses four narrow, pre-computed datasets to serve specific sidebar and search endpoints. This architecture bounds memory and CPU usage while maintaining exact filter semantics for source, date, message type, participant, domain, and deletion state.

    Key Endpoints Served:

    • POST /api/v1/relationships
    • POST /api/v1/people/search
    • POST /api/v1/domains/search

  11. Understand API rate limiting

    main

    The msgvault API enforces rate limiting to prevent abuse.

    • Limit: 10 requests per second per client IP.
    • Burst: 20 requests.
    • Error Handling: When the limit is exceeded, the server returns an HTTP 429 status code. The response includes a Retry-After header indicating the number of seconds to wait before retrying.

  12. Understand the msgvault daemon architecture

    main

    Starting with version 0.17.0, msgvault CLI commands no longer access the SQLite database directly. Instead, they communicate with a msgvault daemon over HTTP.

    Key architectural changes:

    • Archive Access: Commands like stats, search, and tui query the daemon via HTTP. A local daemon is automatically started if one isn't running.
    • Operation Execution: Mutating commands (e.g., sync-full, import-*) are executed by the daemon, which then streams the output back to your terminal.
    • Single Writer Model: The daemon acts as the single writer for the SQLite database, preventing database is locked errors. Concurrent mutating operations are queued, and you may see a Waiting: message if an operation is in progress.
    • Read-Only Access: Commands that only read data (e.g., search, stats, tui, logs, list-deletions, embeddings list, or SQL query) are exempt from queuing and run immediately even during long-running mutations.
    • Data Format: The archive format remains unchanged (SQLite + Parquet in ~/.msgvault/).