DOCSEARCHbyAlgolia
Submit Docs

LLM Wiki

repository·master·Indexed 23 days ago

https://github.com/lucasastorian/llmwiki

An autonomous, self-maintaining personal knowledge base that uses the Model Context Protocol (MCP) to allow LLMs like Claude to ingest research, clip web content, and synthesize information into a structured wiki. It supports local and hosted modes, featuring a filesystem-as-source-of-truth model, a Chrome extension for web clipping, and a converter for extracting content from PDF and Office documents.

Tokens
19.8K
Snippets
26
Records
116
Agent score
80%

What's inside llmwiki

  1. Core features and scope of LLM Wiki V0

    master

    The V0 release focuses exclusively on rendering courses as markdown and making user progress legible. The primary value proposition is the ability to resume learning across sessions.

    Included Features (V0):

    • Folder-driven Part/Lesson hierarchy.
    • Progress visualization (rail-as-progress-meter, reading tracker that checks off sections on scroll).
    • Completion metrics (lesson completion, overall and per-Part counts).
    • Soft locks for content progression.
    • Markdown overview with a 'resume' action.
    • Markdown rendering with syntax highlighting for code blocks.

    Deferred Features (Phase 2):

    • Quizzes.
    • Highlights and notes.
    • Executable/graded coding exercises.
    • Video embeds.

  2. Understand the Recent Changes feature behavior

    master

    The Recent Changes feature provides a deterministic, newest-first activity timeline for a wiki. It replaces the manual log.md file as the source of truth for activity.

    Key Behaviors:

    • Accessing the feed: Opening a wiki root URL without a page parameter (e.g., /wikis/{slug}) opens the Recent Changes feed.
    • Direct Page Access: URLs with a page parameter (e.g., /wikis/{slug}?p=12) bypass the feed to open a specific page.
    • Navigation: 'Recent' is a top-level item in the wiki sidebar. Clicking it creates a browser-history entry and removes the ?p= parameter.
    • Privacy: In V1, public wiki surfaces do not expose this feed to prevent leaking source filenames or private work patterns.

  3. Understand the LLM Wiki architecture

    master

    LLM Wiki operates using a unified storage abstraction called VaultFS. This abstraction allows the system to behave identically regardless of whether you are running in local mode or hosted mode.

    Storage Modes

    • Local mode: Uses SQLite combined with your local filesystem.
    • Hosted mode: Uses Postgres combined with S3.

    Client Access Patterns

    • Claude: Interacts via an MCP server (Model Context Protocol).
    • Web app & Chrome extension: Interact via an API over HTTP.
    • Converter: A specialized service used for extracting text from PDF and Office documents.

  4. Component Design: Sidebar and Navigation

    master

    The sidebar uses a tri-state status glyph system to indicate progress without relying on color:

    • Completed: An accent check on accSoft.
    • In-progress: A partial accent ring.
    • Not-started: An empty border circle.

    Sidebar Elements

    • Sidebar nav item: Includes a status glyph and a label. States include default, hover (raised/60), and active (raised background + ink text + medium weight).
    • Module group: Displays an uppercase label, a count (e.g., 2/4), and a 1px hair progress hairline.
    • Sidebar timeline: A single rail connecting units of a Part. Units are nodes (completed/in-progress/locked). Current units expand to show sub-steps like Lessons and Quizzes.
    • Gating: Locked units are 'soft' locks. They are dimmed with a lock icon and a hint (e.g., "pass the quiz to unlock"), but remain clickable.

  5. Understand the LLM Wiki core concept and modes

    master

    LLM Wiki is a local-first knowledge environment designed for reading, annotating, and progressing through long-form content. It uses a single engine to support two distinct content modes:

    • Wiki Mode: A reference graph compiled from user sources intended for non-linear exploration (dipping in and out).
    • Course Mode: An ordered learning journey featuring progress tracking and quizzes designed for linear progression.

    Both modes share the same renderer, sidebar, and underlying file structure on disk. Switching between modes is a change in affordances (like progress tracking), not a change in the application itself.

  6. Event Capture Rules and Data Privacy

    master

    The system tracks specific events to build a changelog.

    What triggers an event:

    • wiki.created: Triggered when a new wiki is created.
    • page.created: Triggered when a new page is created.
    • source.added: Triggered when a source file is added (occurs even while extraction is pending).
    • page.updated: Triggered when page content changes. Note that updating content to the same value does not trigger an event.
    • source.updated: Triggered when a source is renamed or moved.
    • page.updated (with changes = ['content']): Triggered by content changes.
    • No Event Triggered: Highlight-only edits, status-only changes, parser-only updates, or writes to hidden assets do not generate events.

    Privacy & Retention:

    • Privacy: Activity is private to the wiki owner.
    • Data Minimization: Event metadata never stores document bodies, selected text, comments, or extracted excerpts.
    • Deletion Behavior: Deleting a document leaves a snapshot of its title and path in the changelog. Deleting a knowledge base cascades and deletes all associated events.

  7. How event coalescing works

    master

    To prevent the feed from being noisy due to frequent autosaves, the client performs deterministic coalescing on raw events. This is a pure function applied to loaded events.

    Coalescing Logic:

    1. Sort: By id DESC.
    2. Eligibility: Only page.updated and source.updated events can be coalesced.
    3. Identity: Events must have the same document_id and subject_kind.
    4. Continuity: Events must be consecutive in the raw stream.
    5. Time Gap: The gap between adjacent events must be $\le$ 10 minutes.
    6. Calendar Day: Events must fall on the same browser-local calendar day.

    Non-coalescable events: page.created, source.added, page.archived, page.restored, and page.deleted.

    Result: A group of coalesced events displays the newest event's title/path/version/link, unions the metadata.changes array, retains the oldest and newest timestamps, and reports an event_count.

  8. Component Design: Reading Column and Completion Zone

    master

    The reading column is the primary focus of the interface, designed for high legibility and minimal distraction.

    Reading Column Layout

    • Header: Breadcrumbs, read-time, and a status row.
    • Content: Title, subtitle, and body (max ~68ch width).
    • Right-rail: Contains an 'On this page' Table of Contents (ToC) with an accent for the current item, followed by a 'Your notes' section.
    • Reading Progress: A quiet ink hairline at the very top of the content area, driven by scroll position.

    Completion Zone

    Located under a divider at the end of the content, this region provides:

    • A primary "Mark complete" button.
    • The "Up next" target.
    • A progress indicator (n / total).

  9. How progress and state are managed in LLM Wiki

    master

    LLM Wiki follows a "Claude drives, the UI renders" architecture. State is not stored in browser memory or a centralized database, but is persisted directly to files on disk. This ensures that progress survives across different sessions, days, or machines.

    Key state mechanisms include:

    • Lesson Progress: Persisted via the status field in the markdown frontmatter of individual lessons.
    • Knowledge Base (KB) Metadata: Stores configuration and type information.
    • Quizzes: Stored as .quiz files.

    Because state lives in files, Claude (via MCP) can mutate the content and progress, and the UI simply reflects the current state of those files.

  10. How Course Mode works in LLM Wiki

    master

    Course mode is not a separate application but a specific mode of the existing wiki engine. It uses the same renderer, sidebar, and file structure as the standard wiki, but introduces specific affordances for structured learning: an ordered hierarchy, progress tracking, and soft locks.

    Key Concepts

    • Mode Switching: The engine switches to course mode when a Knowledge Base (KB) has the type: course flag set.
    • Progress Tracking: Progress is derived from the status of individual lessons (completed lessons / total lessons) rather than being stored as a single global value.
    • Soft Locks: Lessons are 'soft locked' until the previous lesson is marked as complete. Locked items are visually dimmed with a lock icon but remain clickable for guidance.
    • Structure: The directory tree on disk defines the course outline. Folder depth determines the role:
      • Depth-0 folders = Parts
      • Deeper folders = Modules
      • Files = Lessons

  11. How LLM Wiki manages files on disk

    master

    LLM Wiki follows a 'filesystem as source of truth' model. It does not move or modify your original files. It only adds two items to your workspace folder:

    1. wiki/: A directory containing ordinary Markdown files generated by Claude. These can be edited manually in any editor.
    2. .llmwiki/: A hidden directory containing the local SQLite search index (index.db) and extracted artifacts (cache/).

    If the index becomes corrupted or needs updating, you can rebuild it using:

    ./llmwiki reindex ~/research

  12. Understand event capture and suppression rules

    master

    Events are triggered by database mutations. Certain actions are explicitly suppressed or handled differently:

    Creation Rules:

    • wiki.created is inserted once during wiki creation. Starter documents are part of this transaction and do not generate separate events.

    Insertion Rules:

    • Documents under /wiki/ generate page.created.
    • Documents elsewhere generate source.added.
    • Suppressed: index.json, log.md, hidden web-clip assets, and any document with metadata.hidden = true or metadata.asset = true generate no events.

    Update Rules:

    • One SQL update creates at most one event.
    • User-visible changes (Triggered): content, title/filename, path, tags, date, metadata.properties, and archived state.
    • Ignored changes (Not Triggered): updated_at, version, status, parser, page_count, indexing hashes, processing errors, highlights, replies, and course progress.
    • Initial Extraction: Transitioning a source from 'pending/processing' to 'ready' does not trigger source.updated; the original source.added event is used.

    Deletion Rules:

    • Visible pages/sources generate page.deleted or source.deleted.
    • Events store snapshots of title, path, document number, and version to ensure history survives hard deletion.