obsidian-llm-wiki

The pipeline transforms raw notes into a structured wiki through three distinct stages:

1. Ingest (olw ingest or olw run): Uses a Fast LLM (3B–8B) to read notes, extract concept names, preserve named references as 'knowledge item candidates', and write quality scores/summaries to state.db. It creates source summary pages in wiki/sources/.
2. Compile (olw compile): Uses a Heavy LLM (7B–14B) to gather all source notes for a concept, inject previous rejection feedback, and write a wiki article with [[wikilinks]]. Articles are placed in wiki/.drafts/ for review.
3. Review (olw review or olw approve): An interactive menu to approve, reject, diff, or edit drafts. Approved articles are published to wiki/, annotations are stripped, wiki/index.md is updated, and changes are Git committed.

Drafts with low confidence or sparse sources include HTML comments that are visible in the Obsidian editor but hidden in preview mode. These serve as warnings for the human reviewer.

Examples:

• <!-- olw-auto: low-confidence (0.32) — verify before publishing -->
• <!-- olw-auto: single-source — cross-reference recommended -->

These annotations are automatically stripped when a draft is approved via olw review.

The olw project organizes your vault into specific directories. Crucially, the raw/ directory is immutable and is never modified by olw.

• raw/: Your original notes.
• wiki/: The generated knowledge base.
  • wiki/sources/: Auto-generated source summary pages.
  • wiki/.drafts/: Pending articles awaiting human review.
  • wiki/index.md: The auto-generated navigation and routing layer.
  • wiki/queries/: Saved Q&A answers.
  • wiki/synthesis/: Saved synthesis articles.
• wiki.toml: Global configuration.
• vault-schema.md: LLM context defining vault conventions.
• .olw/: Internal state.
  • state.db: SQLite database containing all metadata, concepts, and rejections.
  • compare/: Comparison reports and artifacts.
  • pipeline.lock: Advisory lock file.

On iOS, you can save web content to the raw/ folder using the Obsidian share sheet. To ensure files land in the correct location, configure Obsidian settings:

1. Go to Core plugins.
2. Select Note composer.
3. Set the Default location to raw/.

To process notes automatically as you write them, run the file watcher. It monitors the raw/ directory and triggers incremental ingestion and compilation for any new or changed files.

olw watch

To validate how different models (especially smaller local models) handle citation markers, you can run the provided smoke test scripts. You must set the INLINE_SOURCE_CITATIONS=1 environment variable to enable the feature during the test run.

Use the following commands to run standard smoke tests and comparison tests using LM Studio as a provider:

# Run standard smoke test
PROVIDER=lm_studio FAST_MODEL=gemma4:e4b HEAVY_MODEL=gemma4:e4b INLINE_SOURCE_CITATIONS=1 bash scripts/smoke_test.sh

# Run comparison smoke test
PROVIDER=lm_studio FAST_MODEL=gemma4:e4b HEAVY_MODEL=gemma4:e4b INLINE_SOURCE_CITATIONS=1 bash scripts/compare_smoke.sh

To create a new Obsidian vault structure compatible with olw, use the init command. This creates the necessary folder structure and a wiki.toml file pre-filled with your setup choices.

olw init ~/my-wiki

To automate the ingestion of web articles, configure the official Obsidian Web Clipper to save files directly into your vault's raw/ folder. This allows olw to process the content automatically.

1. Install the Obsidian Web Clipper.
2. In the extension settings, navigate to Template.
3. Set the Save location to raw/ within your vault.
4. Apply the following template to ensure metadata is correctly captured for the pipeline:

---
source_url: {{url}}
source_title: {{title}}
captured: {{date}}
---

# {{title}}

{{content}}

Once clipped, you can manually run olw ingest raw/{{filename}}.md or use olw watch to automate the process.

Context window settings in wiki.toml impact how much data the models process and the resulting article quality. Adjust these based on your available VRAM.

heavy_ctx (Article Generation)

Controls how much source material the heavy model reads. The source budget is approximately heavy_ctx / 2 characters.

• 8 GB VRAM: 8192 (Short articles)
• 16 GB VRAM: 32768 (Default)
• 32 GB+ VRAM: 65536 (Rich multi-source articles)

fast_ctx (Ingest Analysis)

Controls ingest analysis. Notes longer than fast_ctx / 2 characters are automatically split into chunks and analyzed sequentially.

• 8 GB VRAM: 8192 (~4K chars)
• 16 GB VRAM: 16384 (~8K chars - Default)
• 32 GB+ VRAM: 32768 (~16K chars)

After editing wiki.toml, run olw compile --force to regenerate articles with the new budget.

The pipeline processes raw notes into wiki articles. You can run the entire process at once or step-by-step.

Full pipeline: Runs ingest, compile, lint, and optional auto-approval.

olw run

Step-by-step:

1. Ingest: Read new files from raw/.

   olw ingest --all

2. Compile: Extract concepts and create/update wiki articles.

   olw compile

3. Review: Interactively review generated drafts.

   olw review

Note: If you haven't set a default vault in olw setup, use the --vault flag or set the OLW_VAULT environment variable.

For vaults with many long notes (>8K chars), you can enable parallel chunk analysis to reduce wall time.

1. In wiki.toml, set ingest_parallel = true under [pipeline].
2. Before starting Ollama, set the OLLAMA_NUM_PARALLEL environment variable in your shell.

Example for 16 GB VRAM:

OLLAMA_NUM_PARALLEL=4 ollama serve

In wiki.toml:

[pipeline]
ingest_parallel = true

When you run olw compile, generated articles are initially placed in the wiki/.drafts/ directory. Because Obsidian hides dotfolders by default, these articles will not appear in your graph immediately. To make them visible in the wiki/ folder, you must review or approve them.

Use olw review for an interactive review process, or olw approve --all to move all drafts to the main wiki directory automatically.

olw review        # interactive review
# or
olw approve --all