Jarvis AI Voice Assistant

Jarvis is a 100% private, local AI voice assistant designed for natural, conversational interaction. Key features include:

• Conversational Awareness: Understands context in ongoing discussions. You can say "Jarvis, what do you think?" and it will understand the preceding conversation.
• Unlimited Memory: Maintains a knowledge graph of your conversations and preferences. Includes a Memory Viewer GUI.
• Natural Voice Interaction: Supports natural wake word placement (say "Jarvis" anywhere in a sentence) and echo detection to ignore its own speech.
• Dictation Mode: A free, offline alternative to WisprFlow. Hold a hotkey, speak, and release to paste text into any application.
• Extensible via MCP: Integrates with the Model Context Protocol (MCP) to connect to tools like GitHub, Slack, Home Assistant, and more.
• Smart Tool Selection: Uses embedding-based relevance filtering to select the correct tools from your collection without performance degradation.
• Built-in Tools: Includes web search (DuckDuckGo → Brave → Wikipedia), screenshot OCR, weather, file access, and nutrition tracking.

Overview

Note: This component is deprecated. The task-list planner (planner.spec.md) replaces its per-turn correction role. This information is preserved for reference only.

In the legacy system, the Evaluator was a lightweight LLM used after each agentic-loop turn that produced natural-language content (instead of a tool call). Its purpose was to decide whether the loop should terminate (the agent has finished) or continue (the agent replied in prose when it should have used a tool).

Decision Logic

The evaluator uses a binary decision model:

• Terminal: The agent has satisfied the user request, delivered a real answer, or truthfully stated it cannot perform the action. Both 'satisfied' and 'needs_user_input' states are treated as terminal.
• Continue: The user expressed a clear action that a tool in the allow-list could perform, but the agent responded with prose (suggestions, offers, or descriptions) instead of invoking the tool.

It also acts as a safety mechanism for garbled output. If the agent produces raw tool-protocol markers, truncated JSON, or special sentinel tokens (like <unused88>), the evaluator attempts to salvage the intent or nudges the agent to produce a natural-language reply.

Jarvis operates in three distinct listening modes that transition based on user interaction and system state:

1. Wake Word Mode (Default): The system listens for the specific wake word. When detected, it triggers the Intent Judge.
2. Hot Window Mode: Triggered after TTS (Text-to-Speech) finishes. The system stays in a high-sensitivity listening state for a set duration (hot_window_seconds) to allow for immediate follow-up questions without repeating the wake word.
3. During TTS: While Jarvis is speaking, the system monitors for a 'Stop' command to interrupt the playback.

State Transition Logic

• WakeWord $\rightarrow$ IntentJudge: Triggered when a wake word is detected via text.
• IntentJudge $\rightarrow$ DuringTTS: Triggered when a query is successfully dispatched and TTS starts.
• DuringTTS $\rightarrow$ HotWindow: Occurs when TTS ends, subject to echo_tolerance.
• HotWindow $\rightarrow$ IntentJudge: Triggered if speech is detected during the hot window.
• HotWindow $\rightarrow$ WakeWord: Occurs if the hot window timer expires without speech detection.

stateDiagram-v2
    direction LR
    [*] --> WakeWord: System Starts

    WakeWord: Listening for Wake Word
    HotWindow: Listening for Follow-up
    DuringTTS: TTS Playing

    WakeWord --> IntentJudge: Wake detected (text-based)
    IntentJudge --> DuringTTS: Query dispatched, TTS starts
    IntentJudge --> WakeWord: Not directed / no query
    DuringTTS --> HotWindow: TTS ends + echo_tolerance
    HotWindow --> IntentJudge: Speech detected
    HotWindow --> WakeWord: Timer expires
    DuringTTS --> WakeWord: Stop command detected

The startup sequence is designed to be resilient, especially when local dependencies like Ollama are involved.

The sequence follows these logic gates:

1. Single Instance Check: Prevents multiple copies. If an instance is already running, it offers to terminate the old one.
2. Splash Screen: An animated loading screen is shown immediately.
3. Setup Check: If no setup has been completed, the Setup Wizard is launched.
4. Ollama Gating:
   • If using an OpenAI-compatible provider (remote chat/embeddings), Ollama checks are skipped.
   • If Ollama is used, the app attempts to auto-start the Ollama server (up to 15s wait).
   • If Ollama fails to start or models are missing, the Setup Wizard is presented for diagnosis.
5. Reachability Check: For OpenAI-compatible setups, the app performs a one-off check (GET /v1/models) to ensure the remote server is reachable, showing a warning if it fails.
6. Initialization: Once dependencies are verified, the system tray is initialized, the daemon thread starts, and the splash screen closes.

The Jarvis desktop app manages the daemon in two different modes depending on whether you are running a production build or a development environment:

1. Bundled Mode (Production): The daemon runs within the same process using a QThread. This allows for direct callback registration via set_diary_update_callbacks() for features like diary updates.
2. Subprocess Mode (Development): The daemon runs as a separate process. Communication (IPC) is handled via stdout. The desktop app intercepts log lines to drive the UI.

Daemon Callbacks (Bundled Mode)

When the daemon is running in a QThread, the DiaryUpdateDialog receives the following signals:

• on_chunks: A list of conversation chunks being summarized.
• on_token: Streaming tokens as the diary is generated.
• on_status: Status messages (e.g., "Writing diary entry...").
• on_complete: A signal indicating completion (success or failure).

IPC via stdout (Subprocess Mode)

In development, the daemon emits JSON events prefixed with __DIARY__: to communicate with the UI. For example: __DIARY__:{"type":"token","data":"Hello"}

Jarvis uses a "rewrite-on-write" strategy to keep the Knowledge Graph clean via merge_node_data(). This prevents the graph from becoming cluttered with redundant or contradictory information.

Consolidation Rules: When new facts are added to an existing node, the system applies these rules:

1. Contradictions: If a new fact contradicts an old one, the old version is dropped.
2. Near-Duplicates: Similar phrasings are collapsed into a single entry.
3. Patterns: Repeated daily activities are consolidated into general patterns.
4. Pruning: Common-knowledge facts are removed.

Safety Mechanisms:

• Hallucination Guard: If the LLM produces a rewrite that is significantly larger than the input (len(existing) + len(new) + 2), the rewrite is rejected, and the system falls back to a simple append_to_node to prevent data loss or bloat.
• Maintenance: Users can trigger a full consolidation of all populated nodes using the 🧹 button in the memory viewer (invokes consolidate_all_populated_nodes).