DOCSEARCHbyAlgolia
Submit Docs

OpenJarvis Documentation

repository·main·Indexed 27 days ago

https://github.com/open-jarvis/openjarvis

A local-first personal AI framework and modular backend for running agents on-device. OpenJarvis provides composable intelligence primitives, shared agent building blocks, and evaluation metrics for energy, latency, and cost. It features built-in agents for deep research, code assistance, and scheduled monitoring, along with a skill system following the agentskills.io standard. The framework supports native Windows installation, Docker Compose deployment, and integration with local LLMs via Ollama.

Tokens
149.5K
Snippets
358
Records
798
Agent score
93%

What's inside OpenJarvis

  1. Overview of LLM-Guided Spec Search

    main

    LLM-guided spec search (Frontier-Driven Harness Learning) uses a high-capability 'teacher' model to optimize the configuration of a local 'student' model. Instead of fine-tuning weights, it optimizes the surrounding environment: prompts, routing, agent classes, and tool descriptions.

    The Four-Phase Loop:

    1. Diagnose: TeacherAgent analyzes traces to identify failure clusters.
    2. Plan: LearningPlanner creates a LearningPlan with risk tiers.
    3. Execute: EditApplier applies changes, validated by BenchmarkGate.
    4. Record: SessionStore persists the session and artifacts.

    Risk Tier System for Edits:

    • auto: Model routing/params, tool configuration, agent params. (Applied if gate passes).
    • review: System prompt edits, agent class, few-shot exemplars. (Requires user approval).
    • manual: LoRA fine-tuning. (Never auto-applied).

  2. Overview of vLLM-Pearl Mining Integration

    main
    The openjarvis.mining subsystem allows users with H100/H200 hardware running vLLM to solo-mine the Pearl PoUW chain using their existing LLM inference. It uses a MiningProvider abstraction to support various hardware/engines (e.g., Apple Silicon, AMD, Ollama) and employs a runtime sidecar at ~/.openjarvis/runtime/mining.json to manage the mining lifecycle independently from the vLLM engine.

  3. Integrate with supported messaging clients

    main

    OpenJarvis supports several bidirectional messaging interfaces for interacting with agents:

    • iMessage + SMS: Handled via SendBlue. Supports bidirectional communication, auto-detection of protocol, thread replies, and progress updates.
    • Slack: Handled via Socket Mode (slack-bolt). Supports bidirectional DMs, thread replies, Slack formatting, and progress updates.
    • Desktop/Browser: Provides an interactive interface with real-time streaming, tool progress visibility, and a telemetry footer.

  4. Understand the Pearl mining engine handoff

    main

    When mining is active, OpenJarvis automatically integrates the mining workload into its engine discovery process:

    1. jarvis mine start launches the container and writes a sidecar file at ~/.openjarvis/runtime/mining.json.
    2. The engine discovery module (engine/_discovery.py) detects this sidecar on every lookup.
    3. A specialized engine instance named vllm-pearl-mining is automatically registered, pointing to the vllm_endpoint defined in the sidecar.
    4. Commands like jarvis ask or SDK calls are transparently routed to this mining endpoint. Your standard inference tasks become the mining work.

  5. Understand the Skills Architecture

    main

    Skills in OpenJarvis act as a cross-cutting orchestration layer that connects tools, agents, memory, and learning into reusable workflows. They sit above the core primitives and are managed by the SkillManager to provide agents with structured capabilities.

    Key architectural roles include:

    • SkillManager: The central coordinator for discovery, tool wrapping, and prompt injection.
    • SkillTool: An adapter that makes skills appear as standard BaseTool objects to agents.
    • SkillExecutor: A sequential pipeline runner that handles tool delegation and template rendering.
    • SkillOverlay: A sidecar mechanism used to store optimized descriptions and few-shot examples generated by the SkillOptimizer.

  6. Understand the OpenJarvis Source Directory Layout

    main

    OpenJarvis is organized into several functional primitives located in src/openjarvis/. Understanding this layout helps in locating specific components for extension or integration:

    • core/: Shared infrastructure including the EventBus, RegistryBase, and configuration loading.
    • intelligence/: Model definitions and the model_catalog.
    • engine/: Inference runtime backends (e.g., Ollama, Cloud backends like OpenAI/Anthropic, and OpenAI-compatible engines like vLLM/llama.cpp).
    • agents/: Pluggable agent logic (e.g., SimpleAgent, OrchestratorAgent, NativeReActAgent, OpenHandsAgent).
    • sandbox/: Isolated execution environments using ContainerRunner (Docker/Podman).
    • memory/: Persistent storage backends (e.g., SQLite, FAISS, ColBERT, BM25) and document ingestion tools.
    • learning/: Router policies and reward functions that optimize model selection.
    • traces/: Interaction recording via TraceStore and TraceCollector.
    • tools/: Pluggable tools like CalculatorTool, RetrievalTool, and FileReadTool.
    • telemetry/: Inference metrics and statistics.
    • server/: FastAPI-based OpenAI-compatible HTTP API.
    • security/: Guardrails, scanners (Secret/PII), and audit logging.
    • channels/: Messaging integrations (e.g., WhatsApp via WhatsAppBaileysChannel).
    • scheduler/: Task scheduling (cron/interval) and MCP scheduler tools.
    • cli/: Command-line interface commands (ask, serve).
    • mcp/: Model Context Protocol layer.

  7. Understand OpenJarvis telemetry privacy and data collection

    main

    OpenJarvis collects anonymous usage telemetry to improve the product.

    Privacy Guarantees:

    • No Chat Content: Prompts, model outputs, system messages, and tool arguments are never collected.
    • No PII: Emails, names, phone numbers, addresses, and IP addresses are not collected.
    • No Sensitive Paths: File paths matching home directories or local user paths are redacted.
    • No Credentials: API keys, OAuth tokens, and passwords are matched and dropped.
    • Anonymity: A single UUID v4 is stored at ~/.openjarvis/anon_id to track usage without identifying the user.

    Data Retention:

    • Data is stored on a PostHog instance and is automatically deleted after 365 days.

    Redaction Layers:

    1. src/openjarvis/analytics/redaction.py: Value-level pattern matching using 20+ regexes.
    2. src/openjarvis/analytics/events.py: Structural allowlist (event name + property name + type validator).

  8. Hardware Auto-Detection and Engine Recommendations

    main

    When running jarvis init, OpenJarvis automatically detects hardware to recommend an optimal inference engine.

    Detection Logic

    • NVIDIA GPU: Detected via nvidia-smi.
    • AMD GPU: Detected via rocm-smi.
    • Apple Silicon: Detected via system_profiler on macOS.
    • CPU/RAM: Detected via /proc/cpuinfo, /proc/meminfo (Linux), or sysctl (macOS).

    Engine Recommendations

    HardwareRecommended EngineReason
    No GPUllamacppEfficient CPU inference with GGUF quantized models
    Apple SiliconollamaNative Metal acceleration, easy model management
    NVIDIA consumer GPU (RTX 3090, 4090, etc.)ollamaSimple setup, good performance for single-user
    NVIDIA datacenter GPU (A100, H100, etc.)vllmHigh-throughput batched serving, continuous batching
    AMD GPUvllmROCm support via vLLM

  9. Quickstart Simple Chat setup

    main

    Set up a lightweight conversational AI using Ollama and OpenJarvis for local, tool-free chat.

    1. Install Ollama and pull a model:
      ollama pull qwen3.5:4b
    2. Install and initialize OpenJarvis:
      git clone https://github.com/open-jarvis/OpenJarvis.git
cd OpenJarvis
uv sync
jarvis init --preset chat-simple
    3. Ask a question:
      jarvis ask "What is quantum computing?"
    # Install Ollama and pull a model
ollama pull qwen3.5:4b

# Install and initialize OpenJarvis
git clone https://github.com/open-jarvis/OpenJarvis.git
cd OpenJarvis
uv sync
jarvis init --preset chat-simple

# Ask a question
jarvis ask "What is quantum computing?"

  10. Quickstart the OpenJarvis Browser App

    main

    To run the full chat UI locally, clone the repository and run the quickstart script. This process installs dependencies, starts Ollama with a local model, launches both the backend and frontend, and opens the interface in your browser at http://localhost:5173.

    git clone https://github.com/open-jarvis/OpenJarvis.git
cd OpenJarvis
./scripts/quickstart.sh

  11. Connect Dropbox as a Data Source

    main

    OpenJarvis indexes files and documents from your Dropbox.

    Setup

    1. Create a Dropbox app: Go to the Dropbox App Console and choose "Scoped access" -> "Full Dropbox".
    2. Set Permissions: Under the Permissions tab, enable files.metadata.read and files.content.read.
    3. Generate Token: Go to the Settings tab and generate an access token.

    Connecting in OpenJarvis

    • Desktop/Browser: Go to Agents -> Channels tab -> Dropbox and paste the token.
    • CLI: uv run jarvis connect dropbox

    Troubleshooting

    • Token Expiration: Dropbox short-lived tokens expire after 4 hours. If you get an "Invalid access token" error, generate a new one.
    uv run jarvis connect dropbox