Harbor Documentation

Harbor integrates with various backend services to provide LLM inference capabilities. These backends can range from local runners to large-scale inference engines. Key categories include:

• General LLM Inference: Services like Ollama, vLLM, llama.cpp, mistral.rs, Aphrodite, and Text Generation Inference.
• Apple Silicon Optimized: MLX and oMLX provide host-native inference for Apple hardware via OpenAI-compatible proxies.
• Audio (TTS/STT): Speaches (supports both TTS and STT), openedai-speech (uses Coqui AI's xtts_v2 or piper), and Parler.
• Specialized/Hardware Optimized: Lemonade (optimized for AMD), AirLLM (70B inference on single 4GB GPU), and KTransformers.
• Multi-modal/Tooling: SGLang (LLM and Vision), Needle (function-calling), and Lemonade (LLM, Speech, and Image).

Harbor provides a pre-wired environment for local AI development. Key features include:

• Pre-connected Backends: All backends are automatically connected to Open WebUI.
• Service Orchestration: Handles Docker Compose, configuration, and connectivity for multiple services.
• Extensible Stack: Supports adding web search (SearXNG), voice chat (Speaches), and image generation (ComfyUI).
• Automated Setup: Eliminates manual configuration for complex local LLM stacks.

boost is an optimizing LLM proxy that provides OpenAI-compatible, Anthropic-compatible, and OpenAI Responses API interfaces. It acts as a drop-in proxy to combine multiple LLM backends behind a single endpoint with a single API key.

Key capabilities include:

• OpenAI Compatibility: Works with most LLM providers and clients.
• Anthropic Compatibility: Automatically converts Anthropic Messages API requests to OpenAI format, routes them through the Boost pipeline (modules, workflows, model routing), and converts responses back to Anthropic format.
• OpenAI Responses API Support: Supports the /v1/responses endpoint, converting these requests to Chat Completions internally.
• Modules: Allows running custom code (e.g., for web research, reasoning, or style) during the chat completion process.

The faster-whisper-server is an OpenAI API-compatible Speech-to-Text (STT) server. It handles the stt service type and exposes the /v1/audio/transcriptions endpoint.

Note: Harbor also supports speaches, a newer OpenAI-compatible STT/TTS server that supersedes faster-whisper-server. For new setups, consider using speaches instead.

Harbor includes built-in specialized services:

• Harbor Boost: An LLM optimizer proxy.
• Harbor Bench: A built-in LLM benchmarking service.

Harbor provides a wide ecosystem of integrated services categorized into three main types:

1. UIs (Frontends): User interfaces for interacting with models (e.g., Open WebUI, LibreChat, Lobe Chat, SillyTavern).
2. Backends: Inference engines and model runners (e.g., Ollama, llama.cpp, vLLM, Aphrodite, MLX).
3. Satellites: Specialized tools, agents, and integrations (e.g., Aider, Dify, Open Interpreter, SearXNG, n8n).

For detailed information on how to use or configure any specific service, refer to the official services documentation.

If the ollama service is enabled, Harbor automatically injects the KH_OLLAMA_URL and local model settings into Kotaemon. To run both services together, include both in your harbor up command.

harbor up ollama kotaemon --open

To download gated models or push fine-tuned models to the Hugging Face Hub, set your token using the Harbor CLI. This token is forwarded to the container as HF_TOKEN.

harbor config set hf.token "hf_your_token_here"

Model packs allow you to group specific models and parameters (temperature, Top P) for different roles within Plandex (e.g., planner, builder, summarizer).

1. Create a pack: Use harbor plandex model-packs create to define a new pack and assign models/parameters to roles like planner, summarizer, builder, names, commit-messages, auto-continue, verifier, and auto-fix.
2. Set as default: Use harbor pdx set-model default to apply a model pack to all new plans going forward.