Ollama

Ollama provides a RESTful API for interacting with large language models. The API supports generating completions (both text and chat), managing models (create, list, show, copy, delete, pull, push), generating embeddings, and monitoring running models.

Note: Official documentation is migrating to https://docs.ollama.com/api.

When building semantic search, retrieval, or RAG (Retrieval-Augmented Generation) pipelines, follow these guidelines:

• Similarity Metric: Use cosine similarity for most semantic search use cases.
• Model Consistency: Always use the same embedding model for both the indexing phase (storing vectors in a database) and the querying phase (searching with a new vector).

If an error occurs while streaming a response (using application/x-ndjson), the error will be emitted as a JSON object containing an error property.

Important: Because the response has already started streaming, the initial HTTP status code of the response will not be changed to reflect the error. You must monitor the individual NDJSON chunks for the presence of an error key to detect mid-stream failures.

{"model":"gemma4","created_at":"2025-10-26T17:21:21.196249Z","response":" Yes","done":false}
{"model":"gemma4","created_at":"2025-10-26T17:21:21.207235Z","response":".","done":false}
{"error":"an error was encountered while running the model"}

Accessing Ollama's API locally via http://localhost:11434 does not require authentication.

Authentication is mandatory only when performing the following actions:

• Running cloud models via ollama.com
• Publishing models
• Downloading private models

When using ollama launch opencode, the command uses an inline configuration for the selected model. This configuration is additive: it does not overwrite your existing ~/.config/opencode/opencode.json file, and existing settings remain in effect.

Important Note: Models that are defined only in your opencode.json file will not appear in the ollama launch model picker.

Ollama attempts to autodetect the best LLM library (e.g., cuda, rocm, cpu_avx2) for your hardware. If autodetection fails or causes crashes, you can force a specific library using the OLLAMA_LLM_LIBRARY environment variable.

Available libraries typically include:

• cpu_avx2: Best performance for compatible CPUs.
• cpu_avx: High compatibility for older CPUs.
• cpu: Most compatible (used for Rosetta emulation on macOS).
• cuda_v11: For NVIDIA GPUs.
• rocm_v5 / rocm_v6: For AMD GPUs.

Example: To force CPU with AVX2 support instead of using a CUDA card:

OLLAMA_LLM_LIBRARY="cpu_avx2" ollama serve

While Ollama provides broad compatibility with the Anthropic Messages API, some features are not supported or behave differently:

Not Supported

• /v1/messages/count_tokens endpoint
• tool_choice (forcing specific tool use)
• metadata (request metadata like user_id)
• Prompt caching (cache_control blocks)
• Batches API (/v1/messages/batches)
• Citations (citations content blocks)
• PDF support (document content blocks)

Behavioral Differences

• API Key: Accepted but not validated.
• Headers: anthropic-version is accepted but not used.
• Token Counts: Approximations based on the underlying model's tokenizer.
• Images: Base64 images are supported; URL-based images are not.
• Extended Thinking: Basic support; budget_tokens is accepted but not enforced.

Ollama uses a templating engine based on Go's built-in templating engine to construct prompts for Large Language Models (LLMs). A template consists of three components:

1. Layout: The overall structure of the prompt.
2. Variables: Placeholders for dynamic data (e.g., .Prompt, .Messages) that are replaced during rendering.
3. Functions: Logic or actions used to manipulate content (e.g., range to iterate over messages).

Using templates allows you to define specific markers (like <|im_start|> or [INST]) required by instruction or chat models, rather than sending user input verbatim.

{{- range .Messages }}
{{ .Role }}: {{ .Content }}
{{- end }}

OpenCode requires a model with a context length of 64k or higher to effectively process your repository.

You can use:

• Cloud models: Larger models accessed without local download.
• Local models: Models downloaded via Ollama, provided you set a context window of 64k+.