ollama-rs

Use Ollama::default() to connect to the default local address (localhost:11434), or Ollama::new() to specify a custom base URL and port.

use ollama_rs::Ollama;

// Connect to localhost:11434
let ollama = Ollama::default();

// Connect to a custom address and port
let ollama = Ollama::new("http://localhost".to_string(), 11434);

Add ollama-rs to your Cargo.toml dependencies. You can use the stable version or the master branch for the latest features.

Stable version:

[dependencies]
ollama-rs = "0.3.6"

Latest (master branch):

[dependencies]
ollama-rs = { git = "https://github.com/pepperoni21/ollama-rs.git", branch = "master" }

Note: The master branch may be unstable and contain breaking changes.

To maintain state in a conversation, use the context field from a GenerationResponse. The context is returned as a GenerationContext (a wrapper around Vec<i32>). You can include this context in subsequent GenerationRequest objects to provide the model with the history of the conversation.

// 1. Get response with context
let response = ollama.generate(request).await?;

// 2. Extract context
if let Some(context) = response.context {
    // 3. Use context in the next request to maintain memory
    let next_request = GenerationRequest::new("llama3", "Tell me more.")
        .with_context(context);
    let next_response = ollama.generate(next_request).await?;
}

When an LLM decides to call a tool, it returns a ToolCall containing a ToolCallFunction. The ollama-rs library handles the complexity of parsing these calls.

Because LLM outputs can vary in format (sometimes nesting arguments differently), the library includes a robust parsing mechanism that attempts to extract arguments from several possible JSON structures:

1. The standard ToolCallFunction format (arguments key).
2. A ToolInfo format (function.parameters key).
3. A raw parameter object.

This ensures that even if the model's JSON structure is slightly non-standard, your Tool::call method receives the correctly deserialized Params struct.

To interact with the Ollama service, you must first create an Ollama client instance. You can do this using the Ollama::builder() pattern, which is the recommended approach for configuring the host, port, and HTTP client.

By default, an Ollama instance connects to http://127.0.0.1:11434 using a standard reqwest client.

use ollama_rs::Ollama;

let ollama = Ollama::builder()
    .host("http://localhost")
    .port(11434)
    .build();

The library provides methods to list, inspect, create, copy, and delete models.

• List models: ollama.list_local_models() returns a Vec<LocalModel>.
• Show model info: ollama.show_model_info(model_name) returns a ModelInfo struct.
• Create a model: Use ollama.create_model(CreateModelRequest) to create a model from a base model with a custom system prompt. Returns CreateModelStatus.
• Create a model (Streaming): Requires stream feature. ollama.create_model_stream(CreateModelRequest) returns a CreateModelStatusStream.
• Copy a model: ollama.copy_model(source, destination).
• Delete a model: ollama.delete_model(model_name).

The Coordinator manages tool use (function calling) by automatically executing tools and feeding results back to the LLM. You can add built-in tools or custom tools.

use ollama_rs::coordinator::Coordinator;
use ollama_rs::generation::chat::ChatMessage;
use ollama_rs::generation::tools::implementations::{DDGSearcher, Scraper, Calculator};
use ollama_rs::models::ModelOptions;

let mut history = vec![];

let mut coordinator = Coordinator::new(ollama, "qwen2.5:32b".to_string(), history)
    .options(ModelOptions::default().num_ctx(16384))
    .add_tool(DDGSearcher::new())
    .add_tool(Scraper {})
    .add_tool(Calculator {});

let resp = coordinator
    .chat(vec![ChatMessage::user("What is the current oil price?")])
    .await.unwrap();

println!("{}", resp.message.content);

You can customize generation parameters like temperature, repeat_penalty, top_k, and top_p using ModelOptions passed to the GenerationRequest.

use ollama_rs::generation::completion::GenerationRequest;
use ollama_rs::models::ModelOptions;

let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();

let options = ModelOptions::default()
    .temperature(0.2)
    .repeat_penalty(1.5)
    .top_k(25)
    .top_p(0.25);

let res = ollama.generate(GenerationRequest::new(model, prompt).options(options)).await;

if let Ok(res) = res {
    println!("{}", res.response);
}

You can generate vector embeddings for single strings or batches of strings.

Single string:

use ollama_rs::generation::embeddings::request::GenerateEmbeddingsRequest;

let request = GenerateEmbeddingsRequest::new("llama2:latest".to_string(), "Why is the sky blue?".into());
let res = ollama.generate_embeddings(request).await.unwrap();

Batch strings:

use ollama_rs::generation::embeddings::request::GenerateEmbeddingsRequest;

let request = GenerateEmbeddingsRequest::new("llama2:latest".to_string(), vec!["Why is the sky blue?", "Why is the sky red?"].into());
let res = ollama.generate_embeddings(request).await.unwrap();

Returns a GenerateEmbeddingsResponse containing a vector of floats.

To receive text chunks as they are generated, use ollama.generate_stream(). This requires the stream feature enabled in your Cargo configuration.

use ollama_rs::generation::completion::GenerationRequest;
use tokio::io::{self, AsyncWriteExt};
use tokio_stream::StreamExt;

let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();

let mut stream = ollama.generate_stream(GenerationRequest::new(model, prompt)).await.unwrap();

let mut stdout = io::stdout();
while let Some(res) = stream.next().await {
    let responses = res.unwrap();
    for resp in responses {
        stdout.write_all(resp.response.as_bytes()).await.unwrap();
        stdout.flush().await.unwrap();
    }
}