async-openai Rust Library

The Gemini compatibility implementation supports the following operations:

• Model Management: List available models and retrieve specific model details (e.g., gemini-1.5-flash).
• Chat & Streaming: Perform standard chat completions and real-time streaming of chat messages.
• Multimodal Inputs: Generate images, analyze/understand images, and process/understand audio.
• Advanced Reasoning: Generate structured outputs for complex queries and perform function calling.
• Embeddings: Create embeddings for text or other data types.
• Custom Types: Use the byot (Bring Your Own Type) feature to handle Gemini-specific response structures.

The realtime example demonstrates a bidirectional communication pattern using WebSockets. It follows this workflow:

1. Input: The application reads user input from stdin.
2. Client Events: For every input provided, the client sends two specific events to the server:
   • conversation.item.create: Creates a new item in the conversation with a content type of input_text.
   • response.create: Triggers the model to generate a response.
3. Output: All server events and model responses are streamed to stderr. This separation allows the user to continue typing into stdin without being interrupted by the incoming stream of data.

To terminate the session, type quit and press Enter.

Ollama provides an OpenAI-compatible API, allowing you to use async-openai to interact with local models.

Important Note on API Keys: When configuring your client for Ollama, you must provide an api_key parameter to satisfy the OpenAI API specification. However, Ollama ignores this value; you can provide any string (e.g., "ollama" or "none") to satisfy the requirement.

To customize the HTTP execution path (e.g., for concurrency limits, timeouts, or retries), enable the middleware feature.

Middlewares sit between the async-openai API groups and the concrete HTTP transport. Instead of passing a reqwest::Request through the middleware stack, the library uses an HttpRequestFactory. This is because reqwest::Request is not cloneable when it contains a streaming body, whereas the HttpRequestFactory is cheap to clone and can rebuild a fresh request for each retry attempt.

When using async-openai in a WebAssembly (WASM) environment, the default SimpleRetryPolicy behaves differently because WASM lacks a universal timer runtime.

• Behavior: It retries immediately upon encountering rate limits.
• Limitation: It does not support delayed backoff by default.
• Workaround: If your application requires delayed backoff, you must compose a tower layer that is compatible with your specific WASM runtime.

To write code that works across different OpenAI-compatible providers, use dynamic dispatch by wrapping your configuration in a Box or Arc and using the Config trait. This allows you to pass a Client<Box<dyn Config>> to functions, enabling them to invoke any compatible API.

use async_openai::{Client, config::{Config, OpenAIConfig}};

// Use `Box` or `std::sync::Arc` to wrap the config
let config = Box::new(OpenAIConfig::default()) as Box<dyn Config>;
// create client
let client: Client<Box<dyn Config>> = Client::with_config(config);

// A function can now accept a `&Client<Box<dyn Config>>` parameter
// which can invoke any openai compatible api
fn chat_completion(client: &Client<Box<dyn Config>>) {
    todo!()
}

The Tower WASM Example is a minimal Dioxus web application that demonstrates using async-openai's Response API via the middleware feature in a WebAssembly environment.

To run the development server, you must first install the Dioxus CLI and then use the dx serve command targeting the tower-wasm package.

cargo install dioxus-cli
dx serve -p tower-wasm

The library automatically reads the OpenAI API key from the OPENAI_API_KEY environment variable.

Other supported official environment variables include:

• OPENAI_ADMIN_KEY
• OPENAI_BASE_URL
• OPENAI_ORG_ID
• OPENAI_PROJECT_ID

# On macOS/Linux
export OPENAI_API_KEY='sk-...'

# On Windows Powershell
$Env:OPENAI_API_KEY='sk-...'

You can replace ReqwestService with a custom service for purposes such as logging, metrics, routing, or mocking. Your service must accept an HttpRequestFactory and return a response that can be converted into an OpenAIError.

use async_openai::{Client, config::OpenAIConfig, error::OpenAIError};
use async_openai::middleware::HttpRequestFactory;
use tower::service_fn;

let service = service_fn(|factory: HttpRequestFactory| async move {
    let request = factory.build().await?;

    // here you can inspect, modify, or log the request, route it somewhere else,
    // or return a synthetic response for testing.

    println!("sending {} {}", request.method(), request.url());

    reqwest::Client::new()
        .execute(request)
        .await
        .map_err(OpenAIError::Reqwest)
});

let client = Client::with_config(OpenAIConfig::default())
    .with_http_service(service);

To use webhook handling and signature verification, you must enable the webhook feature in your Cargo.toml dependencies.

[dependencies]
async-openai = { version = "*", features = ["webhook"] }