DOCSEARCHbyAlgolia
Submit Docs

genai Rust Library

repository·main·Indexed 21 days ago

https://github.com/jeremychone/rust-genai

A native-protocol multi-AI provider library for Rust providing a single ergonomic API to access over 200 LLM models from 26+ providers, including OpenAI, Anthropic, Gemini, Ollama, AWS Bedrock, Vertex, Groq, DeepSeek, Kimi, and GLM. It supports synchronous and streaming chat completions, multimodal image analysis, and unified generation parameters like temperature and max_tokens.

Tokens
30.8K
Snippets
75
Records
133
Agent score
69%

What's inside genai

  1. Understand the genai crate structure

    main

    The genai crate is organized into several key modules:

    • genai::adapter: Contains AdapterKind and logic for adapter dispatch.
    • genai::chat: Handles chat interactions (ChatRequest, ChatResponse, ChatStream, Tools, etc.).
    • genai::chat::printer: Provides the print_chat_stream utility for displaying streams.
    • genai::embed: Handles embedding requests and responses (EmbedRequest, EmbedResponse).
    • genai::resolver: Manages authentication and routing (AuthData, AuthResolver, Endpoint, ProviderConfig, ModelMapper, ServiceTargetResolver, Headers).
    • genai::webc: Contains webc::Error and internal web client logic.
    • Flattened Exports: The following are available directly from the crate root:
      • Client, ClientBuilder, ClientConfig
      • ModelIden, ModelName
      • ModelSpec, ServiceTarget, Headers, WebConfig
      • Error, Result, BoxError

  2. Use Bound Adapters to constrain model routing

    main

    A Bound Adapter is an optional constraint you can set on a Client or ClientBuilder.

    When a bound adapter is set, bare model names (e.g., just "gpt-4") are forced to route through that specific adapter instead of using the library's default heuristic inference.

    Note: If you provide a ModelIden that explicitly mismatches the bound adapter's namespace, or if the namespaces do not align, the library will return an AdapterKindMismatch error.

  3. Understand the role of the adapter module

    main

    The adapter module serves as the translation and dispatch layer between the generic GenAI client logic and specific AI providers (e.g., OpenAI, Gemini, Anthropic, Groq, DeepSeek).

    Its primary responsibilities are:

    • Translation: Converting generic requests (like ChatRequest and EmbedRequest) into provider-specific HTTP request data.
    • Normalization: Converting provider-specific web responses back into generic GenAI response structures.
    • Dispatching: Routing requests to the correct provider implementation via the AdapterDispatcher.

  4. How model to adapter resolution works

    main

    By default, genai resolves the AdapterKind (the AI provider) based on the prefix of the model name provided.

    Common resolution rules:

    • OpenAI: gpt-*, o1-*, o3-*, o4-*, chatgpt-*, codex-*
    • Anthropic: claude-*
    • Gemini: gemini-*
    • xAI: grok-*
    • DeepSeek: deepseek-*
    • Moonshot: moonshot-* (moonshot.ai)
    • Kimi: kimi* (moonshot.ai)
    • Zai: glm-*
    • Cohere: command-*, embed-*
    • Ollama: Fallback for any other names, defaulting to local Ollama.

    If a model name does not match these patterns, it defaults to the Ollama adapter.

  5. How streaming responses are normalized via InterStream

    main

    To handle the variance in streaming protocols across different providers, the library uses an intermediary layer.

    When a provider streams data, it is first converted into internal types: InterStreamEvent and InterStreamEnd. This ensures that complex data—such as final usage statistics, reasoning content, or aggregated tool calls—is correctly captured and normalized before being converted into the public ChatStreamResponse events used by the end-user.

  6. Understand the internal webc module architecture

    main

    The webc module is a low-level internal abstraction layer built on reqwest and tokio. It is designed to shield the rest of the genai library from direct reqwest dependencies and to handle the complexities of different AI provider communication protocols.

    Core Components

    • WebClient: A wrapper around reqwest::Client used for standard, non-streaming HTTP GET/POST requests. It assumes responses are JSON and parses them into serde_json::Value via WebResponse.
    • WebStream: A custom futures::Stream implementation. Unlike standard SSE (Server-Sent Events) clients, WebStream is designed to handle non-standard streaming protocols used by providers like Cohere or Gemini.
    • StreamMode: A configuration used by WebStream to define how stream chunks are parsed:
      • Delimiter-based: Splitting chunks by a fixed delimiter (used by providers like Cohere).
      • Pretty JSON Array: Specialized handling for JSON array chunking (used by providers like Gemini).

    Design Patterns

    • Internal-First: Most types (WebClient, WebResponse, WebStream) are pub(crate), meaning they are intended for use by library adapters rather than end-users directly.
    • Generic JSON Handling: Non-streaming responses are immediately parsed into serde_json::Value to allow adapters to perform secondary deserialization into specific structures.

  7. How the adapter dispatching mechanism works

    main

    The library uses a stateless and static dispatch pattern to route requests efficiently:

    1. Resolution: The system determines the correct AdapterKind for a request. This can be done via the default AdapterKind::from_model mapping (which uses model name prefixes/keywords) or by overriding it with a custom ServiceTargetResolver.
    2. Dispatching: The AdapterDispatcher receives the call and routes it to the appropriate concrete implementation of the Adapter trait.
    3. Execution: The concrete adapter (found in the adapters/ submodule) performs the specific request/response translation and executes the web communication.

    Because adapters are designed to be stateless, all methods in the Adapter trait are associated functions (static), which minimizes runtime overhead.

  8. How model resolution works in genai

    main

    The library uses a hierarchy of abstractions to identify and route LLM calls:

    1. ModelIden: The most specific identifier, combining an AdapterKind (the provider) and a ModelName.
    2. ModelSpec: A flexible way to specify a model at three different resolution levels:
      • Name: A bare model name (e.g., "gpt-4").
      • Iden: A specific ModelIden.
      • Target: A fully resolved ServiceTarget.
    3. ServiceTarget: The final, fully resolved call target, consisting of a ModelIden, an Endpoint, and AuthData.
    4. Resolvers: User-provided hooks that allow you to customize how models are mapped, how authentication is handled, and how service endpoints are determined.

  9. Resolve models using `ModelSpec`

    main

    Since v0.6.0, all execution methods accept impl Into<ModelSpec>. ModelSpec defines how a model is identified and resolved.

    Variants:

    • ModelSpec::Name(ModelName): A model name string. The adapter kind is inferred. If the client has a bound adapter, bare names route through it.
    • ModelSpec::Iden(ModelIden): An explicit AdapterKind and ModelName. Skips inference but still resolves auth and endpoints. If the client is bound to an adapter, the ModelIden.adapter_kind must match.
    • ModelSpec::Target(ServiceTarget): A fully resolved target. This bypasses model mapping and auth resolution, running only the service target resolver.

    Constructors:

    • ModelSpec::from_name(name)
    • ModelSpec::from_static_name(name)
    • ModelSpec::from_iden(model_iden)
    • ModelSpec::from_target(target)

    Into<ModelSpec> is implemented for: &str, &&str, String, &String, ModelName, &ModelName, ModelIden, &ModelIden, and ServiceTarget.

  10. Understand the Model Resolution Order

    main

    When a model request is made, genai resolves the target through a specific sequence of components. Understanding this order is critical for configuring proxies, gateways, or bound adapter clients:

    1. ModelMapper: Maps a ModelIden to another before execution. This is the first step.
    2. AuthResolver: Resolves authentication data (API keys, etc.). This runs after model mapping but before the service target is finalized.
    3. Adapter Default Endpoint: The default endpoint provided by the specific adapter.
    4. ServiceTargetResolver: The final step. It can override everything (endpoint, auth, and model identity) to provide a final call target.

    Note: If you use a bound adapter client, the ModelMapper step (via the bound adapter) happens before the AuthResolver runs.

  11. Understand OpenTelemetry GenAI instrumentation limitations

    main

    When using the otel feature, be aware of the following:

    • Spans only: Metric instruments (like gen_ai.client.token.usage) are not emitted directly; instead, this data is available as span attributes. Metrics must be derived from spans by your backend.
    • JSON-encoded arrays: Because tracing lacks an array field type, attributes like gen_ai.response.finish_reasons and gen_ai.request.stop_sequences are encoded as JSON strings.
    • Events as Spans: Spec 'events' (e.g., evaluation results) are implemented as zero-duration spans.
    • No Cost: Monetary cost is not recorded; this is left to downstream tooling.
    • Evolving Spec: The GenAI semantic conventions are currently in Development status; attribute names may change.

  12. Use Namespacing to Target Specific Adapters

    main

    You can force a specific adapter by using the namespace::model_name syntax. This is particularly useful when using providers that might otherwise be misidentified or when using specialized endpoints.

    Common Namespaces:

    • open_router::model_name -> OpenRouter
    • vertex::model_name -> Google Vertex AI
    • groq::model_name -> Groq (required for direct targeting)
    • bedrock_api::model_name -> AWS Bedrock Converse API (Bearer auth)
    • bedrock_sigv4::model_name -> AWS Bedrock Converse API (SigV4 auth)
    • aliyun::model_name -> Aliyun OpenAI-compatible service
    • baidu::model_name -> Baidu
    • aihubmix::model_name -> AIHubMix
    • moonshot::model_name -> Moonshot AI
    • ollama_cloud::model_name -> Ollama Cloud
    • opencode_go::model_name -> OpenCode Go

    Bound Adapter Behavior: If you have a client bound to a specific adapter (using ClientBuilder::with_adapter_kind), the model name must match that adapter. If you provide a namespaced name that targets a different adapter, an AdapterKindMismatch error will be returned.