ReqLLM

Model Context Protocol (MCP) can be integrated into a ReqLLM application through two distinct seams. Do not attempt to hide these behind a single generic option, as they have different network paths and security models.

1. Application-side MCP

In this mode, the application (or an optional package) owns the MCP client, including connection setup, discovery, and authentication. An adapter can convert MCP tool descriptions into ReqLLM.Tool structs. When the model calls the tool, the ReqLLM.Tool callback delegates the invocation to the application's MCP client.

{:ok, search_tool} =
  ReqLLM.Tool.new(
    name: "search_docs",
    description: "Search application documentation",
    parameter_schema: [query: [type: :string, required: true]],
    callback: fn %{query: query} -> MyApp.MCP.call_tool("search_docs", %{query: query}) end
  )

2. Provider-hosted MCP

In this mode, the model provider (e.g., OpenAI) connects to the MCP server directly. ReqLLM acts only as a transport for the provider's specific tool configuration. The provider, not ReqLLM, owns the protocol connection, lifecycle, and authentication. ReqLLM only projects exact tool, source, or usage overlaps into canonical values.

The ResponseBuilder behaviour centralizes provider-specific response assembly logic. This ensures that both streaming and non-streaming paths produce identical Response structs and that provider-specific quirks are handled in one place.

Routing Logic

ResponseBuilder.for_model/1 routes to specific builders:

• Anthropic models $\rightarrow$ Anthropic.ResponseBuilder
• Google/Vertex models $\rightarrow$ Google.ResponseBuilder
• OpenAI Responses API models $\rightarrow$ OpenAI.ResponsesAPI.ResponseBuilder
• All others $\rightarrow$ Provider.Defaults.ResponseBuilder

When to implement a custom ResponseBuilder

Most providers can use Provider.Defaults.ResponseBuilder. Implement a custom one if you need to handle:

• Content block requirements: e.g., Anthropic requiring non-empty content blocks.
• Provider-specific metadata: e.g., OpenAI Responses API needing response_id.
• Finish reason detection: e.g., Google needing to detect functionCall.
• Custom tool call handling: Non-standard tool call representations.

Implementation Pattern

You can delegate to the default builder and then apply post-processing:

@impl ReqLLM.Provider.ResponseBuilder
def build_response(chunks, metadata, opts) do
  with {:ok, response} <- DefaultBuilder.build_response(chunks, metadata, opts) do
    response = apply_provider_quirks(response, metadata)
    {:ok, response}
  end
end

defmodule ReqLLM.Providers.Zephyr.ResponseBuilder do
  @moduledoc "Custom ResponseBuilder for Zephyr provider."

  @behaviour ReqLLM.Provider.ResponseBuilder

  alias ReqLLM.Provider.Defaults.ResponseBuilder, as: DefaultBuilder

  @impl true
  def build_response(chunks, metadata, opts) do
    # Delegate to default builder for standard processing
    with {:ok, response} <- DefaultBuilder.build_response(chunks, metadata, opts) do
      # Apply provider-specific post-processing
      response = apply_zephyr_quirks(response, metadata)
      {:ok, response}
    end
  end

  defp apply_zephyr_quirks(response, metadata) do
    # Example: Zephyr includes session_id in metadata
    case metadata[:session_id] do
      nil -> response
      sid -> %{response | provider_meta: Map.put(response.provider_meta, :session_id, sid)}
    end
  end
end

Unlike other providers, Google Gemini supports image editing by including an existing image in the conversation context. You can perform style transfers, add/remove objects, or refine images iteratively.

To edit an image, create a ReqLLM.Context containing a ReqLLM.Message with both a ContentPart.image/2 and a ContentPart.text/1 describing the changes. Pass this context to ReqLLM.generate_image/3 instead of a string prompt.

alias ReqLLM.{Context, Message}
alias ReqLLM.Message.ContentPart

# Load an existing image
{:ok, original_image} = File.read("photo.jpg")

# Create a context with the image and editing instructions
context = Context.new([
  %Message{
    role: :user,
    content: [
      ContentPart.image(original_image, "image/jpeg"),
      ContentPart.text("Add a rainbow in the sky above the mountains")
    ]
  }
])

# Generate the edited image
{:ok, response} = ReqLLM.generate_image(
  "google:gemini-2.5-flash-image",
  context,  # Pass the full context instead of a string
  aspect_ratio: "16:9"
)

edited_image = ReqLLM.Response.image_data(response)
File.write!("photo_with_rainbow.png", edited_image)

Every response from ReqLLM includes a usage field containing normalized token counts and best-effort USD costs.

Token Usage: Includes input_tokens, output_tokens, total_tokens, and individual costs for input/output.

Tool & Image Usage:

• Web Search: If using provider_options: [web_search: %{max_uses: N}], usage metadata includes tool_usage.web_search (count and unit) and specific tool costs.
• Image Generation: response.usage.image_usage provides details on the number of images generated and their size_class.

Note: Pricing is an estimation for observability and is not a guarantee of provider billing accuracy.

# Standard text generation usage
{:ok, response} = ReqLLM.generate_text("anthropic:claude-haiku-4-5", "Hello")
# response.usage contains: %{input_tokens: ..., output_tokens: ..., total_cost: ...}

# Web search usage
{:ok, response} = ReqLLM.generate_text(model, prompt, 
  provider_options: [web_search: %{max_uses: 5}]
)
# response.usage.tool_usage contains: %{web_search: %{count: 2, unit: "call"}}

# Image generation usage
{:ok, response} = ReqLLM.generate_image("openai:gpt-image-1.5", prompt)
# response.usage.image_usage contains: %{generated: %{count: 1, size_class: "1024x1024"}}

ReqLLM 1.x follows semantic versioning to ensure that applications and third-party providers do not need to rewrite integrations during minor updates. All supported surfaces are classified into one of four categories:

• Stable: Documented without experimental or deprecated labels. These contracts may gain additive behavior in minor releases, but incompatible changes (removals, renames, default changes) are reserved for major releases.
• Experimental: Explicitly labeled as such in modules, functions, options, or guides. These may change or be removed in minor releases. It is recommended to isolate experimental usage behind an adapter.
• Deprecated: Functional but marked for removal. Deprecations include an actionable warning, a replacement path, and a minimum of two minor releases of overlap before removal in a major release.
• Internal: Hidden from public documentation or explicitly marked as internal. These are not considered stable contracts.