RubyLLM Documentation

If you are using a new model release, a custom fine-tune, or a private deployment that is not in the RubyLLM registry, use assume_model_exists: true.

Note: When using this flag, you must specify a provider:, as RubyLLM cannot infer the provider from the registry. RubyLLM will build a synthetic model object with all capabilities enabled (function_calling, streaming, vision, structured_output), but it will include a metadata warning that capabilities may not be accurate. You are responsible for ensuring the model actually supports the features you attempt to use.

Local providers like ollama and gpustack do not require this flag; they assume models exist automatically.

chat = RubyLLM.chat(
  model: "my-custom-deployment",
  provider: :openai,
  assume_model_exists: true
)

chat.model.capabilities  # => ["function_calling", "streaming", "vision", "structured_output"]
chat.model.metadata      # => { warning: "Assuming model exists, capabilities may not be accurate" }

To use RubyLLM instrumentation in non-Rails environments, assign an object to RubyLLM.configure { |config| config.instrumenter = ... }. The object must implement an instrument(name, payload) method that yields to a block.

If an exception occurs within the block, it is recommended to merge :exception and :exception_object into the payload to maintain consistency with Rails-style instrumentation.

class AppInstrumenter
  def instrument(name, payload)
    started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)

    result = yield if block_given?
    result
  rescue StandardError => error
    payload = payload.merge(
      exception: [error.class.name, error.message],
      exception_object: error
    )
    raise
  ensure
    duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
    Observability.record(name, payload.merge(duration: duration))
  end
end

RubyLLM.configure do |config|
  config.instrumenter = AppInstrumenter.new
end

RubyLLM 2.0 separates providers (host, auth, catalog) from protocols (wire format). While the public API (RubyLLM.chat, with_params, embed, paint) remains unchanged, the underlying implementation has shifted.

OpenAI Protocol Changes

OpenAI now defaults to the Responses API. To continue using the Chat Completions protocol, you must explicitly configure it globally or per chat:

# Global configuration
RubyLLM.configure do |config|
  config.openai_protocol = :chat_completions
end

# Per-chat selection
RubyLLM.chat(model: 'gpt-5.4', protocol: :chat_completions)

Note: If you use with_provider_options with Chat Completions-only keys (like response_format), you must either switch the chat to :chat_completions or use the Responses API equivalents (e.g., text: { format: ... }).

Internal Module Changes

Wire-format internals have moved to RubyLLM::Protocols. For example, RubyLLM::Providers::OpenAI::Chat is now RubyLLM::Protocols::ChatCompletions::Chat. Provider classes no longer inherit from each other; instead, a provider declares its protocols.

RubyLLM.configure do |config|
  config.openai_protocol = :chat_completions
end

# or per chat, as part of model selection
RubyLLM.chat(model: 'gpt-5.4', protocol: :chat_completions)

To implement Retrieval-Augmented Generation (RAG), add neighbor and ruby_llm to your Gemfile. Use the neighbor generator to set up pgvector support in your database.

1. Add gems to Gemfile:

   gem 'neighbor'
   gem 'ruby_llm'

2. Generate and run migrations:

   bin/rails generate neighbor:vector
   bin/rails db:migrate

3. Create a migration with a :vector column. For OpenAI embeddings, use a limit of 1536 and an hnsw index with vector_l2_ops for efficient similarity search.

# Gemfile
gem 'neighbor'
gem 'ruby_llm'

# Generate migration for pgvector
bin/rails generate neighbor:vector
bin/rails db:migrate

class CreateDocuments < ActiveRecord::Migration[7.1]
  def change
    create_table :documents do |t|
      t.text :content
      t.string :title
      t.vector :embedding, limit: 1536 # OpenAI embedding size
      t.timestamps
    end

    add_index :documents, :embedding, using: :hnsw, opclass: :vector_l2_ops
  end
end

Configure API keys for the specific providers you intend to use via the RubyLLM.configure block. RubyLLM only requires configuration for the providers you actually call; attempting to use an unconfigured provider will raise RubyLLM::ConfigurationError.

Supported providers include Anthropic, Azure, Bedrock, DeepSeek, Gemini, GPUStack, Mistral, Ollama, OpenAI, OpenRouter, Perplexity, Vertex AI, and xAI. Most providers support an optional *_api_base configuration for custom endpoints (available in v1.16+ for most).

RubyLLM.configure do |config|
  # Anthropic
  config.anthropic_api_key = ENV['ANTHROPIC_API_KEY']
  config.anthropic_api_base = ENV['ANTHROPIC_API_BASE']

  # Azure
  config.azure_api_base = ENV['AZURE_API_BASE']
  config.azure_api_key = ENV['AZURE_API_KEY']
  config.azure_ai_auth_token = ENV['AZURE_AI_AUTH_TOKEN']

  # Bedrock
  config.bedrock_api_key = ENV['AWS_ACCESS_KEY_ID']
  config.bedrock_secret_key = ENV['AWS_SECRET_ACCESS_KEY']
  config.bedrock_region = ENV['AWS_REGION']
  config.bedrock_session_token = ENV['AWS_SESSION_TOKEN']
  config.bedrock_api_base = ENV['BEDROCK_API_BASE']

  # DeepSeek
  config.deepseek_api_key = ENV['DEEPSEEK_API_KEY']
  config.deepseek_api_base = ENV['DEEPSEEK_API_BASE']

  # Gemini
  config.gemini_api_key = ENV['GEMINI_API_KEY']
  config.gemini_api_base = ENV['GEMINI_API_BASE']

  # GPUStack
  config.gpustack_api_base = ENV['GPUSTACK_API_BASE']
  config.gpustack_api_key = ENV['GPUSTACK_API_KEY']

  # Mistral
  config.mistral_api_key = ENV['MISTRAL_API_KEY']
  config.mistral_api_base = ENV['MISTRAL_API_BASE']

  # Ollama
  config.ollama_api_base = 'http://localhost:11434/v1'
  config.ollama_api_key = ENV['OLLAMA_API_KEY']

  # OpenAI
  config.openai_api_key = ENV['OPENAI_API_KEY']
  config.openai_api_base = ENV['OPENAI_API_BASE']

  # OpenRouter
  config.openrouter_api_key = ENV['OPENROUTER_API_KEY']
  config.openrouter_api_base = ENV['OPENROUTER_API_BASE']

  # Perplexity
  config.perplexity_api_key = ENV['PERPLEXITY_API_KEY']
  config.perplexity_api_base = ENV['PERPLEXITY_API_BASE']

  # Vertex AI
  config.vertexai_project_id = ENV['GOOGLE_CLOUD_PROJECT']
  config.vertexai_location = ENV['GOOGLE_CLOUD_LOCATION']
  config.vertexai_service_account_key = ENV['VERTEXAI_SERVICE_ACCOUNT_KEY']
  config.vertexai_api_base = ENV['VERTEXAI_API_BASE']

  # xAI
  config.xai_api_key = ENV['XAI_API_KEY']
  config.xai_api_base = ENV['XAI_API_BASE']
end