DOCSEARCHbyAlgolia
Submit Docs

vLLora

website·Indexed 32 days ago

https://vllora.dev

An AI agent debugger for cloning and experimenting with request traces in real time. Features include prompt engineering, model comparison, and parameter tuning. Supports routing OpenAI-style requests to providers like Anthropic, AWS Bedrock, Gemini, and OpenAI via VlloraLLMClient, as well as OTLP and console tracing for request lifecycle observability.

Tokens
13.1K
Snippets
46
Records
60
Agent score
62%

What's inside vLLora

  1. Overview of the Responses API

    The Responses API is a unified interface for building advanced AI agents. It is compatible with OpenAI's Responses API format and offers several advantages over traditional Chat Completions:

    • Structured Workflows: Supports multi-step tasks and built-in tools.
    • Multimodal Outputs: Handles rich data types that can be processed programmatically.
    • Tool Orchestration: Integrated support for web search, image generation, and other tools.
    • Streaming: Supports real-time response processing.

  3. Get system health snapshots with get_recent_stats

    The get_recent_stats tool provides a quick overview of error rates and call counts across different models and tools within a specified time window.
    // Request
{
  "last_n_minutes": 30
}

// Response
{
  "window_minutes": 30,
  "window_start": "2025-12-15T06:07:00Z",
  "window_end": "2025-12-15T06:12:00Z",
  "llm_calls": [
    { "model": "gpt-4.1-mini", "ok_count": 120, "error_count": 3, "total_count": 123 }
  ],
  "tool_calls": [
    { "tool_name": "web_search", "ok_count": 88, "error_count": 2, "total_count": 90 }
  ]
}

  5. Quickstart guide for vLLora setup

    Follow these steps to start debugging AI agents with vLLora:

    1. Install vLLora: Use Homebrew or build from source.
    2. Configure Provider: Open the UI at http://localhost:9091, select a provider (e.g., OpenAI), and enter your API key.
    3. Send Requests: Use the Chat UI or the provided curl requests in the Chat Section to start generating traces.
    4. Analyze Traces: View token usage, costs, and execution timelines in the Traces View.

  8. Initialize the Vllora LLM Client

    The VlloraLLMClient is initialized using ApiKeyCredentials. Ensure the VLLORA_OPENAI_API_KEY environment variable is set before running the client.
    use vllora_llm::client::VlloraLLMClient;
use vllora_llm::types::credentials::{ApiKeyCredentials, Credentials};

let client = VlloraLLMClient::default()
    .with_credentials(Credentials::ApiKey(ApiKeyCredentials {
        api_key: std::env::var("VLLORA_OPENAI_API_KEY")
            .expect("VLLORA_OPENAI_API_KEY must be set"),
    }));

  10. Use vLLora MCP prompt templates for trace analysis

    vLLora's MCP server provides standardized prompt templates to guide users or agents through common observability tasks:

    • debug_errors: Systematic analysis of errors in LLM traces.
    • analyze_performance: Identification of slow operations and bottlenecks.
    • understand_run_flow: Examination of multi-step agent execution and span trees.
    • search_traces_guide: Best practices for constructing trace searches.
    • monitor_system_health: Quick health check workflow using recent statistics.
    • analyze_costs: Analysis of cost patterns and usage by model/provider.

  11. Enable end-to-end tracing for OpenAI Agents SDK

    To enable tracing for OpenAI agents, install the vLLora package with the openai flag and initialize it before creating any agents. Set the VLLORA_API_BASE_URL environment variable to point to your vLLora instance.
    pip install 'vllora[openai]'
export VLLORA_API_BASE_URL=http://localhost:9090
    from vllora.openai import init

# Initialize before OpenAI setup
init()

from openai import OpenAI
# ...define and run agents...

  12. Process and extract text content from Responses API output

    Responses from the API are returned as a collection of OutputItems. To extract text, iterate through the output and match for OutputItem::Message. Within the message content, look for OutputMessageContent::OutputText. This object contains the text and optional annotations (such as citations or tool references).
    use vllora_llm::async_openai::types::responses::{OutputItem, OutputMessageContent};

for output in &response.output {
    match output {
        OutputItem::Message(message) => {
            for content in &message.content {
                match content {
                    OutputMessageContent::OutputText(text_output) => {
                        println!("\n{}", text_output.text);
                        if !text_output.annotations.is_empty() {
                            println!("Annotations: {:#?}", text_output.annotations);
                        }
                    }
                    _ => println!("Other content type: {:?}", content),
                }
            }
        }
        _ => {}
    }
}