DOCSEARCHbyAlgolia
Submit Docs

Plano Documentation

repository·main·Indexed 27 days ago

https://github.com/katanemo/plano

An AI-native proxy server and data plane designed for agentic applications. Plano handles orchestration, model routing, observability, and safety guardrails. The project includes a CLI (planoai), a Rust-based Envoy WASM filter for the Plano gateway, and the hermesllm crate for parsing LLM requests and responses across providers such as OpenAI, Mistral, Groq, Deepseek, Gemini, Claude, and GitHub.

Tokens
108.3K
Snippets
279
Records
525
Agent score
91%

What's inside Plano

  1. Overview of Plano's Core Capabilities

    main

    Plano is an out-of-process dataplane designed to handle the 'hidden AI middleware' required to move agentic applications from prototypes to production. It moves core delivery concerns away from application code and into a unified, centrally managed layer.

    Key capabilities include:

    • Orchestration: Low-latency orchestration between agents. You can add or change agent routing strategies in the dataplane without modifying your application code.
    • Guardrails & Memory Hooks: Centralized application of jailbreak protection, content policies, and context workflows (rewriting, retrieval, redaction) via Filter Chains.
    • Model Agility: Decouples agents from specific providers by routing via model names, semantic aliases, or automatic preferences. This allows swapping models without refactoring prompts or tool-calling logic.
    • Agentic Signals™: Zero-code capture of traces, metrics (latency, token usage, error rates), and behavior signals. Plano uses the W3C Trace Context standard (traceparent header) and OpenTelemetry for end-to-end observability.

  2. Overview of Plano-Orchestrator

    main

    Plano-Orchestrator is a preference-based routing model designed to align LLM selection with human preferences rather than just benchmarks. It uses Domain-Action mappings to route requests.

    Key Concepts:

    • Domain: The high-level thematic category (e.g., legal, programming).
    • Action: The specific operation requested (e.g., summarization, code generation).

    At inference, the orchestrator analyzes the prompt to infer the domain and action via semantic similarity and applies user-defined preferences to select the optimal model. This approach allows for transparent, controllable, and flexible routing that can be updated without retraining the router.

  3. Overview of Plano delivery infrastructure

    main

    Plano is an AI-native proxy server and data plane designed as delivery infrastructure for agentic applications. It acts as a centralized middleware layer that decouples agent logic from infrastructure concerns.

    Key capabilities include:

    • Agent Routing and Orchestration: Centralizes routing logic instead of implementing it bespoke in every codebase.
    • Observability: Provides rich agentic signals and traces for continuous improvement.
    • Safety and Moderation: Implements guardrail filters for safety.
    • Smart LLM Routing: Provides APIs for routing between different LLM providers to improve UX and DX agility.

    Plano is designed to work with any programming language or AI framework, allowing developers to focus on core product logic while engineering teams standardize policies and access controls.

  4. Overview of Plano

    main

    Plano is an AI-native proxy server and data plane designed for agentic applications. It decouples core delivery concerns from application logic by providing a unified, out-of-process dataplane for:

    • Orchestration: Low-latency orchestration between agents via declarative configuration.
    • Model Agility: Unified LLM routing APIs that allow routing by model name, semantic aliases, or automatic preferences.
    • Agentic Signals & Observability: Zero-code capture of agentic signals and automatic OpenTelemetry (OTEL) traces/metrics across all agents.
    • Moderation & Memory: Consistent application of guardrails, moderation policies, and memory via Filter Chains.

  5. Understand Agentic Signals™

    main

    Agentic Signals are lightweight, model-free behavioral indicators computed from live interaction trajectories. They are attached to your existing OpenTelemetry (OTel) traces to provide a low-cost triage layer for agent performance.

    Instead of using expensive LLM-as-judge evaluations for every session, Signals identify specific interactions (failures, successes, or friction points) that are worth inspecting for model optimization (e.g., DPO, RLHF, or fine-tuning).

    Key Characteristics:

    • Model-free: Computed without additional LLM calls, keeping costs low ($O(\text{messages})$).
    • Non-intrusive: Attaches structured attributes and span events to existing OTel spans without changing online agent behavior or affecting client responses.
    • Closed-loop: Enables a flywheel of: Instrument $\rightarrow$ Sample/Triage $\rightarrow$ Data Construction $\rightarrow$ Model Optimization $\rightarrow$ Deploy.

  6. Explore Plano Signals™ concepts

    main
    Plano uses Signals™ to provide observability into agentic interactions. Signals are categorized into a taxonomy that covers the quality of user-agent conversation (Interaction), the quality of actions taken by the agent (Execution), and the external system or boundary conditions (Environment). This framework implements the concepts described in the paper 'Signals: Trajectory Sampling and Triage for Agentic Interactions'.

  7. Post-Request Processing and Observability

    main

    After a request completes and the stream is destroyed, Plano performs the following background tasks:

    • Monitoring Updates: Statistics such as timing, active requests, upgrades, and health checks are updated. Stats are batched and written by the main thread periodically.
    • Access Logging: Entries are written to the access logs.
    • Tracing: Trace spans are finalized. The HTTP Connection Manager (HCM) creates a span describing the request duration and details, finalizing it during post-request processing.

  8. Understand the Plano Request Lifecycle and Network Topology

    main

    Plano operates within a network topology consisting of Downstream (Ingress) and Upstream (Egress) listeners to manage the flow of requests between clients and LLMs.

    • Downstream (Ingress) listeners: These receive requests from upstream clients (such as a web UI or applications forwarding prompts) and initiate the request lifecycle.
    • Upstream (Egress) listeners: These receive requests from the application logic and forward them to LLMs.

    The lifecycle follows a request from its arrival at a Downstream listener, through internal Plano processing, to the corresponding dispatch via an Upstream listener, and finally back through the response path to the downstream client.

  9. Understand the Plano Agent Architecture

    main

    Plano uses a dual-loop architecture to separate agent business logic from orchestration logic:

    Inner Loop (Agent Logic)

    This is where you implement your agent's specific business logic. You can use any language or framework (Python, TypeScript, etc.) as long as the agent is exposed as an HTTP service.

    Your agent is responsible for:

    • Deciding which tools or APIs to call.
    • Interpreting tool results and determining next steps.
    • Deciding when to call an LLM for reasoning.
    • Determining task completion and returning the final response.

    Outer Loop (Orchestration)

    This is managed by Plano-Orchestrator. It handles the high-level lifecycle of requests across your agents and LLMs.

    Plano-Orchestrator handles:

    • Intent analysis: Analyzing prompts to determine user intent.
    • Routing: Directing requests to the correct agent(s) or LLM(s).
    • Sequencing: Managing collaboration and order between multiple agents.
    • Lifecycle management: Handling retries, failover, circuit breaking, and load balancing.

  10. Understand the Plano High-Level Architecture

    main

    Plano consists of two self-contained processes designed to run alongside or separate from your application servers:

    1. Networking/Connection Management Process: Manages HTTP-level concerns such as protocol management, request ID generation, header sanitization, connection pools, load balancing, retries, and failover.
    2. Controller (Bright Staff): A lightweight, memory-efficient controller for agentic traffic that sits inside the Plano data plane. It makes real-time decisions about how prompts are handled, forwarded, and processed.

    These subsystems are bridged using Envoy's HTTP router filter and cluster manager subsystems. Plano utilizes an event-based thread model where a main thread handles server lifecycle and configuration, while worker threads process requests using an event loop (libevent).

  11. Understand Plano Listeners

    main

    In Plano, Listeners are top-level primitives that bind network traffic to the dataplane. They manage incoming connections from downstream clients (edge) and provide a unified egress endpoint for calls from your applications to upstream LLMs. Plano uses Envoy's Listener subsystem under the hood but simplifies the configuration surface.

    Listeners are modular and can be used in three primary ways:

    • Inbound Listeners: Accept connections from clients (web frontends, backend services) to handle prompt traffic, TLS termination, and guardrails.
    • Outbound/Model Listeners: Provide a single OpenAI-compatible HTTP endpoint for your applications to call upstream LLMs. Plano handles provider selection, retries, and failover.
    • Both: You can configure both inbound and outbound listeners in a single deployment to cover the full request path.

  12. Understand Filter Chains in Plano

    main

    Filter chains are ordered lists of mutations that a request flows through before reaching its final destination (such as an agent, an LLM, or a tool backend). They allow you to capture reusable workflow steps in the dataplane without coupling logic into your application code.

    Key capabilities of a filter:

    1. Inspect: View incoming prompts, metadata, and conversation state.
    2. Mutate/Enrich: Rewrite queries or build context (e.g., RAG context envelopes).
    3. Short-circuit: Block requests early based on compliance or safety (e.g., returning an error on a policy violation).
    4. Observe: Emit structured logs and traces for debugging.

    Common Use Cases:

    • Guardrails and Compliance: Masking sensitive data or blocking unsafe requests.
    • Query Rewriting and RAG: Normalizing entities and assembling context from memory/profiles.
    • Observability: Injecting correlation IDs and sampling traces.