cascadeflow Documentation

Cascadeflow is an agent runtime intelligence layer designed to optimize AI workflows through several key capabilities:

• Speculative Cascading: Intelligently escalates tasks from cheap models to more capable ones.
• Cost & Latency Optimization: Proven to reduce costs by 40-85% and improve speed by 2-10x.
• Provider Support: Works with OpenAI, Anthropic, Groq, Ollama, vLLM, Together, and LiteLLM.
• Agentic Capabilities: Supports agent loops (multi-turn tool execution), tool calling in a universal format, and Hermes Agent Routing.
• Governance & Control: Includes budget enforcement (per-run and per-user), compliance gating (GDPR, HIPAA, etc.), and user profile systems.
• Observability: Provides decision traces, cost tracking via OpenTelemetry, and hooks/callbacks for telemetry and streaming.

The @cascadeflow/vercel-ai integration provides several advanced capabilities:

• Stream Support: Compatible with AI SDK v4 data stream and v5/v6 UI streams.
• Multi-turn Chat: Full support for useChat multi-turn interactions.
• Message Formats: Supports the parts message format (AI SDK v6).
• Tooling: Tool call streaming visibility and server-side tool execution loops.
• Control: Multi-step controls and cascade decision stream parts.
• Security: Request-level overrides managed via an allowlist.

Speculative Execution

The cheaper model (drafter) runs first. If its response passes quality validation (based on configurable confidence thresholds, minimum token counts, or semantic validation), cascadeflow returns it immediately. If it fails, the verifier model runs as a fallback.

Cost Tracking

Every response includes totalCost and savingsPercentage to measure the impact of the optimization.

cascadeflow implements speculative execution with quality validation to reduce LLM costs and latency.

1. Speculative Execution: It first executes queries using small, fast, and inexpensive models.
2. Quality Validation: It validates the response against configurable thresholds such as completeness, confidence, and correctness.
3. Dynamic Escalation: If quality validation fails, it automatically escalates the query to larger, more capable models.
4. Pattern Learning: It optimizes future routing and cascading decisions based on learned patterns.

This approach typically results in 40-85% cost reduction and 2-10x faster responses by handling 60-70% of queries with efficient models without requiring manual escalation.

For every LLM call or tool execution, the harness follows this sequence:

1. Record: Captures model, step number, cumulative cost, latency, and energy.
2. Compliance Check: Verifies if the model is in the allowlist for the configured regulation.
3. Hard Cap Check: Validates against budget, tool calls, latency, and energy limits.
4. KPI Scoring: Calculates scores for quality, cost, latency, and energy based on kpi_weights.
5. Decision: Determines the action: allow, switch_model, deny_tool, or stop.
6. Enforcement: Executes the decision (enforced in enforce mode, logged in observe mode).
7. Trace: Appends the full decision record for auditability.

The cascadeflow harness provides several core runtime capabilities for managing AI agent execution:

• Observe mode: Provides zero-change runtime visibility into your existing application.
• Enforce mode: Allows you to set runtime caps and execute control actions.
• Budget enforcement: Enables per-run and per-user spend control.
• Compliance gating: Uses model allowlists to enforce policies like GDPR or HIPAA.
• KPI weighting: Manages trade-offs across quality, cost, latency, and energy.
• Energy tracking: Provides a compute-intensity proxy for carbon-aware routing.
• Decision traces: Generates an audit trail for every decision made during runtime.

cascadeflow supports various cascading and integration patterns to enhance agent workflows:

• Speculative model cascading: Optimizes execution by cascading models based on complexity/cost.
• Multi-provider routing: Routes requests across different model providers.
• Framework Integrations:
  • LangChain and LangGraph
  • OpenAI Agents SDK
  • CrewAI
  • Google ADK
  • Vercel AI SDK
  • n8n
  • Hermes Agent (via cascadeflow.integrations.hermes)

The Harness Engine provides runtime intelligence by wrapping agent execution. It monitors every LLM call or tool execution to make decisions based on constraints.

Decision Flow:

1. Record: Logs the model, step number, and cumulative metrics.
2. Evaluate: Checks constraints like budget, compliance, tool calls, latency, and energy.
3. Score: Evaluates the call against configured KPI weights.
4. Decide: Selects an action: allow, switch_model, deny_tool, or stop.
5. Enforce: Executes the decision if in enforce mode (or just logs in observe mode).
6. Trace: Appends a record for auditability.

Cascadeflow provides specialized patterns for workflows that are interactive, tool-heavy, or involve multi-step agentic execution. Key areas of focus include:

• Tools: Managing tool loops where budget and runtime controls are critical.
• Streaming: Preserving low latency and clear runtime behavior for interactive user experiences.
• Multi-step execution: Managing costs and failure compounding across multiple sequential steps.

The Cascade Engine optimizes model selection using speculative execution with quality validation. It follows this workflow:

1. Speculative Execution: Executes small, fast, and inexpensive models first.
2. Quality Validation: Checks the response against configurable thresholds (completeness, confidence, correctness).
3. Dynamic Escalation: If quality validation fails, the engine automatically escalates the request to a larger, more capable model.
4. Pattern Learning: Optimizes future routing decisions based on domain-specific patterns.

This approach aims for 40-85% cost reduction and 2-10x faster responses while maintaining quality.

The LangChain integration includes the following capabilities:

• Full LCEL support: Works with pipes, sequences, and batch operations.
• Streaming: Supports streaming with pre-routing logic.
• Tool calling: Compatible with tool calling and structured output.
• LangSmith integration: Includes LangSmith cost tracking metadata.
• Model discovery: Utilities for model pair validation and discovery.