Open Generative UI

Three.js uses a right-handed Y-up coordinate system. When modeling vehicles like aircraft, follow these conventions:

Axes:

• X: Right (positive) / Left (negative)
• Y: Up (positive) / Down (negative)
• Z: Toward viewer (positive) / Away from viewer (negative)

Aircraft Orientation:

• Fuselage: Long axis along Z (nose at -Z, tail at +Z).
• Wings: Wide along X, thin along Y, short along Z.
• Vertical Stabilizer: Tall along Y, thin along X, short along Z.

Flight Dynamics (Rotations):

• Pitch: Rotation around X (nose up/down).
• Roll: Rotation around Z (wings tilt).
• Yaw: Rotation around Y (nose left/right).

// Correct aircraft orientation example:
// Fuselage along Z
const fuselage = new THREE.Mesh(
  new THREE.CylinderGeometry(0.15, 0.08, 2.0, 12),
  material
);
fuselage.rotation.x = Math.PI / 2; // CylinderGeometry default is Y-up, rotate to Z-forward

// Wings along X
const wing = new THREE.Mesh(
  new THREE.BoxGeometry(2.5, 0.03, 0.4), // wide X, thin Y, short Z
  material
);

// Vertical stabilizer along Y
const tailFin = new THREE.Mesh(
  new THREE.BoxGeometry(0.03, 0.4, 0.3), // thin X, tall Y, short Z
);
tailFin.position.set(0, 0.2, 0.9); // above and behind

Follow these strict rules to maintain visual consistency with the host interface:

Typography

• Headings: h1 (22px), h2 (18px), h3 (16px). All must have font-weight: 500.
• Body: 16px, font-weight: 400, line-height: 1.7.
• Weights: Use only 400 (regular) and 500 (medium). Never use 600 or 700.
• Casing: Use sentence case. Never use Title Case or ALL CAPS.
• Formatting: No mid-sentence bolding. Use code style for entity, class, or function names.
• Minimum Size: No font-size below 11px.

Component Styling

• Borders: Use 0.5px solid var(--color-border-tertiary).
• Cards: background: var(--color-background-primary), border: 0.5px solid var(--color-border-tertiary), border-radius: var(--border-radius-lg), padding: 1rem 1.25rem.
• Visual Effects: Do not use gradients, drop shadows, blur, glow, or neon effects.
• Icons: Do not use emojis; use CSS shapes or SVG paths.
• Containers: The background of the outer container must always be transparent.

Data Integrity

• Number Formatting: Always round displayed numbers to avoid floating-point artifacts. Use Math.round(), .toFixed(n), or Intl.NumberFormat.

Use this decision matrix to determine which technology the agent should use to satisfy a user request:

The generateSandboxedUi tool is used to stream rich UI components. To ensure smooth streaming and correct rendering, you must emit parameters in this EXACT order:

1. initialHeight: Estimated height of the finished UI in pixels.
2. placeholderMessages: 2-4 short, playful progress messages.
3. css: All styles. This must be sent up front so the user sees a styled placeholder immediately. Keep it lean.
4. html: Clean body markup. Do not include <style> blocks (use the css parameter instead) or monolithic <script> blocks.
5. jsFunctions: Named function declarations representing a reusable toolbox of behavior. Use parameterized generators (e.g., drawWing(color)) to allow for easy refinements via expressions.
6. jsExpressions: Small statements that invoke the functions in jsFunctions, applied one-by-one to show live updates.

Sandbox Constraints:

• The sandbox iframe has no same-origin access (no localStorage, sessionStorage, cookies, IndexedDB, or same-origin fetch).
• Use the host bridge for communication:
  • await Websandbox.connection.remote.sendPrompt({ text })
  • await Websandbox.connection.remote.openLink({ url }) (must be https only).
• Pre-injected libraries: three, gsap, d3, and chart.js are available via an importmap.

Open Generative UI is a runtime-level feature that allows the agent to stream free-form HTML/SVG widgets directly into a sandboxed iframe. Unlike hook-registered components, these are driven by the runtime's openGenerativeUI option.

Workflow

1. Runtime Configuration: Enable the rail by setting openGenerativeUI: true in your CopilotKit API route. This injects the generateSandboxedUi tool.
2. Streaming: The agent calls generateSandboxedUi, streaming parameters in a specific order: initialHeight → placeholderMessages → css → html → jsFunctions → jsExpressions.
3. Rendering: The frontend uses the renderActivityMessages prop on the <CopilotKit> provider to register a renderer. The renderer uses Idiomorph to morph HTML updates and eventually boots a websandbox iframe.
4. Sandbox Bridge: The generated UI can communicate with the host via Zod-validated functions like Websandbox.connection.remote.sendPrompt({ text }) and Websandbox.connection.remote.openLink({ url }).

Environment Capabilities

• ES Modules: Pre-injected via importmap. You can use <script type="module"> with bare imports (e.g., import * as THREE from "three") or await import(...) inside jsFunctions. Note: jsFunctions run as classic scripts, so top-level await is not allowed.
• Available Libraries: three, gsap, d3, chart.js/auto.
• Theming: Access CSS variables like --color-background-primary, --color-text-primary, --font-sans, etc.
• SVG Classes: .c-purple, .c-teal, .c-coral, .c-pink, .c-gray, .c-blue, .c-green, .c-amber, .c-red.

// app/layout.tsx
const renderActivityMessages = [OPEN_GEN_UI_ACTIVITY_RENDERER];

const openGenerativeUI = {
  sandboxFunctions: [...SANDBOX_FUNCTIONS], // sendPrompt, openLink
  designSkill: OPEN_GEN_UI_DESIGN_SKILL,    // teaches the agent the design system
};

<CopilotKit
  runtimeUrl="/api/copilotkit"
  renderActivityMessages={renderActivityMessages}
  openGenerativeUI={openGenerativeUI}
>

Open Generative UI is built as a Turborepo monorepo consisting of three primary applications that work together to enable agentic, generative user interfaces:

1. Frontend (apps/app/): A Next.js 16 application (using React 19 and TailwindCSS 4) that provides the user interface and handles generative UI registration.
2. Agent (apps/agent/): A Python-based LangGraph agent running on FastAPI (using uv for management) that executes logic and tools.
3. MCP (apps/mcp/): An optional Model Context Protocol server that provides resources, prompts, and tools.

Request Flow

The interaction flow follows this path:

• The Browser interacts with the Next.js App (:3000).
• The Next.js app uses a /api/copilotkit route which hosts the CopilotRuntime.
• The CopilotRuntime communicates with the FastAPI Agent (:8123) via LangGraphHttpAgent.
• The Agent executes LangGraph tools, uses CopilotKitMiddleware, and manages state.
• On the client side, the React UI uses specialized hooks (useAgent, useComponent, useFrontendTool, useHumanInTheLoop) to interact with the agent and render generative components.

Select a diagram type based on the information you need to convey:

1. Flowchart

• Use for: Sequential processes, decision trees, or pipelines.
• Layout: Top-to-bottom or left-to-right. Use a single direction.
• Rules: Arrows must never cross unrelated boxes (use L-bends to route around). Keep same-type boxes at the same height. Max 4-5 nodes per diagram.

2. Structural Diagram

• Use for: Containment and hierarchy (architecture, org charts, system components).
• Layout: Nested rectangles where the outer rectangle is the container and inner rectangles are regions.
• Rules: Max 2-3 nesting levels. Minimum 20px padding inside every container. Use different color ramps for parent vs child to show hierarchy.

3. Illustrative Diagram

• Use for: Building intuition for abstract concepts (e.g., "How does X work?").
• Layout: Freeform, following the subject's natural geometry (paths, ellipses, polygons).
• Rules: Color should encode intensity (warm = active, cool = dormant). Overlap shapes for depth, but ensure strokes do not cross text. Use leader lines for margin labels.