Dust AI Agent Platform Documentation

repository·main·Indexed 23 days ago

https://github.com/dust-tt/dust

Documentation for Dust, a custom AI agent platform for building and deploying specialized agents. Includes guides for the Dust CLI (@dust-tt/dust-cli), the dust-sandbox CLI (dsbx) for interacting with sandbox functions, the Rust-based egress-proxy for enforcing outbound network policies, and the Connectors API server.

Tokens
159.2K
Snippets
233
Records
961
Agent score
81%

What's inside Dust

  1. Overview of the Dust MCP Server

    main

    Dust acts as an MCP (Model Context Protocol) resource server, allowing external clients like Cursor or Claude Desktop to call Dust functions using OAuth.

    Architecture

    • Dust: The MCP resource server providing tools and data.
    • WorkOS AuthKit: The authorization server handling OAuth, organization picking, and JWT issuance.
    • Session Scoping: Each authenticated session is scoped to a specific Dust user + workspace, which is derived from the WorkOS organization selected during the OAuth flow.

    Key Implementation Details

    • Statelessness: Each HTTP request uses its own transport and server. There is no Mcp-Session-Id; the system is designed to work across multiple front-api pods.
    • Workspace Mapping: Dust workspaces map to WorkOS organizations via the workOSOrganizationId field.
  2. Overview of Egress Proxy

    main

    The egress-proxy is a Rust-based service designed to enforce Dust sandbox outbound network policies. It acts as a TLS-terminated proxy listener for sandbox forwarder connections, providing security through several mechanisms:

    • Policy Enforcement: Validates connections using JWTs and enforces per-sandbox/workspace policies stored in a Google Cloud Storage (GCS) bucket, alongside a global DNS-over-HTTPS (DoH) blocklist.
    • Security Checks: Performs server-side DNS resolution and SSRF (Server-Side Request Forgery) checks on resolved addresses before establishing upstream TCP connections.
    • Traffic Handling: Manages bidirectional byte forwarding between sandboxes and upstream destinations.
    • Health Monitoring: Provides a /healthz endpoint to monitor process health.
  3. Understand the Sparkle component organization

    main

    The Sparkle library is organized into several functional categories in the Storybook sidebar:

    • Foundations: Design tokens (colors, typography, shadows, motion).
    • Assets: Brand assets (logos, icon sets, avatar sets).
    • Actions: Triggers like Button, SplitButton, IconButton, and SliderToggle.
    • Forms & Inputs: Controls like Input, TextArea, Checkbox, RadioGroup, and Dropdown.
    • Data Display: Presentational components like Avatar, Icon, Card, Chip, Counter, DataTable, and Tree.
    • Feedback & Status: Spinners, loading states, and notifications.
    • Navigation: Breadcrumbs, Tabs, Toolbars, and navigation lists.
    • Overlays: Dialogs, Sheets, Popovers, and Tooltips.
    • Layout: Scaffolding and spacing like Page, Container, ScrollArea, and Separator.
    • Lists: List and item components.
    • Product: App-specific UI for Conversation (messages, citations, code blocks) and Agent surfaces.
    • Effects & Motion: Animated and decorative effects.
    • Lab: Experimental, unstable components.
  4. Compare io-ts and zod syntax patterns

    main

    When migrating from io-ts to zod, use the following pattern mappings for schema definitions, type extraction, and validation.

    Schema Definition

    io-ts

    const Schema = t.type({
      name: t.string,
      age: t.number,
      email: t.union([t.string, t.undefined]),
    });

    zod

    const Schema = z.object({
      name: z.string(),
      age: z.number(),
      email: z.string().optional(),
    });

    Type Extraction

    io-ts

    type MyType = t.TypeOf<typeof Schema>;

    zod

    type MyType = z.infer<typeof Schema>;

    Validation

    io-ts

    const result = Schema.decode(data);
    if (isLeft(result)) {
      const errors = reporter.formatValidationErrors(result.left);
      return Err(errors);
    }
    return Ok(result.right);

    zod

    const result = Schema.safeParse(data);
    if (!result.success) {
      return Err(result.error.errors.map((e) => e.message));
    }
    return Ok(result.data);
    // io-ts
    const Schema = t.type({
      name: t.string,
      age: t.number,
      email: t.union([t.string, t.undefined]),
    });
    
    // zod
    const Schema = z.object({
      name: z.string(),
      age: z.number(),
      email: z.string().optional(),
    });
  5. Avoid N+1 database calls

    main

    Avoid executing database queries or using DB-backed helpers inside loops, Promise.all, or concurrentExecutor. This leads to the N+1 pattern, which can exhaust the database connection pool as data scales.

    Solution: Use batching. Fetch all required related rows in a single scoped query using methods designed for multiple IDs (e.g., fetchByModelIds) and then reconstruct the data in memory.

    // BAD: one user query per membership
    const users = await Promise.all(
      memberships.map((membership) =>
        UserResource.fetchByModelId(membership.userId)
      )
    );
    
    // GOOD: one query, then reconstruct by id.
    const users = await UserResource.fetchByModelIds(
      memberships.map((membership) => membership.userId)
    );
  6. Configure function userIdentity requirements

    main

    When defining a function's schema, you can specify userIdentity to control access requirements:

    • Omit userIdentity: The function is callable without a specific user.
    • workspace_user_required: The function requires a current member of the Pod's workspace.
    • interactive_workspace_user_required: The function must be called directly from a logged-in member's live Dust session (prevents calls from agents, schedules, or API clients acting on a member's behalf).

    Example schema definition:

    export const schema = {
      userIdentity: "workspace_user_required",
      input: z.object({}),
      output: z.object({}),
    };
    export const schema = {
      userIdentity: "workspace_user_required",
      input: z.object({}),
      output: z.object({}),
    };
  7. Separate business logic from API handlers

    main

    API handlers in front-api/routes/ should remain thin. They are responsible for:

    • Authentication/Authorization
    • HTTP method dispatch
    • Request validation (using zod)
    • Calling the business layer (lib/api/*)
    • Mapping business results to HTTP responses

    Business Logic Location: Non-trivial logic (sequential DB operations, conditional branching on state, coordination across resources) must live in lib/api/* or within a Resource.

    Crucial Rule: Business logic must not return HTTP-specific error envelopes like APIErrorWithStatusCode. It should return domain results (e.g., Result<T, DomainError>). The handler is the only layer that should map domain errors to HTTP status codes.

    // BAD — business layer encoding HTTP status codes
    export async function getThing(
      auth: Authenticator,
      id: string
    ): Promise<Result<Thing, APIErrorWithStatusCode>> {
      if (!found) {
        return new Err({
          status_code: 404,
          api_error: { type: "thing_not_found", message: "Not found." },
        });
      }
    }
    
    // GOOD — domain error, handler maps to HTTP
    export class ThingError extends Error {
      constructor(readonly type: "not_found" | "unauthorized") {
        super(type);
      }
    }
    
    export async function getThing(
      auth: Authenticator,
      id: string
    ): Promise<Result<Thing, ThingError>> {
      if (!found) {
        return new Err(new ThingError("not_found"));
      }
    }
    
    // Handler does the mapping:
    const result = await getThing(auth, id);
    if (result.isErr()) {
      switch (result.error.type) {
        case "not_found": return apiError(req, res, { status_code: 404, ... });
        case "unauthorized": return apiError(req, res, { status_code: 403, ... });
      }
    }
  8. Use the Steering feature for message redirection

    main

    The Steering feature (gated by the enable_steering feature flag) allows users to redirect an agent's work by sending a new message while an agent is still running.

    When steering is enabled, postUserMessage follows a 'pending' path if the message contains user mentions and the currently running agent, but does not address a different agent.

    Constraints for Steering:

    • A message can contain at most one agent mention.
    • You cannot address a different agent than the one currently running.
    • API callers bypass the pending path and always follow the normal path.

    Workflow:

    1. A UserMessage is created with visibility: "pending".
    2. A UserMessageNewEvent (pending) is published.
    3. gracefullyStopAgentLoop() is called with reason: "steering".
    4. Once the current loop stops (with status "gracefully_stopped" or "succeeded"), the pending message is promoted to visibility: "visible", and a new AgentMessage is created to handle the new instruction.
  9. Understanding the Asynchronous Sandbox Invocation Flow

    main

    To prevent deadlocks during tool approvals, Dust uses an asynchronous execution model for sandbox function invocations. Instead of executing the sandbox function directly within the HTTP request lifecycle, the system follows an agent loop ownership model:

    1. HTTP Request: The POST invocation creates a durable invocation record and immediately returns an invocation ID to the client.
    2. Temporal Workflow: The system launches a runSandboxFunctionInvocationWorkflow to handle the long-running execution.
    3. Temporal Activity: A runSandboxFunctionInvocationActivity owns the actual sandbox.exec() call.

    This architecture allows the UI/visualization to subscribe to the invocation event stream via the invocation ID before the function execution reaches a state that requires user approval (e.g., a tool blocking for validation).

  10. Understand the Sandbox Secret Swap mechanism

    main

    The Sandbox Secret Swap is a security architecture designed to prevent secret exfiltration from sandbox environments. Instead of injecting real secrets into the agent process environment in plaintext, the system injects deterministic, opaque placeholders.

    How it works:

    1. Placeholder Injection: The agent process environment contains a random nonce (placeholder) instead of the real secret.
    2. Just-in-Time Substitution: When an HTTPS request leaves the sandbox, the dsbx (sandbox) component performs a Man-in-the-Middle (MITM) on the egress path. It substitutes the placeholder with the real secret value just before the bytes hit the upstream server.
    3. Destination Gating: Substitution only occurs if the destination domain is explicitly listed in the secret's allowedDomains. If a placeholder is detected going to a domain not in its allowedDomains, dsbx drops the connection.

    Key Security Invariant: The real secret value is never forwarded to a destination outside the matching secret's allowedDomains.

  11. How Steering and Graceful Stops work together

    main

    The steering mechanism relies on a specific sequence of events to ensure user messages are not lost when an agent is interrupted:

    1. User sends message: postUserMessage is called. If an agent is active, it creates a UserMessageModel and a MessageModel with visibility: "pending". It then triggers a GracefulStopRequestedEvent.
    2. Agent stops: The agent loop receives the signal, finishes its current step, and exits.
    3. Promotion: The Temporal activity finalizeGracefullyStoppedAgentLoopActivity calls the API to promote all "pending" messages to "visible" and starts a new agent loop.

    Race Safety: The getConversationRankVersionLock advisory lock serializes postUserMessage (API side) and finalizeGracefulStopAgentMessage (Temporal side). This prevents a user message from being created after the promotion logic has already run, or vice versa.