DOCSEARCHbyAlgolia
Submit Docs

Claw3D

repository·main·Indexed 24 days ago

https://github.com/iamlukethedev/claw3d

A 3D virtual workspace for AI agents that provides a visual 'Office' layer to make AI workflows, such as code reviews and standups, observable and interactive. It acts as a UI and proxy layer for backend runtimes like OpenClaw or Hermes, utilizing a Next.js client and a custom WebSocket proxy to visualize agent activities in a shared 3D environment.

Tokens
47.6K
Snippets
71
Records
224
Agent score
83%

What's inside claw3d

  1. Navigate the Claw3D Codebase Structure

    main

    The repository is organized into several key directories:

    • src/app: Next.js App Router entry points and API routes. Route files (e.g., src/app/office/page.tsx) should primarily compose feature modules and server boundaries.
    • src/features: Vertical slices for UI and feature-specific state. Includes agents (fleet UI, chat, workflows), office (screens, panels, builder), and retro-office (the 3D React Three Fiber runtime).
    • src/lib: Shared domain logic and adapters. Includes gateway (browser client), office (intent parsing, animation triggers), and studio (settings persistence).
    • server: Custom Studio server and WebSocket proxy. server/gateway-proxy.js bridges browser traffic to the upstream OpenClaw Gateway to keep credentials server-side.
    • scripts: Utilities like scripts/sync-openclaw-gateway-client.ts (updates vendored client) and scripts/studio-setup.js (prepares local prerequisites).

  2. Security Hardening Summary for Production

    main

    The following security measures are implemented in the hardened version of Claw3D to prepare for production use:

    • Telemetry: @vercel/otel is removed and src/instrumentation.ts is a no-op; no data is sent to external services.
    • Token Validation: Uses crypto.timingSafeEqual() in server/access-gate.js to prevent timing attacks.
    • Auth Rate Limiting: Limits failed auth attempts to 10 failures per IP per 60 seconds to prevent brute-force attacks.
    • WebSocket Protection: Enforces a maximum frame size of 256 KB and a per-connection rate limit of 30 frames/second.
    • Security Headers: Baseline headers (CSP, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, and cross-origin isolation) are set via next.config.ts.
    • Media Security: The /api/gateway/media route rejects symlinked local files by verifying the realpath is within the allowed root.

  3. Discover where skills are loaded from

    main

    OpenClaw merges skills from multiple sources using a specific precedence order. If name conflicts occur, the source with the higher precedence wins ("last writer wins").

    Precedence Order (Lowest to Highest):

    1. Extra/Plugin Dirs: skills.load.extraDirs and plugin-contributed directories (source: openclaw-extra).
    2. Bundled Skills: Built-in skills (openclaw-bundled).
    3. Managed/Global Local Skills: Located at ~/.openclaw/skills (openclaw-managed).
    4. Personal Agent Skills: Located at ~/.agents/skills (agents-skills-personal).
    5. Project Agent Skills: Located at <workspace>/.agents/skills (agents-skills-project).
    6. Workspace Skills: Located at <workspace>/skills (openclaw-workspace).

  4. Understand the Main Runtime Flow

    main

    The runtime follows a specific data flow to ensure the UI stays in sync with the agent:

    1. Connection: The browser connects to Studio at /api/gateway/ws.
    2. Proxying: Studio proxies the connection to the upstream OpenClaw Gateway.
    3. Event Reception: GatewayClient receives runtime events.
    4. Subscription: src/app/office/page.tsx installs the main runtime subscription.
    5. Orchestration: gatewayRuntimeEventHandler.ts classifies and routes runtime events.
    6. Planning: Runtime workflow modules plan state updates and effect commands.
    7. Reconciliation: History sync pulls canonical chat.history if live streams are incomplete.
    8. Consumption: Agent and office UIs consume the resulting agent/session state.

  5. How hierarchy affects office operations

    main

    Hierarchy is intended to be operational rather than just theatrical. It influences several core office systems:

    • Delegation: Authority to route work is tied to roles. Owners, executives, managers, and leads can delegate. Members can hand off work but cannot broadly route it across the organization. Contractors and interns have highly restricted delegation capabilities.
    • Meetings: Roles determine who can call planning meetings (managers/leads) and who is required for review meetings.
    • QA & Reviews: Hierarchy dictates review authority. Leads can review work, managers can route items into QA, and owners/executives can override final readiness decisions.
    • Information Flow: Hierarchy shapes the weight and priority of announcements on the Bulletin Board or Whiteboard, though it should not be used to hide information.

  6. Map Custom Agents and Sessions to Claw3D models

    main

    Agent Mapping

    Do not model agents as fixed backend processes unless the runtime requires it. Instead, map agents to internal concepts like roles, lanes, or strategies. Expose these as office-meaningful identities (e.g., Main, Assistant, Coder, Reviewer) rather than leaking raw backend topology.

    Session Mapping

    Map Claw3D sessions to runtime conversations or execution threads. The provider should normalize internal identifiers into a stable session model containing:

    • sessionKey
    • conversation or thread id
    • active role
    • lane
    • requested model
    • resolved model
    • request id

  7. Distinguish between the Immersive Office and Builder Stacks

    main

    Claw3D maintains two distinct office-related stacks. It is critical to identify which one you are modifying:

    Immersive Office Stack (/office)

    Powered by React Three Fiber and src/features/retro-office. It renders the 3D world.

    • Composition Root: src/features/office/screens/OfficeScreen.tsx.
    • Logic: src/lib/office/eventTriggers.ts (derives animation from events) and src/lib/office/deskDirectives.ts (parses natural language).
    • Rendering: src/features/retro-office/RetroOffice3D.tsx.

    Builder Stack (/office/builder)

    Powered by Phaser and src/features/office. Used for editing office layouts.

    • Components: src/features/office/components/OfficeBuilderPanel.tsx and OfficePhaserCanvas.tsx.
    • Logic: src/features/office/phaser/OfficeBuilderScene.ts and src/lib/office/schema.ts (uses the OfficeMap schema).

  8. Understand the Studio-to-Gateway WebSocket Architecture

    main

    Claw3D Studio does not connect directly to the OpenClaw Gateway. Instead, it uses a two-hop WebSocket architecture to bridge the browser to the upstream service.

    The Network Path:

    1. Browser (Studio UI) connects to the Studio Server via a same-origin WebSocket at /api/gateway/ws.
    2. Studio Server (WS Proxy) acts as a bridge, opening a separate WebSocket connection to the OpenClaw Gateway (the upstream service) using the URL configured in settings.json.
    3. Protocol Handshake: The browser sends a req(connect) frame. The Studio proxy intercepts this, injects necessary authentication tokens (if not provided by the browser), and forwards the connection to the upstream Gateway.

    Key Concept: The browser always speaks to the Studio proxy. The 'upstream gateway URL' configured in Studio settings is used by the server-side proxy, not the client-side browser.

    sequenceDiagram
  participant B as Browser (Studio UI)
  participant S as Studio server (WS proxy)
  participant G as OpenClaw Gateway (upstream)

  B->>S: WS connect /api/gateway/ws
  B->>S: req(connect) (Gateway protocol frame)
  S->>G: WS connect upstream (url from settings.json)
  S->>G: req(connect) (injects token if missing)
  G-->>S: res(connect)
  S-->>B: res(connect)
  G-->>S: event(chat/agent/presence/heartbeat)
  S-->>B: event(...)

  9. Understand the Studio connection and reconnect behavior

    main

    Studio manages connections to runtimes using a specific lifecycle:

    1. Auto-connect: Studio only attempts to auto-connect if it has a verified last-known-good runtime state.
    2. Connect Overlay: If no verified state exists, Studio displays a connect overlay and waits for an explicit operator choice.
    3. Connection States: During gateway loading or prompt states, the 'office' remains mounted and visible behind the runtime overlay rather than performing a full page swap.
    4. Manual Connection: Initiating a manual connection will cancel any pending auto-connect or retry attempts to prevent overlap.

  10. Understand how Office Motion is derived

    main

    Office motion in the 3D scene is derived, not pushed directly. This separation ensures that transport-specific runtime details do not leak into the 3D scene logic:

    1. Events: Runtime events arrive from the gateway.
    2. Immediate Latches: reduceOfficeAnimationTriggerEvent() records immediate actions (e.g., working, thinking, user directives).
    3. Durable Holds: reconcileOfficeAnimationTriggerState() re-derives long-term holds from agent state and transcript history.
    4. Collapse: buildOfficeAnimationState() collapses the trigger state into a simplified shape.
    5. Render: RetroOffice3D converts that state into concrete paths, destinations, and actors.

  11. Configure agent state-to-animation mapping

    main

    To ensure agent operational states are reflected visually in the office, Claw3D aims for a data-driven approach. Instead of hardcoding animations, mappings should be defined via operator-facing configuration. This allows different runtimes (OpenClaw, Hermes, Vera, etc.) to map their specific semantics to visual behaviors.

    Target mappings include:

    • idle
    • writing
    • executing
    • syncing
    • error

  12. How packaged skills work in Claw3D

    main

    Claw3D uses a three-tier system to manage marketplace skills. This allows developers to create human-friendly source files that are then embedded into the codebase for safe distribution via the gateway.

    1. Source Layout: Files are authored in assets/skills/<package-id>/.
    2. Embedded Copy: Files are mirrored into src/lib/skills/packaged.ts as embedded strings. This is what the marketplace install flow actually uses.
    3. Registration: The skill is registered in src/lib/skills/catalog.ts to make it visible in the marketplace.
    4. Installation: When a user installs a skill, src/lib/skills/install-gateway.ts creates a temporary gateway agent that writes the embedded files into the user's workspace at <workspace>/skills/<skillKey>/.