LobsterAI Documentation

repository·main·Indexed 26 days ago

https://github.com/netease-youdao/lobsterai

LobsterAI is an all-scenario desktop office assistant Agent built by NetEase Youdao. It operates in local environments to automate workflows by interacting with files, terminal commands, browsers, and IM channels. The documentation covers installation for macOS, Windows, and Linux, development guides for the Electron-based architecture and OpenClaw runtime, and detailed setup for integrated skills such as IMAP/SMTP email and web search.

Tokens
217.9K
Snippets
376
Records
1.1K
Agent score
91%

What's inside LobsterAI

  1. Overview of LobsterAI Usage Analytics Reporting

    main

    LobsterAI implements a unified usage analytics reporting system to track application installation, core feature entry points, and key interactions. This data helps maintainers optimize features, improve compatibility, and prioritize development.

    Key tracked data points include:

    • Skills, MCPs, and Expert Kits used.
    • Model sources and model types.
    • Settings, Agents, and Scheduled Tasks.
    • Session input box usage, message interactions, and artifact/browser previews.

    Core Principles:

    • Unified Entry Point: All logs are sent through a centralized service that handles log service addresses, common parameters, user identifiers, timestamps, and environment information.
    • Non-Intrusive: Analytics reporting failures must not impact the application's original functionality.
    • Privacy & Security: The system does not upload prompt text, conversation content, file contents, full file paths, full URLs, keys, API Keys, MCP environment variables/headers, IM credentials, or prompt hashes. It also avoids high-frequency tracking like mouse movements or streaming tokens to ensure performance.
  2. Overview of LobsterAI Features

    main

    LobsterAI is a desktop-level Agent designed for real-world work environments. Key features include:

    • Desktop Cowork Sessions: Execute long-running tasks around local projects and files with real-time progress streaming and user approval for sensitive actions (file ops, terminal, network).
    • Multi-Agent Workflows: Create custom Agents with independent identities, models, skills, and IM bindings.
    • Expert Kits: Reusable workflows that bundle specific capabilities and reference information.
    • Skills: 28 built-in skills (e.g., Web Search, Excel, PowerPoint, PDF, Browser Automation, Email, etc.) configured in SKILLs/skills.config.json.
    • MCP Services: Integration with external tools and data via Model Context Protocol.
    • Scheduled Tasks: Create periodic tasks (e.g., daily news summaries, website monitoring) via natural language or UI.
    • IM Remote Control: Connect to the desktop Agent via WeChat, DingTalk, Feishu/Lark, Telegram, Discord, Email, and more.
    • Artifacts: Preview and manage generated HTML, SVG, images, videos, Mermaid diagrams, and code.
    • Local Memory: Uses local SQLite for session data and files like MEMORY.md, USER.md, and SOUL.md for cross-session context.
  3. Overview of the LobsterAI Scheduled Task System

    main

    The LobsterAI scheduled task system is an end-to-end automation framework spanning three layers: Renderer (UI), Main Process (IPC/Business Logic), and the OpenClaw Gateway (Scheduling Engine).

    Users can create scheduled tasks via the UI, IM chat, or Cowork sessions. The tasks are scheduled and executed by the OpenClaw Cron engine, and results are delivered back to users via IM channels or Webhooks.

    Core Design Principles:

    • OpenClaw Driven: All scheduling, execution, and delivery are handled natively by the OpenClaw Gateway. LobsterAI manages task CRUD and UI display but does not handle message delivery logic.
    • Policy Pattern: Different task sources (UI, IM, Cowork, Legacy) use independent policy classes to control default parameters, binding relationships, and read-only fields.
    • Origin Inference: The system uses the format of the sessionKey to reverse-infer the task source and execution binding, ensuring compatibility with legacy data.
    • Streaming Polling: A 15-second polling mechanism is used to push real-time task status updates from OpenClaw to the UI.
  4. Overview of Office and PDF Sharing Design

    main

    LobsterAI supports sharing Office documents (Word, PPT, Excel) and PDFs via the lobsterai-server. The system allows users to share files through a public page (/s/{shareId}/) protected by share codes, featuring content auditing, access statistics, and administrator previews.

    Key technical details:

    • Upload Path: Clients package content into a zip and upload via /api/html-shares.
    • Storage: Files are stored in html_shares / html_share_files and uploaded to NOS.
    • Access: Public content is served via /s/{shareId}/content/ using a same-origin proxy to avoid exposing NOS URLs.
    • Rendering: The web preview uses the same libraries as the client: docx-preview for .docx, pdfjs-dist for .pdf, pptx-preview for .pptx, and xlsx for spreadsheet formats.
  5. Understand HTML Share Access Modes

    main

    The HTML sharing feature supports two distinct access modes for sharing artifacts:

    1. code (Share Code Mode): Requires a specific share code to access the content. The server returns the plaintext share code upon creation.
    2. public (Public Access Mode): The content is accessible via the link without a share code. No share code credentials are created in this mode.

    When creating a share without specifying a mode, the system defaults to code mode.

  6. Understand the Static Site Deployment Optimization Design

    main

    The project is optimizing how local services are shared and deployed. Instead of wrapping all projects in a Node.js service (which consumes runtime resources and increases cold starts), projects that produce pure static files (like Vite, React, Vue, or Next.js static exports) will now use a dedicated Static Deployment path.

    Key Improvements

    • Resource Efficiency: Static sites use a dedicated share host instead of a Node runtime, preserving dynamic service quotas for actual backend services.
    • Improved UX: Supports SPA history fallback (routing), root path resources, and static asset caching.
    • Security: No user code is executed on the server for static deployments.

    Deployment Logic Summary

    Project TypeDeployment PathExampleDetails
    Static SiteStatic DeploymentVite, React, Vue, Astro, Next static exportUploads dist, build, or out directories to a static host.
    Node ServiceNode Service DeploymentExpress, NestJS, Next SSR, Nuxt SSRUses existing Node runtime and process management.

    Supported Scenarios

    • Vite / React / Vue / Angular / Svelte: Detects build scripts and uploads the resulting static directory (e.g., dist/).
    • Next.js Static Export: If out/index.html exists and no .next/standalone/server.js is present, it is treated as a static_site.
    • SPA Routing: Supports history router fallback. If a requested path (e.g., /settings/profile) is not found in storage, the server returns index.html to let the frontend router handle it (when spaFallback = true).
  7. Understand HTML Share Access Modes (Public vs. Code)

    main

    LobsterAI supports two distinct access modes for HTML artifact sharing:

    1. Code Mode (code): The default mode. Users receive a link and a specific share code. Accessing the link requires the user to input the correct share code.
    2. Public Mode (public): The link is directly accessible to anyone without requiring a share code.

    Key behaviors during mode switching:

    • Switching code $\rightarrow$ public: The existing share code is preserved in the system but no longer required for access.
    • Switching public $\rightarrow$ code: If a historical share code exists and is decryptable, it is reused. If no historical code exists, a new share code is generated.
    • Content Updates: Switching the access mode without updating the artifact content does not trigger a contentUpdatedAt change or reset the moderationStatus. However, updating the content and switching the mode simultaneously will trigger content updates and moderation resets.
  8. Understand LobsterAI Cowork Steer behavior and states

    main

    The Cowork Steer feature allows users to provide real-time adjustments to an active agent session. It operates through a specific state machine to manage how user inputs are handled during active runs:

    • Idle: Default state.
    • SteerComposing: User is actively typing a steer instruction.
    • PendingSteer: The instruction has been submitted and is waiting for the runtime to accept it.
    • Accepted: The runtime has confirmed the steer; the input is included in the current or next turn.
    • RejectedSteer: The runtime determined the current turn is not steerable. The instruction is moved to a rejected area and will be automatically submitted as a standard follow-up once the current turn completes.
    • SendingAfterInterrupt: The user chose to 'Interrupt and Send', which stops the current task and immediately starts a new turn with the merged steer messages.
  9. Understand WeChat Login State Persistence in LobsterAI

    main

    LobsterAI manages WeChat connections using a scan-to-login model. Unlike configuration-based IMs (Feishu, DingTalk) that use stable credentials like appId or appSecret, WeChat uses a botToken stored in a specific local directory.

    Important Directory Boundaries:

    • LobsterAI Managed State: %APPDATA%\LobsterAI\openclaw\state\openclaw-weixin\accounts. This directory contains the botToken for each accountId. This directory must be preserved during updates to maintain login sessions.
    • Standalone OpenClaw State: %USERPROFILE%\.openclaw\openclaw-weixin\accounts. LobsterAI does not manage or read from this directory to avoid account cross-contamination or security boundary issues.
  10. Understand the macOS Atomic App Update Design

    main

    The macOS application update process (specifically installMacDmg() in src/main/libs/appUpdateInstaller.ts) has been redesigned to move from a non-atomic 'delete-then-copy' sequence to an atomic 'staged-copy-and-rename' sequence. This prevents the application from being deleted from the disk if the copy process fails (e.g., due to insufficient disk space or read errors).

    Key Improvements:

    • Staged Copy: The new version is copied to a hidden staging directory before the old version is touched.
    • Atomic Swap: The old version is renamed to a backup, and the new version is renamed into its place using rename(2), which is atomic on the same filesystem.
    • Automatic Rollback: If the swap fails, the system automatically restores the backup to the original location.
    • Privilege Escalation Safety: The elevated (sudo) update path uses a single composite shell command to ensure the entire sequence (staging, backup, swap, and rollback) is executed atomically within a single password prompt session.
  11. Understand LobsterAI Cowork Context Compaction

    main

    LobsterAI Cowork implements context compaction to manage long conversations. It provides users with visibility into context usage via a visual indicator (a ring) and allows for both automatic and manual context compaction to prevent context window overflow.

    Key Features:

    • Context Usage Monitoring: Displays a ring indicator showing the percentage of the context window used. Tooltips provide specific token counts.
    • Automatic Compaction: Triggered by compactionCheckpointCount increments, inserting a system notification into the chat stream.
    • Manual Compaction: Users can trigger compaction via the sessions.compact API. During manual compaction, the UI shows a loading state, and successful compaction refreshes the usage percentage and inserts a notification after the current message and media cards.
    • Model Switching: Changing models automatically recalculates context usage based on the new model's context window.
    • Compaction Metadata: Detailed information about compaction events (summary, tokens before/after, pre/post compaction metadata) is available via the OpenClaw gateway.
  12. Understand LobsterAI Continuity Capsule and OpenClaw Compaction

    main

    LobsterAI enhances the existing OpenClaw compaction process by adding a continuity capsule layer. While OpenClaw's native compaction is responsible for compressing old chat history, the LobsterAI capsule is designed specifically to preserve the state of coding tasks.

    Key components:

    • OpenClaw summary: Compresses old dialogue.
    • LobsterAI capsule: Preserves coding task state (objectives, constraints, files, decisions, next steps).
    • Top-K Evidence / RAG evidence: Retrieves original details on demand.

    The capsule bridge injects the capsule into the model context via chat.send.message without requiring changes to the OpenClaw API or its summary generation logic.