AISuite

repository·main·Indexed 12 days ago

https://github.com/andrewyng/aisuite

A lightweight uniform access layer for LLMs providing a unified Chat Completions API and a high-level Agents API across multiple providers including OpenAI, Anthropic, Groq, and Mistral. It supports tool-calling, autonomous task execution, and Automatic Speech Recognition (ASR) via Deepgram and OpenAI Whisper. Includes a React/TypeScript chat application for real-time model comparison and a local coding agent CLI called aisuite-code.

Tokens
100.6K
Snippets
290
Records
449
Agent score
97%

What's inside AISuite

  1. What is OpenWorker?

    main

    OpenWorker is an open agent harness designed for automating daily tasks via a desktop application. It is built on top of the Agent API and Chat Completions layers.

    Key features include:

    • Multi-folder file access: Per-folder read-only or read-write grants.
    • Connectors & tools: Browser automation, integrations, and MCP servers.
    • Artifacts: Support for Markdown, images, PDF, CSV, spreadsheets, and Office files.
    • Automations: Scheduled runs that function as continuous conversations.
    • Model Support: Use your own API keys for OpenAI, Anthropic, Gemini, or local models via Ollama.
  2. Understand OpenCoworker GUI UX Decisions

    main

    The OpenCoworker GUI is designed around a connector-agnostic source model, meaning the user experience remains consistent whether a connector is a standalone application or an MCP (Model Context Protocol) server. Key design principles include:

    • Session-Centricity: Conversations are organized into sessions where models and tools are fixed per session.
    • Connection Hierarchy: Connections can be enabled at a persona level or a session level.
    • Modular UI: The interface uses a dual-layout left navigation, a 'Session settings' drawer (replacing the old Sources bar), and a 'Progress + Artifacts' rail (renamed to 'Files').
    • Persona-Driven Workflow: Users start new sessions by selecting a persona, which may include specific recommended tools or connections via a recommends manifest.
  3. Configure standing scoped approvals for automations

    main

    To prevent recurring automations from being blocked by manual approval prompts (e.g., for a tool like send_message), OpenCoworker uses standing scoped approvals. These are narrow, remembered rules that allow an automation to execute specific actions without human intervention.

    A rule is defined by the triplet: tool + target + owner (task id). No wildcards are supported in v1.

    There are two ways to mint a rule:

    1. Creation Consent Card: When creating a scheduled task via create_scheduled_task, the UI surfaces the required permissions. Users can grant access to specific tools/targets at this stage.
    2. 'Allow every time': On a recurring run's approval card, users can select Allow every time to persist the permission to the specific ScheduledTask record.

    Safety Invariants:

    • Rules are never offered for risk=exec or destructive tools; these always require manual confirmation.
    • Rules are additive and never grant full-access upgrades silently.
    • Rules are stored on the ScheduledTask record and can be revoked on the task detail page.
  4. Understand the connection hierarchy and effective enablement

    main

    Connector availability is determined by a three-layer hierarchy. A connector is only 'effective' (active and delivering messages) if it passes all three checks:

    1. account-connected: Does a valid profile with credentials exist? (Managed via connector_list[].connected)
    2. persona-default-enabled: Is the connector enabled by default for the current persona? (User-editable in persona settings)
    3. session-override: Has the user explicitly enabled or disabled this connector for the current specific session?

    Logic for Effective Status: Effective = connected AND (session_override if present ELSE persona_default)

    Runtime Impact:

    • Inbound: Messages from a connector are only delivered to a session if the connector is effective-enabled. Muted connectors are buffered for catch-up.
    • Outbound/Tools: Tools associated with a connector are only exposed to the agent if the connector is effective-enabled for that session.
  5. How Inbox interaction works via messaging buttons

    main

    To avoid the brittle process of users typing correlation tokens (like [ocw:id]) into a chat, AISuite uses interactive UI elements (e.g., Slack Block Kit buttons) to resolve Inbox items.

    1. Rendering: The adapter renders a card with buttons for discrete choices (e.g., Approve/Deny or specific ask_user options). The item_id is embedded inside the button's value payload.
    2. Interaction: When a user clicks a button, the platform sends an interaction payload to the AISuite gateway.
    3. Resolution: The gateway routes the payload to manager._on_interaction, which calls inbox.resolve(id, choice). This resolves the specific await in the agent's engine, allowing it to resume with the selected value.

    Constraint: Free-text answers are currently not supported via messaging buttons to maintain high correlation fidelity; users requiring free-text input are prompted to "open the app to respond" in the main AISuite interface.

  6. Use the Inbox to manage agent attention and approvals

    main

    The Inbox is the central queue for human attention. It decouples agent progress from your immediate presence by routing items that require human intervention to a single location. This acts as a lightweight safety mechanism: consequential actions can be gated by requiring an Inbox item to be resolved before the agent proceeds.

    Inbox Item Types

    • Approval: Requires an actionable allow or deny (e.g., "Ops Coworker wants to restart the service").
    • Question: Requires a free-text answer from the user to allow the agent to continue.
    • Notification: An FYI or completion notice (e.g., "your report is ready") that deep-links back to the originating session or artifact.

    Messaging connectors (Slack, Telegram, etc.) act as transports for these items; they deliver the notification and allow user replies to flow back as steering messages.

  7. Understand the Persona-based UI and Session States

    main

    The AISuite GUI organizes functionality into Personas (e.g., Cowork, Code, Ops). The interface uses several visual indicators to communicate the state of sessions and pending tasks:

    • Attention Badges (Amber): Indicates the count of pending items in the Inbox for a specific session. These items are typically generated by Unattended sessions.
    • Liveness Dot: Indicates the real-time activity of a session:
      • Green Pulse: The session is working (an in-flight turn via is_running).
      • Grey: The session is idle or sleeping (a self-wake is pending via wakes.pending).
    • Inbox Integration: Inbox items can be answered in-context. Clicking an Inbox item chip (persona icon + session title) opens the originating session, where the pending item is rendered inline above the composer (InboxItemCard). Resolving the item in the session context resolves it for the Inbox.
    • Sidebar Grouping: Projects and groups are organized by their persona family (e.g., familyOf(id) === "code").
  8. Privacy and Security for the Email connector

    main

    When using the email connector, keep the following security considerations in mind:

    1. Credential Safety: Your app_password is stored locally in the SecretStore. It is used only to connect to your mail server and is never sent to any LLM provider or to the AISuite developers.
    2. Data Exposure: While credentials are safe, the content of the emails the agent reads is sent to your configured LLM provider as context. This is consistent with how agents handle other files or documents.
    3. Access Control: An app password grants the agent full access to your mailbox (reading, searching, and sending). If you stop using the service, you should revoke the app password via your email provider's security settings (e.g., Google or Apple account settings).
  9. How long-running agents work via self-wake tools

    main

    In AISuite, long-running agents do not run as continuous processes. Instead, they use a suspend/resume (event-driven) model. An agent sleeps at near-zero cost and is re-invoked by the scheduler only when a trigger occurs. This allows for many concurrent long-running agents without high resource costs.

    Available Triggers

    1. Timer: Using sleep_for(duration) or sleep_until(time) to schedule a future resumption.
    2. On-completion: Using wake_on(job_id) to resume when a specific backgrounded command or job exits. The exit code and output are provided to the session upon resumption.
    3. On-message: A user sending a steering message into the session wakes it.
    4. On-event (Planned): Future support for webhooks or connectors (e.g., an email arriving or a PR opening) to trigger a session.
    # Conceptual examples of self-wake triggers
    sleep_for(3600)          # Timer trigger
    sleep_until(timestamp)   # Timer trigger
    wake_on(job_id)          # On-completion trigger
  10. Understand the Tool Catalog and Risk Classes

    main

    AISuite uses a centralized Tool Catalog to manage tool permissions via declared Risk Classes. This replaces hardcoded tool sets with a more granular system.

    Risk Classes

    Instead of checking for specific tools (like SHELL_TOOL), the system evaluates the risk class of a tool. The primary risk classes are:

    • read
    • write_local
    • exec
    • external (This class acts as the hook for routing tasks to the Inbox when in Unattended mode).

    Tool Identifiers

    Tools in the catalog are identified using specific URI-like patterns:

    • connector:<x>
    • secret:<y>

    Risk Evaluation Logic

    The effective risk of a tool is determined by the following hierarchy:

    1. User-local overrides: Users can define local overrides (matching by server or glob patterns) to downgrade risk. These are strictly user-local and are never carried by a persona.
    2. Catalog defaults: If no override exists, the risk class defined in the catalog is used.
  11. Understand the Generic Account Layer for Connectors

    main

    AISuite uses a generic account layer to support multiple accounts for the same SaaS provider (e.g., multiple Notion workspaces or PostHog projects). This layer is managed via coworker/connectors/accounts.py and uses a specific naming convention for profiles in secrets:

    • Profile Key: <connector>:account:<account_id>
    • Default Pointer: <connector>:default (contains {"default_account": <id>} and connector-wide settings, but never contains tokens).

    Key Operations:

    • resolve(secrets, connector, account=""): Returns the (profile_key, profile) for the specified account, the default account, or the sole available account. Returns {} if nothing is connected.
    • add_account(secrets, connector, account_id, fields): Adds a new account profile and sets it as the default if it is the first account added.
    • set_default(secrets, connector, account_id): Sets a specific account as the default.
    • disconnect_account(secrets, connector, account_id): Removes the account; if it was the default, the pointer is also deleted.

    Account ID Mapping:

    ConnectorAccount ID SourceDisplay Name
    notionworkspace_id (from OAuth)workspace_name
    attioworkspace_id (from GET /v2/self)workspace name
    posthogproject_idhost · project
    mixpanelproject_idproject_id
    amplitudeLast 6 chars of api_keykey …abc123
    apolloAccount emailemail
    hunterAccount emailemail
  12. Extend aisuite by adding a Provider

    main

    To add a new LLM provider, implement a lightweight adapter following the naming convention for automatic discovery:

    1. Module file: Name it <provider>_provider.py (e.g., myprovider_provider.py).
    2. Class name: Name the class <Provider>Provider (e.g., MyproviderProvider).

    This allows the system to automatically load the new integration.

    # providers/openai_provider.py
    class OpenaiProvider(BaseProvider):
        ...