DOCSEARCHbyAlgolia
Submit Docs

OpenAI ChatKit Advanced Samples

repository·main·Indexed 20 days ago

https://github.com/openai/openai-chatkit-advanced-samples

A collection of scenario-driven demos for OpenAI ChatKit, demonstrating the integration of FastAPI backends using the ChatKit Python SDK with Vite and React frontends using ChatKit.js. Included examples feature the Cat Lounge virtual caretaker, a Customer Support workflow, a Metro Map GUI interaction demo, and a News Guide, showcasing advanced patterns such as server-side tooling, streaming UI widgets, client-server interaction, and context tracking.

Tokens
4.9K
Snippets
9
Records
33
Agent score
70%

What's inside openai-chatkit-advanced-samples

  1. Overview of ChatKit demo scenarios

    main

    The repository contains several scenario-driven demos demonstrating different ChatKit capabilities:

    • Cat Lounge: A virtual cat caretaker managing energy, happiness, and cleanliness stats.
    • Customer Support: An airline concierge featuring live itinerary data, timeline syncing, and domain-specific tools.
    • News Guide: A newsroom assistant with article search, @-mentions, and page-aware responses.
    • Metro Map: A chat-driven metro planner utilizing a React Flow network of lines and stations.

  2. Key features of the News Guide implementation

    main

    The News Guide demonstrates several advanced ChatKit patterns:

    • Retrieval Tool Suite: Uses tools like list_available_tags_and_keywords, search_articles_by_tags/keywords/exact_text, and search_articles_by_author to power article list widgets.
    • Page-Aware Context: Uses the article-id request header and the get_current_page tool to allow the model to answer questions about the specific article the user is viewing.
    • Entity Tagging: Uses <ARTICLE_REFERENCE> and <AUTHOR_REFERENCE> markers. These drive get_article_by_id lookups. Users can trigger these via the @ button or by typing @ in the composer.
    • Progress Streaming: Utilizes ProgressUpdateEvent to stream progress during long-running searches or page loads, ensuring the UI remains responsive.
    • Explicit Agent Routing: Uses tool_choice to allow users to explicitly select tools like event_finder or puzzle from the composer menu.
    • Composer Commands: Implements slash commands (e.g., /find-events) that use chatkit.setComposerValue to open submenus and prefill the composer.
    • Widget Actions: Demonstrates both client-side actions (e.g., open_article buttons in article lists) and server-handled actions (e.g., view_event_details in event timelines).

  3. Cat Lounge features and capabilities

    main

    The Cat Lounge demo demonstrates several advanced ChatKit patterns:

    Server-side Tooling

    Uses server tools to read and mutate per-thread cat state. Available tools include:

    • get_cat_status
    • feed_cat
    • play_with_cat
    • clean_cat
    • set_cat_name
    • speak_as_cat

    UI Widgets and Client-Server Interaction

    • Selectable Widgets: A name suggestion workflow uses a selectable widget with client-handled actions (cats.select_name, cats.more_names) and server reconciliation for the chosen name.
    • Presentation-only Widgets: The show_cat_profile widget is streamed from the server for displaying profile cards.
    • One-way Client Effects: The server streams effects like update_cat_status and cat_say to the client to keep UI stats in sync and surface speech bubbles after tool invocations.

    Context and Automation

    • Context Tracking: Uses hidden context tags (e.g., <FED_CAT>, <PLAYED_WITH_CAT>, <CLEANED_CAT>, <CAT_NAME_SELECTED>) to provide the agent with memory of recent actions.
    • Quick Actions: Uses chatkit.sendUserMessage to send canned requests via UI buttons.
    • Composer Commands: The /care command opens a state-aware submenu (Feed/Play/Clean) and sends the selection via chatkit.sendUserMessage.
    • Image Generation: Uses ImageGenerationTool for generating cat pictures with partials.

  4. Use annotations for entity references

    main

    Annotations allow the assistant to link specific parts of its text to underlying data entities.

    Example: In the Metro Map demo, a plan_route tool renders stations as inline annotations. When a user clicks an annotation, the client can trigger an action, such as panning a map canvas to that specific station.

  5. Use client tool calls to mutate or fetch UI state

    main

    Client tools allow the agent to interact with the client-side state of the application. This enables the agent to 'see' what the user is doing on the UI and react accordingly.

    Example:

    • In the Metro Map demo, the client tool get_selected_stations pulls currently selected nodes from a canvas so the agent can incorporate that selection into its response.

  6. Implement entity tags (@-mentions)

    main

    Enable users to reference specific entities (like articles, authors, or stations) using @ mentions in the composer.

    Workflow:

    1. Search: The composer provides an entity search list.
    2. Conversion: Tagged entities are converted into model-readable markers (e.g., <ARTICLE_REFERENCE>, <STATION_TAG>) by the thread_item_converter.
    3. Resolution: The agent uses tools (e.g., get_article_by_id) to resolve these markers into full content for context.

  7. Use server tool calls to retrieve application data

    main

    Agents can use function tools to pull real-time application data for inference. This allows the model to ground its responses in current state.

    Examples of patterns:

    • Retrieval: Using tools like get_cat_status, get_article_by_id, or get_map to fetch specific entities.
    • Search: Using tools like search_articles_by_tags/keywords/exact_text or list_stations to find relevant data.
    • Stateful Context: Prepending snapshots (e.g., <CUSTOMER_PROFILE>) to the conversation or using tools that interact with a state manager (e.g., AirlineStateManager) to manage per-thread data.

  8. Use widgets with and without actions

    main

    Widgets are UI components streamed from the server to present structured information or interactive elements.

    Widgets without actions

    Used for pure presentation (e.g., a profile_card_widget showing cat stats).

    Widgets with actions

    Used to capture user input via specific action payloads.

    • Client-handled actions: The client intercepts the action via a callback like handleWidgetAction in ChatKitPanel.tsx (e.g., cats.select_name).
    • Server-handled actions: The action is processed on the backend to update state, persist data, or stream a new version of the widget (e.g., flight.select or view_event_details).

  9. Explore Metro Map features and tool patterns

    main

    The Metro Map implementation showcases several advanced ChatKit patterns:

    Server-side Tool Calls (Grounding)

    Tools like get_map, list_lines, list_stations, get_line_route, and get_station are used to keep the agent grounded in the latest network data.

    Client-side Tool Calls (UI State)

    • Selection Awareness: The agent calls get_selected_stations to retrieve the user's current canvas selection. This is handled via onClientTool in the frontend and backend.
    • UI Mode Switching: A location_select_mode client effect can flip the UI into a specific placement mode for user interaction.

    Widget and Effect Patterns

    • Streaming Widgets: show_line_selector streams a clickable line.select widget. The server manages state (e.g., stashing <LINE_SELECTED>) to drive subsequent actions.
    • Progress Updates: The application streams progress events during heavy operations, such as the initial map fetch.
    • Entity Tags: Typing @ in the composer adds <STATION_TAG> content, allowing the agent to recognize entities and allowing users to click tags to focus stations on the canvas.
    • Inline Annotations: Route planning responses attach entity sources as inline annotations to keep the canvas focused on relevant stops.

  10. Use composer commands and tool choice

    main

    Enhance the chat composer with specialized commands and manual tool selection.

    Composer Commands

    Implement slash commands (e.g., /care or /find-events) that open submenus or prefill prompts using chatkit.setComposerValue() and chatkit.sendUserMessage().

    Tool Choice (Composer Menu)

    Configure composer.tools to show specific agent options in the UI. This allows users to force a specific agent (e.g., event_finder) by setting the tool_choice on the request.

  11. Implement fire-and-forget client effects

    main

    Client effects are used to sync UI state or trigger visual changes (like speech bubbles or canvas updates) without requiring a full model response cycle. These are often streamed from the server.

    Key patterns:

    • UI Syncing: Using effects like update_cat_status or customer_profile/update to ensure side panels or status indicators mirror the latest data.
    • Canvas/Map Updates: Streaming effects like location_select_mode or add_station to immediately update a visual canvas (e.g., a metro map) after an agent action.

  12. Quickstart the Cat Lounge demo

    main

    To run the Cat Lounge virtual cat caretaker demo, follow these steps:

    1. Export your OpenAI API key to the environment:
      export OPENAI_API_KEY='your-api-key-here'
    2. Run the application from the repository root:
      npm run cat-lounge
      Alternatively, you can run it from the example directory:
      cd examples/cat-lounge && npm install && npm run start
    3. Access the frontend at http://localhost:5170.
    npm run cat-lounge