DOCSEARCHbyAlgolia
Submit Docs

GenUI Documentation

repository·main·Indexed 23 days ago

https://github.com/flutter/genui

An experimental Flutter SDK that transforms LLM text outputs into interactive, JSON-driven graphical user interfaces using existing widget catalogs. It includes the A2UI protocol core, genai_primitives for technology-agnostic AI data structures, and developer tools like Composer for generating and editing A2UI surfaces.

Tokens
14.4K
Snippets
41
Records
82
Agent score
82%

What's inside GenUI

  1. Overview of GenUI SDK for Flutter

    main

    The GenUI SDK for Flutter is an experimental library designed to replace static text responses from Large Language Models (LLMs) with dynamic, interactive, and graphical user interfaces.

    Instead of generating code at runtime, GenUI uses a JSON-based format to compose UIs from your existing Flutter widget catalog. This allows for a high-bandwidth interaction loop where user interactions with UI elements (like sliders or checkboxes) update a client-side data model that is fed back to the AI agent, influencing its next steps.

  2. Overview of genai_primitives

    main
    The genai_primitives package provides technology-agnostic primitive types and data structures designed for building Generative AI applications in Dart. It serves as a foundational layer for the genai ecosystem, ensuring consistency and interoperability between different AI providers through core definitions like ChatMessage, Parts, and ToolDefinition.

  3. Overview of genui

    main
    genui is a Flutter package designed for building dynamic, conversational user interfaces powered by generative AI models. Instead of static, predefined UIs, genui allows an AI to construct the interface in real-time based on the conversation context. This enables highly flexible and interactive user experiences where the UI adapts to the user's needs.

  6. Features of Composer

    main

    Composer includes the following capabilities:

    • Create: Describe a UI in plain text and use Gemini to generate an A2UI surface from that description.
    • Gallery: Browse and preview pre-built sample surfaces.
    • Components: View the full catalog of available built-in components.
    • Surface Editor: Edit A2UI JSONL and data models with a live preview.

  7. Build and validate schemas with json_schema_builder

    main

    The json_schema_builder package provides a fluent API to programmatically construct JSON schemas and validate data against them.

    Key features demonstrated in the library include:

    • Nested Objects: Using ObjectSchema to define structured data.
    • Lists: Using ListSchema with constraints like unique items.
    • Strings: Using StringSchema with regular expression patterns.
    • Integers: Using IntegerSchema with range constraints.
    • Combinators: Using oneOf to enforce that data matches exactly one of several defined schemas.
    • Validation: Running a validator against data to produce detailed, human-readable error messages for invalid inputs.

  8. Handle actions in CatalogItem widgetBuilder

    main

    When a CatalogItem schema includes an action callback, you must resolve the context and dispatch a UserActionEvent within the widgetBuilder.

    To do this:

    1. Extract the action details from the parsed data.
    2. Use resolveContext(itemContext.dataContext, contextDefinition) to fetch variables for the action.
    3. Call itemContext.dispatchEvent(...) with a UserActionEvent containing the event name, the sourceComponentId (from itemContext.id), and the resolvedContext.

    This ensures that the AI agent or system receiving the event has the correct, resolved state context.

    // Inside widgetBuilder...
final JsonMap resolvedContext = await resolveContext(
  itemContext.dataContext,
  contextDefinition,
);
itemContext.dispatchEvent(
  UserActionEvent(
    name: name,
    sourceComponentId: itemContext.id,
    context: resolvedContext,
  ),
);

  9. How the genui interaction cycle works

    main

    The interaction between the user, the AI, and the UI follows a specific loop managed by the Conversation, SurfaceController, and A2uiTransportAdapter:

    1. User Input: The user provides a prompt via conversation.sendRequest().
    2. AI Invocation: The Conversation triggers the A2uiTransportAdapter.onSend method.
    3. Stream Handling: The application's onSend implementation calls the LLM and pipes response chunks into A2uiTransportAdapter.addChunk().
    4. Parsing: The A2uiTransportAdapter uses A2uiParserTransformer to parse chunks into TextEvents or A2uiMessageEvents.
    5. UI State Update: The SurfaceController processes these messages to update the DataModel.
    6. UI Rendering: Surface widgets listening to the SurfaceController rebuild automatically to reflect the new state.
    7. User Interaction: User actions (like button clicks) trigger events captured by the SurfaceController, which then emits ChatMessage events via onSubmit.
    8. Loop: The Conversation listens for onSubmit and automatically triggers a new request to the AI, continuing the cycle.

  10. Connecting GenUI to an AI agent

    main

    The GenUI framework is backend-agnostic and can work with any AI SDK (e.g., firebase_ai or dartantic_ai). To bridge the gap between an AI response and the UI, you use adapters.

    • Standard AI Responses: Use adapters like A2uiTransportAdapter to ingest AI responses and render them.
    • A2UI Protocol Servers: If you are using a custom agent server that implements the A2UI protocol, use the genui_a2a package to connect via the A2uiAgentConnector.

  11. Understand the core workflow of a GenUI chat application

    main

    A GenUI chat application follows a specific lifecycle to interleave text messages with AI-generated UI surfaces:

    1. Initialization: Create a SurfaceController to manage the state and lifecycle of dynamic UI surfaces.
    2. User Input: Capture user text via a standard TextField and add it to the local conversation history.
    3. AI Interaction: Send the user's message to an AiClient. The client returns a stream of A2uiMessage objects.
    4. Surface Management: Pipe the A2uiMessage stream into the SurfaceController. This controller manages the UiDefinition and tracks new surfaces.
    5. Dynamic Rendering: Listen to SurfaceController.surfaceUpdates (or A2uiTransportAdapter streams). When a new surface is detected, render a Surface widget to display the AI-generated UI.

    In this minimal implementation, the AI uses the default coreCatalog provided by genui, allowing it to generate basic widgets like Text, Column, and ElevatedButton without requiring a custom widget catalog definition.

  12. Core types in genai_primitives

    main

    The package defines several foundational classes used to structure GenAI interactions:

    • Part: The base type for message parts. You can extend this to define custom part types.
    • Parts: A collection of Part instances that includes utility methods for managing multiple parts.
    • StandardPart: A sealed class extending Part with a fixed set of implementations. This is used by ChatMessage to maintain cross-provider compatibility.
    • ChatMessage: A class representing a chat message, designed to be compatible with most GenAI providers.
    • ToolDefinition: A class used to define a tool that an LLM (Large Language Model) can invoke.