Chainlit Cookbook

This application provides a question-answering service for PDF documents. It uses a RAG (Retrieval-Augmented Generation) pattern:

1. PDF Processing: PDF files are chunked into text segments.
2. Embedding & Indexing: Chunks are converted into embeddings using Azure OpenAI's text-embedding-ada-002 and stored in a Pinecone vector store.
3. Question Answering: When a user asks a question, the app retrieves relevant chunks from Pinecone and uses an Azure OpenAI language model (e.g., gpt-35-turbo-16k) to generate an answer with source transparency.

The application implements a RAG (Retrieval-Augmented Generation) workflow using Chroma as a vector store and OpenAI for generation. The core logic in app.py follows these steps:

1. PDF Processing (process_pdfs): Chunks PDF files into text segments, generates embeddings, and stores them in Chroma.
2. Document Indexing: Uses SQLRecordManager to track document writes to ensure efficient indexing.
3. Question Answering (on_message): Retrieves relevant chunks from Chroma based on user queries and uses OpenAI to generate answers with source citations.

This cookbook provides a pattern for building realtime copilots using Chainlit and the OpenAI Realtime API. It enables a multimodal experience where users can interact with an assistant via both voice (audio) and text simultaneously.

Key capabilities include:

• Realtime Python Client: Utilizes the OpenAI realtime API beta implementation.
• Multimodal Interaction: Supports simultaneous speaking and writing to the assistant.
• Tool Calling: Allows the assistant to execute tasks with visible output in the Chainlit UI.
• Visual Presence: Provides UI cues to indicate the assistant's state (e.g., listening or speaking).

This project uses a specific pattern to bridge Anthropic's function calling with Chainlit's UI:

• Tool Execution: Tool calls are handled using the @cl.step(type="tool") decorator, which allows the tool execution to appear as a distinct, structured step in the Chainlit chat interface.
• Routing: A call_tool function is used to route the model's requested tool call to the appropriate Python function.
• Lifecycle: The chat session is initialized via @cl.on_chat_start and user interactions are processed through @cl.on_message.

The integration uses custom subclasses of AutoGen agents to bridge the gap between AutoGen's logic and Chainlit's interface:

• ChainlitAssistantAgent: A subclass of AssistantAgent designed to send messages to other agents within the Chainlit context.
• ChainlitUserProxyAgent: A subclass of UserProxyAgent that handles user input and facilitates sending messages to other agents.

Tip: If you need to modify agent behavior, you can monkey-patch methods of the Agent class directly instead of creating a new subclass.

The application in app.py uses the following core functions to manage the chat lifecycle:

• start_chat: Initializes the chat session and configures the Claude avatar.
• call_claude: Handles the communication with the Anthropic API, sending the user's query and streaming the model's response back to the Chainlit interface.
• chat: The primary message handler that receives user input and triggers the call_claude workflow.

The Resume-Chat application uses specific Chainlit lifecycle hooks to manage conversation state and allow users to pick up where they left off.

• on_chat_start(): Triggered when a new session begins. It initializes the ConversationBufferMemory and the Runnable pipeline for the current session.
• on_chat_resume(thread: ThreadDict): Triggered when a user resumes an existing conversation. This function uses the provided ThreadDict to repopulate the memory with the previous conversation history, ensuring the AI maintains context.
• on_message(message: cl.Message): Handles the actual interaction loop. It takes the incoming user message, processes it through the langchain Runnable pipeline, and streams/sends the response back to the user.

The application utilizes several Chainlit hooks and handlers to manage the lifecycle of the chat session:

• on_chat_start: Initializes the Chainlit session and prepares necessary components for the QA workflow.
• on_message: The primary event handler that receives user input, performs retrieval, and returns the generated answer.
• PostMessageHandler: A callback handler used to post retrieved document sources as a Chainlit element for transparency.

This implementation uses a single LangGraph workflow to serve all user requests. Memory is managed at the thread level, which is the recommended approach by LangGraph.

By relying on the context's thread_id, the application can persist conversations across different sessions. In this specific example, MemorySaver is used as the checkpoint saver. Because MemorySaver is an in-memory checkpoint saver, all conversation history will be lost if the Chainlit application is restarted.

To test the memory functionality:

1. Login with admin/admin.
2. Start a conversation.
3. Switch to a different conversation and then back to the previous one; the assistant should retain awareness of previous questions within that thread.

The application implements a multi-iteration tool loop using the OpenAI Responses API with stream=True. The loop processes an event stream containing:

1. Assistant text tokens: Standard token-by-token text streaming.
2. Function call creation events: Signals that the model intends to use a tool.
3. Function argument deltas: Incremental updates to function arguments (e.g., function_call_arguments.delta), which are used to live-render generated code (like Python) in the UI before execution.

Once a tool is called, the output is returned via function_call_output, and the loop continues until the model completes its response.