FastAPI MCP LangGraph Template

The template uses the following core technologies:

• FastAPI: Python backend API.
• SQLModel: Python SQL database interactions (ORM + Validation), wrapping SQLAlchemy.
• LangGraph: Customizable Agentic Orchestration with native streaming and persisted chat history/state management.
• MCP (Model Context Protocol): Standardizes how apps provide context to LLMs, preventing LLM provider lock-in.
• LangFuse: LLM Observability and Metrics.
• Pydantic: Data Validation and Settings Management.
• Supabase: Database RBAC, utilizing PostgreSQL and PGVector for vector storage.
• Nginx: Reverse Proxy.
• Docker Compose: Orchestration for development and production.

To handle the stateless nature of LLMs, this template uses LangGraph's persistence capabilities via AsyncPostgresSaver. This abstraction manages chat history and metadata serialization/deserialization automatically.

Instead of manually injecting chat history into every query, you only need to provide a thread_id. LangGraph uses this ID to retrieve the relevant state and context for the conversation, enabling seamless multi-turn interactions and faster iteration on agentic workflows.

# Concept: Using a thread_id to maintain state
config = {"configurable": {"thread_id": "unique_session_id_123"}}

# The graph automatically retrieves history associated with this thread_id
await graph.ainvoke(input_data, config=config)

Langfuse is used in this template to provide traces, evaluations, prompt management, and metrics. It allows you to debug and improve your LLM application by visualizing the lifecycle of an agentic interaction.

When inspecting traces in Langfuse for a Math Agent, you can observe the following flow:

1. User to Agent: The initial natural language input from the user (e.g., What is 1 + 1?).
2. Agent to Tool: The agent's decision to invoke a tool, including the structured arguments passed to it (e.g., args: { a: 1, b: 1 }).
3. Tool to Agent: The result returned by the tool after execution (e.g., 2).
4. Agent to User: The final natural language response generated by the agent (e.g., 1 + 1 = 2).

Langfuse also provides visibility into the full chat history, latency, and cost associated with each node in the LangGraph workflow.

For a containerized development environment with IntelliSense and a debugger for the FastAPI server, use the VSCode Devcontainer setup.

1. Modify compose-dev.yaml (Optional): If you plan to use the FastAPI debugger, replace the api entrypoint in ./compose-dev.yaml to allow the container to stay alive:

   api:
     image: api:prod
     build:
       dockerfile: ./backend/api/Dockerfile
     entrypoint: bash -c "sleep infinity"
     env_file:
       - ./envs/backend.env

2. Open in Container:

   • Run code --no-sandbox .
   • Press F1 and select Dev Containers: Rebuild and Reopen in Container.

This template leverages LangChain's astream_events to provide programmatic access to the execution lifecycle of the Agentic Workflow. Because a compiled LangGraph is a Runnable, you can observe and interact with key components such as the LLM, prompt, and tool during their start, stream, and end stages. This allows for granular monitoring of the agent's internal reasoning and tool usage.

# Example concept: using astream_events to monitor the workflow
async for event in graph.astream_events(input_data, version="v2"):
    kind = event["event"]
    if kind == "on_chat_model_stream":
        # Handle LLM streaming
        pass
    elif kind == "on_tool_start":
        # Handle tool execution start
        pass

To run the development environment without using Devcontainers, use the following command:

docker compose -f compose-dev.yaml up -d

By configuring the OpenAI Integration within the Grafana Stack, you can monitor and gain insights into:

• Token usage rates: Track how many tokens are being consumed over time.
• Response times: Monitor the latency of OpenAI API responses.
• Overall costs: Gain visibility into the financial impact of your API usage.

This allows for data-driven decisions regarding the optimal utilization of OpenAI APIs. For detailed implementation steps, refer to the official Grafana OpenAI integration documentation.

The MCP Inspector allows you to explore community and custom MCP servers via a web interface at http://localhost:6274 (available during development).

To connect to a server using Server-Sent Events (SSE):

1. In the Left Sidebar, select SSE as the Transport Type.
2. In the URL field, input the SSE endpoint: http://<mcp server>:<MCP_SERVER_PORT>/sse.
3. Click Connect.

Once connected, you can explore the server's capabilities using the following tabs in the Top Navbar:

• Resources
• Prompts
• Tools

If you are developing and need to route services properly, you may need to reload the Nginx reverse proxy configuration using the following command:

docker compose -f compose-dev.yaml exec nginx sh -c "nginx -s reload"