DOCSEARCHbyAlgolia
Submit Docs

FastAPI LangGraph Agent Production-Ready Template

repository·master·Indexed 25 days ago

https://github.com/wassim249/fastapi-langgraph-agent-production-ready-template

A production-ready backend template for AI agents using FastAPI and LangGraph. It features stateful conversations, long-term memory via mem0 and pgvector, observability with Langfuse and Prometheus, and resilient LLM services with exponential backoff and circular fallback. The template includes a two-node StateGraph for orchestration, JWT authentication, and a full Docker Compose stack including PostgreSQL, Valkey, and Grafana.

Tokens
16.7K
Snippets
51
Records
83
Agent score
81%

What's inside langgraph-fastapi-template

  1. How the LLM Service works

    master

    The LLM service (app/services/llm/) provides a resilient wrapper for all language model calls. Instead of calling model providers directly, you call llm_service.call(messages).

    The service manages:

    • Automatic Retries: Handles transient errors like rate limits or timeouts using exponential backoff.
    • Circular Fallback: If a model fails after its maximum retries, the service automatically switches to the next model in the registry. It follows a circular path through the registry and stops after one full cycle.
    • Total Timeout: A global timeout (LLM_TOTAL_TIMEOUT) ensures the entire operation (including all retries and fallbacks) does not exceed a specific duration.
    • Tool Binding: Tools are bound to the model at startup and are automatically re-bound whenever the service switches to a fallback model.

  3. How the LangGraph Agent orchestrates conversation

    master

    The agent is implemented as a two-node StateGraph that manages the conversation loop. It uses a chat node to interact with the LLM and a tool_call node to handle external function executions.

    The workflow follows this logic:

    1. chat node: Constructs the system prompt, calls the LLM, and returns a Command that routes the graph to either tool_call or END based on whether the LLM requested tool usage.
    2. tool_call node: Executes all requested tool calls concurrently and feeds the results back into the chat node.
    3. END: The process terminates when the LLM provides a final response without tool calls.

    State Persistence: The AsyncPostgresSaver acts as a Checkpointer, persisting the full GraphState per thread_id (session). This enables multi-turn memory and the ability to resume conversations after interrupts.

    graph LR
    START --> chat
    chat -->|tool_calls present| tool_call
    tool_call --> chat
    chat -->|no tool_calls| END

  4. How the long-term memory system works

    master

    The template implements a long-term memory system using mem0 and pgvector. It provides semantic context from past sessions by extracting memories from conversations and storing them as vector embeddings.

    The Lifecycle:

    1. Retrieval (On every request): When a chat request is made, the MemoryService searches for relevant memories using the user_id and the current query. It first checks a cache layer (Valkey/Redis or in-memory) to avoid redundant pgvector queries.
    2. Response: The agent receives the relevant memories as a string to inform its response.
    3. Update (Background): After the LLM responds, the system triggers a background task (asyncio.create_task) to call memory.add(messages, user_id). This ensures memory extraction and embedding storage do not block or slow down the user's chat response.
    sequenceDiagram
    participant G as LangGraph
    participant MS as MemoryService
    participant Cache as Cache (Valkey/TTL)
    participant M as mem0
    participant PG as pgvector

    Note over G: On every chat request
    G->>MS: search(user_id, query)
    MS->>Cache: get(memory:{user_id}:{hash})
    alt cache hit
        Cache-->>MS: cached result
    else cache miss
        MS->>M: memory.search(user_id, query)
        M->>PG: vector similarity search
        PG-->>M: top-k memories
        M-->>MS: formatted results
        MS->>Cache: set(key, result, TTL)
    end
    MS-->>G: relevant memories string

    Note over G: After LLM response (background)
    G-)MS: add(user_id, messages)
    MS->>M: memory.add(messages, user_id)
    M->>PG: store new embeddings

  5. Understand the application database schema

    master

    The application schema consists of three primary entities:

    • User: Represents an account. email is a unique identifier. username is optional and used for system prompt personalization.
    • Session: Represents a conversation. A user can have multiple sessions. The username is denormalized from the User table at creation time to avoid extra lookups during chat requests. The session JWT scopes all chat requests.
    • Thread: Mirrors LangGraph's AsyncPostgresSaver checkpoint thread to track existing threads in the application context.

    Note on Managed Tables:

    • LangGraph manages its own tables (checkpoints, checkpoint_blobs, checkpoint_writes).
    • mem0 manages the longterm_memory collection table via pgvector.
    • These tables are not managed by Alembic and should not be modified through migrations.

  6. Use the conciseness metric prompt for evaluation

    master

    The conciseness prompt is used to evaluate the brevity and relevance of an LLM's generation on a continuous scale from 0 to 1. A score of 1 indicates a generation that is direct, succinct, and avoids unnecessary or irrelevant details, while a lower score indicates verbosity or the inclusion of unasked supplementary information.

    Scoring Criteria

    A generation receives a score of 1 if it:

    • Directly and succinctly answers the question.
    • Focuses specifically on the requested information.
    • Avoids unnecessary, irrelevant, or excessive details.
    • Provides complete information without being verbose.

    Note: Scientific explanations that directly support the answer are considered valid and should not be penalized as unnecessary detail.

  7. Understand the Request Lifecycle

    master

    When a client sends a request (e.g., POST /chat), the system follows a specific sequence to ensure low latency and high reliability:

    1. Middleware & Auth: The request passes through middleware (rate limiting, metrics, request ID generation) and is authenticated via JWT to identify the session.
    2. Concurrent Initialization: To save latency (approx. 200–500ms), the system runs aget_state (to check for graph interrupts) and memory.search (to fetch semantic context) in parallel using asyncio.gather.
    3. LLM Interaction: The chat node sends the system prompt, context, and message history to the LLM.
    4. Tool Execution: If the LLM returns tool_calls, the agent executes them concurrently via asyncio.gather and returns the results to the LLM for a final response.
    5. Background Tasks: While the JSON response is being returned to the client, the system performs background tasks like adding new memories to the Memory Service.

  8. Understand memory caching and isolation

    master

    Caching Behavior

    Memory search results are cached using the key format memory:{user_id}:{sha256(query)[:16]}.

    • TTL: Controlled by CACHE_TTL_SECONDS (default 60s).
    • Rules: Only successful, non-empty results are cached. Errors are never cached.
    • Storage: Use VALKEY_HOST for distributed caching; otherwise, it uses local in-memory storage.

    Per-user Isolation

    Memories are strictly isolated by user_id. The system uses the user_id as a namespace during both storage and retrieval, ensuring users cannot access or retrieve memories belonging to other users.

  9. Trace requests using X-Request-ID

    master

    The application uses asgi-correlation-id to assign a unique X-Request-ID to every request. This ID is critical for observability as it is:

    1. Returned in the HTTP response headers.
    2. Bound to every log line generated during the request.
    3. Used as the filename for profiling reports.

    You can use the X-Request-ID from a response to grep logs, find specific profiling JSON files, or look up the exact trace in Langfuse.

  10. Understand the two-token authentication flow

    master

    The API uses a dual-token system to separate user identity from specific conversation contexts:

    1. User token: Issued upon registration or login. It identifies the user and is used to manage sessions (creating, listing, or deleting them).
    2. Session token: Issued when a new chat session is created via /api/v1/auth/session. This token is scoped to a single session_id and is required for all chat-related endpoints (e.g., /chatbot/chat).

    Both tokens are HS256 signed JWTs. The expiry duration is controlled by the JWT_ACCESS_TOKEN_EXPIRE_DAYS environment variable.

  11. Key Performance and Design Optimizations

    master

    The architecture incorporates several patterns to minimize latency and maximize reliability:

    • Parallel Execution: Both memory searches and tool calls are executed concurrently using asyncio.gather.
    • Prompt Optimization: The system.md file is read once at module load. Per-request overhead is limited to string formatting (user name, datetime, memories) rather than file I/O.
    • Time-Bounded Fallbacks: The LLM fallback loop (retries across different models) is wrapped in asyncio.wait_for(timeout=LLM_TOTAL_TIMEOUT) to prevent the application from hanging indefinitely.
    • Zero-Latency Session Naming: When a new session is created, the API immediately returns the chat response. A background asyncio.Task is used to generate a session title using a fast model, ensuring the user doesn't wait for title generation.
    • Efficient User Context: Usernames are stored in the Session object at creation time, avoiding redundant database lookups on every chat request.

  12. How the evaluation framework works

    master

    The evaluation process follows a three-step pipeline:

    1. Fetch traces: The Evaluator pulls recent LLM traces from Langfuse using the configured LANGFUSE_* environment variables.
    2. Score: For every combination of trace and metric, an LLM judge evaluates the output and returns a pass or fail result based on the metric prompts found in evals/metrics/prompts/*.md.
    3. Report: The system aggregates statistics and per-trace results into a JSON report saved in the evals/reports/ directory.