DOCSEARCHbyAlgolia
Submit Docs

LibreChat Documentation Site

repository·main·Indexed 20 days ago

https://github.com/librechat-ai/librechat.ai

The official documentation, blog, and changelog for LibreChat, an open-source AI chat platform. This Next.js application is built with Fumadocs and MDX, featuring AI-assisted search, multi-language support, and tools for managing documentation content and automated screenshot capture.

Tokens
141.8K
Snippets
422
Records
613
Agent score
68%

What's inside librechat.ai

  1. Compare LibreChat hosting and network services

    main

    LibreChat can be deployed using various hosting and network services depending on your needs for ease of use, cost, and control.

    Recommended Hosting Services

    ServiceDomainProsConsNotes
    DigitalOceanCloud InfrastructureIntuitive interface, stable pricingSmaller network footprintIdeal for enthusiasts and small-to-medium businesses
    HuggingFaceAI/ML SolutionsML/NLP expertiseFocused on ML applicationsExcellent for AI/ML startups
    RailwayApp DeploymentSimplified app deploymentLimited access to containersVery easy to get started

    Network Services

    ServiceDomainProsCons
    CloudflareWeb Performance & SecurityGlobal CDN, DDoS protection, ease of useCustomer support can be slow
    NginxWeb Server, Reverse ProxyHigh performance, stability, resource efficiencyManual setup, limited plugins
    ngrokSecure TunnelingEasy to use, free tier available, secure tunnelingRequires client download, complex domain routing
    TraefikReverse Proxy, Load BalancerAutomatic service discovery, local cluster supportConfiguration can be complex for beginners

  2. What is the RAG API and how does it work?

    main

    The Retrieval-Augmented Generation (RAG) API is a separate FastAPI service that indexes user-uploaded files and retrieves relevant passages to expand prompts. This allows LibreChat to provide context-aware answers based on specific documents. It is supported by a PostgreSQL database with the pgvector extension for vector storage.

    Availability:

    • Works with Agents and Custom Endpoints.
    • Works with OpenAI, Azure OpenAI, Anthropic, and Google.
    • Note on OpenAI Assistants: While Assistants have a built-in 'Retrieval' tool, using the RAG API is recommended to avoid the additional storage and retrieval fees charged by OpenAI.

  3. What are AWS Bedrock Inference Profiles and why use them in LibreChat

    main

    AWS Bedrock Inference Profiles allow you to create custom routing settings for foundation models. Instead of using standard model IDs, you use a unique Amazon Resource Name (ARN) generated by AWS.

    In LibreChat, you can use the inference profile mapping feature to map user-friendly model IDs to these custom ARNs. This provides several benefits:

    • Cross-region load balancing: Automatically distribute requests across multiple AWS regions.
    • Cost allocation: Tag and track costs per application or team.
    • Throughput management: Set dedicated throughput for specific applications.
    • Compliance: Route requests through specific regions to meet data residency requirements.
    • Monitoring: Track usage per inference profile via CloudWatch.

  6. Understand SharePoint integration performance and limits

    main

    The SharePoint integration includes several mechanisms to optimize performance and stability:

    • Token Caching: Tokens are cached to reduce authentication overhead. The cache duration typically matches the token lifetime (usually 50 minutes) and updates automatically before expiration.
    • Concurrent Downloads: The system limits downloads to a maximum of 3 files simultaneously to prevent overloading the browser or server.
    • File Size: Large files may take longer to download. Users should monitor the progress indicator and ensure file sizes do not exceed the limits defined in the LibreChat file upload configuration.

  7. How auto-refill works

    main

    When balance.enabled and autoRefillEnabled are both true, LibreChat automatically adds credits to a user's balance based on a time interval.

    The Refill Process:

    1. When a user attempts to use tokens, the system checks if the current balance is sufficient.
    2. If the transaction would drop the balance to zero or below, the system checks for auto-refill.
    3. The system compares the current date against lastRefill + refillInterval.
    4. If the interval has passed, the refillAmount is added, and lastRefill is updated to the current date.
    5. The transaction proceeds if the balance (after refill) is sufficient.

    Balance Synchronization: When a user logs in, their balance settings are synchronized with the global configuration. If a user has no balance record, one is created using the current startBalance. Existing records are updated to match the global autoRefill settings, intervals, and amounts.

  8. How the auto-refill process works

    main

    When balance.enabled and autoRefillEnabled are both true, the system checks for refills during token consumption attempts:

    1. The system verifies if the current balance is sufficient for the request.
    2. If the balance would drop to or below zero after the transaction, it checks if auto-refill is enabled.
    3. It compares the current date against lastRefill + refillInterval.
    4. If the interval has passed, refillAmount tokens are added, and lastRefill is updated to the current date.
    5. The transaction proceeds if the balance (including any new refill) is sufficient.

  9. How message feedback scores work in Langfuse

    main

    When Langfuse tracing is enabled, LibreChat sends message feedback to Langfuse as a user-feedback BOOLEAN score on the corresponding trace.

    Feedback Logic

    • Thumbs Up: Sent as 1.
    • Thumbs Down: Sent as 0.
    • Tags/Comments: Any selected feedback tag or comment is included in the score.
    • Deleting Feedback: Removing feedback in LibreChat removes the score in Langfuse.

    Metadata and Constraints

    Feedback scores include metadata such as messageId, parentId, conversationId, userId, endpoint, sender, isCreatedByUser, and tokenCount.

    Feedback follows the same configuration as tracing and respects these variables:

    • LANGFUSE_TRACING_ENABLED=false (disables feedback)
    • LANGFUSE_SAMPLE_RATE=0
    • LANGFUSE_TRACING_ENVIRONMENT

    Note: Sending scores is 'best-effort'; the LibreChat UI will not block if Langfuse is temporarily unavailable.

  10. Token Usage and Balance Constraints

    main

    LibreChat implements several logic rules regarding how balances are checked and how tokens are consumed:

    • Summarization Blocking: If the summarization feature is enabled, an API request will be blocked if the cost of the content to be summarized plus the message payload exceeds the current balance.
    • Prompt vs. Completion Tokens: The system is strict about prompt tokens (input) to ensure sufficient funds exist to start a request. It is more lenient with completion tokens (output), allowing for slight deficits caused by generated text.
    • Subagent Usage: For agentic workflows, usage from subagent child-runs is recorded against the parent transaction. The parent agent's run includes all delegated usage in its total.
    • Plugin Accuracy: Prompt token counts for plugins (function calling) are highly accurate but may be slightly conservative (higher by 2-5 tokens).
    • Titling Buffer: A buffer of approximately 150 tokens is reserved for the title generation process (which is a 2-step process) to prevent errors. If funds are insufficient for this buffer, titling is canceled without incurring cost or throwing an error.

  11. How auto-refill works for user balances

    main

    When autoRefillEnabled is true, the system automatically adds refillAmount tokens to a user's balance once the specified refillInterval has passed since their lastRefill date.

    Refill Logic:

    1. When a user attempts to spend tokens, the system checks if the balance is sufficient.
    2. If the transaction would result in a zero or negative balance, the system checks for auto-refill.
    3. If enabled, the system compares the current date to lastRefill + refillInterval.
    4. If the interval has passed, the refillAmount is added, and lastRefill is updated to the current date.
    5. The transaction proceeds if the balance (including the refill) is sufficient.

  12. Use Tunneling and DNS services for LibreChat

    main

    To expose a local development server to the internet or manage domain traffic, you can use the following services:

    Tunneling Services

    Useful for sharing work, testing, or providing an On-Premise installation via a public URL without complex networking.

    • Ngrok: Provides secure local tunneling to the internet.
    • Cloudflare: Enhances web performance and security.

    DNS Services

    Used to translate human-readable domain names into machine-readable IP addresses.

    • Cloudflare DNS: Manages internet traffic, provides DDoS protection, and offers advanced traffic management. See the Cloudflare Guide for details.