DOCSEARCHbyAlgolia
Submit Docs

h2oGPT

repository·main·Indexed 11 days ago

https://github.com/h2oai/h2ogpt

An open-source, private LLM framework for chatting with documents and using local or cloud-based models. It features OpenAI compatibility, multimodal support for vision, voice, and image generation, and a private document database supporting various formats. The ecosystem includes a multi-tenant chatbot, H2O LLM Studio for no-code fine-tuning, and support for models like LLaMa2, Mistral, and Falcon across GPU and CPU backends.

Tokens
95.6K
Snippets
243
Records
317
Agent score
93%

What's inside h2oGPT

  1. Overview of h2oGPT features

    main

    h2oGPT is an Apache V2 open-source project designed for querying and summarizing documents or chatting with local private GPT LLMs.

    Key capabilities include:

    • Private Document Database: Supports PDFs, Excel, Word, Images, Video Frames, YouTube, Audio, Code, Text, and Markdown. Uses persistent databases like Chroma, Weaviate, or in-memory FAISS.
    • Advanced Retrieval: Features HYDE (Hypothetical Document Embeddings) and Semantic Chunking (GPU required).
    • Model Support: Supports LLaMa2, Mistral, Falcon, Vicuna, WizardLM, and more. Compatible with GPU (HF, LLaMa.cpp GGML) and CPU (HF, LLaMa.cpp, GPT4ALL).
    • Multimodal Capabilities:
      • Vision: LLaVa, Claude-3, Gemini-Pro-Vision, GPT-4-Vision.
      • Image Generation: Stable Diffusion (sdxl-turbo, sdxl, SD3), PlaygroundAI, and Flux.
      • Voice: STT via Whisper and TTS via Microsoft Speech T5 or MPL2-licensed TTS (including Voice Cloning).
    • Interfaces: Gradio UI, CLI, and Open Web UI (via OpenAI Proxy).
    • OpenAI Compatibility: Acts as a drop-in replacement for OpenAI servers, supporting Chat/Text completions, STT, TTS, Image generation, Embeddings, and Function tool calling.
    • Agents: Specialized agents for Search, Document Q/A, Python Code, and CSV frames.

  2. Overview of h2oGPT Client APIs

    main

    h2oGPT provides multiple ways to interact with its services:

    1. Gradio API: For direct interaction with the Gradio interface.
    2. OpenAI-compliant API: Allows you to use standard OpenAI client libraries to interact with h2oGPT services.
    3. cURL: Basic API functionality can be accessed via standard curl commands.

  3. Supported LangChain Agents in h2oGPT

    main

    h2oGPT integrates with several LangChain agents, each with varying levels of support and model compatibility. When choosing an agent, note the specific model requirements (e.g., OpenAI vs. non-OpenAI) and the current development stage (e.g., Alpha, Pre-alpha).

    Agent Compatibility and Status

    AgentStatusModel Requirements / Notes
    SearchFunctionalWorks sometimes with non-OpenAI models after improvements beyond LangChain
    CollectionPre-alpha tested
    PythonPre-alpha testedCurrently only allowed with OpenAI
    CSVFunctionalWorks well with OpenAI due to use of Function Tools
    PandasDisabledDisabled until load csv/json with pandas is implemented
    JSONAlpha testedCurrently only allowed with OpenAI
    AutoGPTAlpha testedIncludes multiple tools and memory capabilities

    AutoGPT Tools and Capabilities

    When using the AutoGPT agent, the following tools are available:

    • Search
    • Wikipedia
    • Shell
    • File
    • Python
    • Requests
    • Wolfram Alpha
    • Memory (built-in)

  4. Compare h2oGPT with other LLM and RAG tools

    main

    h2oGPT is a private, open-source alternative to many popular AI services. Key differentiators include:

    • Vs PrivateGPT: h2oGPT provides a full UI with chat export/import, parallel document ingestion (with GPU support and progress bars), collection selection, and the ability to chat with specific documents or entire collections. It also supports fine-tuning (QLORA) and side-by-side model comparisons.
    • Vs Cloud Services (ChatGPT, Bard, Perplexity, etc.): h2oGPT is fully open-source and private, ensuring data does not leave your controlled environment.
    • Vs LocalAI: While LocalAI focuses on audio transcription and image generation, h2oGPT provides specialized document question-answering capabilities.
    • Vs Quiver: Unlike Quiver, which requires Docker and is limited to OpenAI models, h2oGPT supports a wide range of Hugging Face models and local GPU acceleration.

  5. Understand h2oGPTe RAG Benchmark results format

    main

    The h2oGPTe RAG (Retrieval-Augmented Generation) benchmarks evaluate the system's ability to answer specific questions based on provided documents (e.g., financial reports). The benchmark output includes several key components for each test case:

    • Question: The specific query posed to the system.
    • Missing: A list of expected values or substrings that the model failed to include in its answer.
    • Reply: The raw text response generated by the AI assistant.
    • Cited Documents: The indices of the documents used to construct the answer.
    • Answer: The primary response provided by the model.
    • Grounded Answer: A version of the answer where specific spans of text are tagged with citation indices (e.g., <co: 2,4,6>) to show exactly which document parts support the claim.

    This format allows developers to identify whether a failure is due to retrieval (wrong documents cited) or generation (correct documents cited but wrong information provided).

  6. Configure Context-Filling (top_k_docs)

    main

    The --top_k_docs parameter controls how many document chunks or parts are placed into the LLM context for queries, summarization, or extraction.

    • Default: --top_k_docs=3.
    • Auto-fill: Setting --top_k_docs=-1 allows h2oGPT to automatically attempt to fill the available context. This is often the best choice for quality.
    • Manual Control: For a balance of speed and quality, use a fixed value like --top_k_docs=10.

    Token Management: When using --top_k_docs=-1, you can prevent exceeding the LLM's context window by setting:

    • max_input_tokens: Limits tokens per individual LLM call.
    • max_total_input_tokens: Limits the total tokens across all LLM calls.

  7. Understand the difference between Fine-tuning and Pre-training

    main

    When working with LLMs, it is important to distinguish between these two processes:

    FeaturePre-trainingFine-tuning
    Data ScaleTerabytes (TBs)Megabytes (MBs) to Gigabytes (GBs)
    GoalMaster languages and general knowledgeAdapt to specific styles or prompting formats
    DurationWeeks/Months on hundreds of GPUsHours/Days on a few GPUs
    Main RiskUnderfitting and high costOverfitting

  8. Manage models and inference servers in the Models Tab

    main

    The Models Tab is used to select and load LLMs, LoRA adapters, or connect to remote inference servers.

    Using Inference Servers

    If you are using an external inference server (e.g., OpenAI, vLLM, oLLaMa, Google, MistralAI, or Gradio), enter the server address in the Choose/Enter Server field and click Load Model Names from Server to populate the model list.

    Loading and Unloading Models

    • Load (Download) Model: Click this to load the selected model state.
    • Unload Model: Click this to free memory. Alternatively, select [] in the model dropdown.
      • Note: For local models, unloading only works reliably if you did not pre-load the model using the --base_model flag.

    Key Features

    • Choose/Enter Model: Select from the dropdown or paste a filename/HuggingFace link.
    • Choose/Enter LORA: Select a LoRA adapter (must be trained on the chosen base model using PEFT LoRA).
    • Compare Mode: Enables simultaneous comparison of two LLMs in the same window. Note that this uses memory for both models and streams them individually.

  9. Understand UserData vs MyData in h2oGPT

    main

    h2oGPT provides two distinct data scopes for document Q&A and storage:

    • UserData: A shared space accessible to anyone on the server. It persists across sessions in a single location for the entire server, making it ideal for collaborative work. Access to this can be controlled via the allow_upload_to_user_data option.
    • MyData: A personal, private space. It is inaccessible if a user starts a new browser session (unless persistent authentication like cookies is implemented in future updates). This is useful for public demonstrations where each instance needs to be independent, or for users who do not have permission to upload to the shared UserData.

    Note on Source Links: If you use source links to direct users to documents for local download, the original documents must remain on the host system where the database was created (e.g., the user_path used for UserData). If you move only the database to a different host, the links will fail because the host won't have access to the files. URL links (like Wikipedia) will work regardless of the host.

  10. Understand h2oGPTe RAG Benchmark output format

    main

    The h2oGPTe RAG (Retrieval-Augmented Generation) benchmarks evaluate the model's ability to retrieve and ground answers in provided documents. The benchmark output includes several key components for each question:

    • Question: The user query being tested.
    • Oracle: The source document used as the ground truth (e.g., a specific PDF URL).
    • Missing: A list of specific terms or values that the model failed to include in its answer compared to the oracle.
    • Reply: The full response generated by the model, which includes:
      • Relevant Documents: A list of document indices identified as relevant.
      • Cited Documents: A list of document indices actually used in the response.
      • Answer: The natural language response.
    • Grounded Answer: An annotated version of the answer where specific spans of text are tagged with <co: ...> tokens. These tokens indicate the indices of the documents that support that specific piece of information (e.g., <co: 6,7,9> means the text is supported by documents 6, 7, and 9).
    • Error Indicators: The prefix E typically denotes an error or a mismatch in the evaluation (e.g., missing information).

  11. Understand context length and cutoff length

    main

    The total length of input and output text handled by an LLM is limited by its context length. For current models, this is typically 2048 tokens. If the input uses most of this length, the available output space is reduced (e.g., 1900 input tokens leaves 148 tokens for output).

    When fine-tuning, you can use a cutoff_len to truncate inputs. For instruction-type datasets, a cutoff_len of 512 is often sufficient and improves memory and time efficiency. Model names containing a number like 512 (e.g., h2ogpt-oasst1-512-20b) indicate the cutoff length used during fine-tuning.

  12. Configure advanced prompting and document QA in the Expert Tab

    main

    The Expert Tab provides granular control over prompting, document QA, LLM parameters, and speech/voice settings.

    Prompt Control

    • Prompt Type: Select the prompt format. If model lock is not enabled, h2oGPT automatically chooses types based on the model (see prompter.py for details).
    • System Prompt Type and System Prompt: Define the system prompt text to steer model behavior.
    • DocQA Prompting: Use Query pre-prompt, query prompt, summary pre-prompt, and summary prompt to guide Document QA.
    • HYDE LLM Prompt: Used for Hypothetical Document Embeddings (HYDE) to improve embedding quality for vague queries.
    • System Pre-Context: Add arbitrary text to the LLM context.
    • Pre-Conversation: Provide a list of [(human, bot), ...] interactions to seed chat history.
    • Text DocQA: Add specific text as documents.

    Document Control

    • Force xxx: Control tools used for ingesting images, video, audio, PDF, URL, or JSON.
    • Context/Chunking: Control context filling, chunking, sorting, splitting, and merging.
    • HYDE Level: Set the number of HYDE iterations (0 is standard DocQA with a single embedding lookup).

    LLM Control

    • Stream output: Enables streaming (not currently supported for GPT4All/llama.cpp via UI).
    • Sample: Enable sampling to use temperature, top_p, top_k, or beams.
      • seed: Set a fixed seed (>0) for reproducible sampling. Supported by OpenAI, vLLM, TGI, and MistralAI. Not supported by Anthropic, Google, Groq, Replicate, or Sagemaker.
    • Max/Min output length: Set token limits for responses.
    • Repetition Penalty: Adjust penalty for repetitive text.

    Speech and Voice Cloning (Coqui TTS)

    • Language: Manually select the language for better results if autodetect fails.
    • Voice Cloning: Use a clean audio file (File for Clone) or record via microphone (Mic for Clone) to create a new Speaker Style.