DOCSEARCHbyAlgolia
Submit Docs

GPT4All

repository·main·Indexed 13 days ago

https://github.com/nomic-ai/gpt4all

An ecosystem for running large language models (LLMs) privately on consumer-grade hardware without GPUs or cloud APIs. It features a C/C++ backend for CPU-based inference supporting GPTJ, LLAMA, and MPT architectures, with official Python bindings (v4.0.0) and community Node.js/TypeScript bindings for text generation and embeddings.

Tokens
25.8K
Snippets
75
Records
150
Agent score
50%

What's inside GPT4All

  1. Overview of GPT4All Language Bindings

    main
    GPT4All provides language bindings for its backend, allowing developers to interact with GPT4All models and other llama.cpp compatible models. These bindings enable core functionalities such as loading models and generating text. The Python bindings specifically include the ability to embed text as vector representations.

  2. Overview of the GPT4All Backend

    main
    The GPT4All backend is a C/C++ library designed for CPU-based model inference. It serves as a universal wrapper for all model architectures supported by the GPT4All ecosystem. This backend is the foundation for the native GPT4All Chat application and provides the core logic used by various language bindings (such as Python and Node.js) to perform inference.

  3. GPT4All Integrations

    main

    GPT4All integrates with several popular developer tools:

    • Langchain: For building LLM-powered applications.
    • Weaviate Vector Database: Via the text2vec-gpt4all module for retriever-vectorizer capabilities.
    • OpenLIT: For OTel-native monitoring of LLM usage.

  4. Use LocalDocs with the API Server

    main

    You can augment API calls with relevant text snippets from a LocalDocs collection.

    Note: LocalDocs activation must be done via the GPT4All UI, not via the API itself:

    1. Open the Chats view in the GPT4All application.
    2. Scroll to the bottom of the chat history sidebar and select the server chat (distinguished by a different background color).
    3. Activate your desired LocalDocs collections in the right sidebar.

    Once activated, your API calls will include retrieved references in the response object at response["choices"][0]["references"].

  5. How LocalDocs and OneDrive integration works

    main

    The integration relies on a local synchronization and embedding workflow:

    1. File Syncing: OneDrive for Desktop ensures your cloud files are physically present on your local storage.
    2. Indexing: GPT4All's LocalDocs feature maintains a local database of these synced files.
    3. Automatic Updates: As OneDrive updates files on your disk, LocalDocs automatically detects changes to keep the index current.
    4. Semantic Search: LocalDocs uses Nomic Embedding models to perform semantic searches, finding relevant snippets from your local files to provide context to the LLM during chat sessions.

  6. Avoid setting pad token equal to eos token

    main
    During initial experiments, setting the tokenizer pad token equal to the eos token caused the model to fail to learn when to stop, leading to infinite generation or duplication. To fix this, use a separate token for eos and pad. If a pad token is not present in the vocabulary, add one to the tokenizer and expand the model's embedding size.

  7. Understand model parameters and quantization

    main

    When selecting a model for GPT4All, consider the following trade-offs:

    • Parameter Count: Larger models (e.g., 13B) generally provide more coherent instruction following but require more resources. GPT4All is optimized for the 3-13B parameter range.
    • Quantization: Smaller quantization levels (e.g., q4_0 vs 16bit) result in faster performance and lower RAM usage, but may lead to slightly lower model intelligence/performance.
    • Hardware Requirements: Ensure your device has sufficient RAM. For example, an 8B parameter model with q4_0 quantization typically requires ~8 GB of RAM.

  8. Supported Model Architectures in GPT4All

    main

    The GPT4All ecosystem currently supports three primary model architectures:

    1. GPTJ: Based on the GPT-J architecture (e.g., EleutherAI/gpt-j-6b).
    2. LLAMA: Based on the LLAMA architecture.
    3. MPT: Based on Mosaic ML's MPT architecture.

    Note that LLAMA-based models are typically subject to non-commercial licenses, while GPTJ and MPT models generally allow for commercial usage.

  9. How chat sessions and completions work

    main

    GPT4All provides two ways to interact with models: Chat Sessions and Stateless Usage.

    Chat Sessions

    Use model.createChatSession() to maintain context between completions. This is ideal for back-and-forth conversations. A model instance can only have one active chat session at a time. You can set default options (like temperature) and a systemPrompt for the entire session.

    Stateless Usage

    Use createCompletion(model, ...) directly on the model instance for one-off completions. Context is not maintained between calls. If providing an array of messages for a stateless call, the last message must have the role user, otherwise an error is thrown.

    import { createCompletion, loadModel } from "../src/gpt4all.js";

const model = await loadModel("orca-mini-3b-gguf2-q4_0.gguf", {
    verbose: true,
    device: "gpu",
    nCtx: 2048,
});

// Chat Session (Stateful)
const chat = await model.createChatSession({
    temperature: 0.8,
    systemPrompt: "### System:\nYou are an advanced mathematician.\n\n",
});
const res1 = await createCompletion(chat, "What is 1 + 1?");

// Stateless (One-off)
const res2 = await createCompletion(model, "What is 1 + 1?");

model.dispose();

  10. How LocalDocs works

    main

    LocalDocs uses Nomic AI's on-device embedding models to index a folder into text snippets. Each snippet is assigned an embedding vector.

    When you enter a prompt, the system performs a semantic search to find snippets from your files that are mathematically similar to your question. These semantically relevant snippets are then included in the prompt sent to the LLM to provide context.

    To interact with these embedding models directly outside of the desktop application, use the Nomic Python SDK.

  11. How GPT4All v1 templates work

    main

    GPT4All supports a nonstandard template syntax starting with {# gpt4all v1 #}. Unlike standard templates, which automatically combine user messages, LocalDocs sources, and file attachments into a single content field, GPT4All v1 templates require you to manually handle these components within the template logic for them to function correctly.

    Use v1 templates if you need complete, granular control over how sources (from LocalDocs) and prompt_attachments are formatted and inserted into the conversation stream.

    {# gpt4all v1 #}
{%- for message in messages %}
    {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' }}
    {%- if message['role'] == 'user' %}
        {%- for source in message['sources'] %}
            {%- if loop.first %}
                {{- '### Context:\n' }}
            {%- endif %}
            {{- 'Collection: ' + source['collection'] + '\n'   +
                'Path: '       + source['path']       + '\n'   +
                'Excerpt: '    + source['text']       + '\n\n' }}
        {%- endfor %}
    {%- endif %}
    {%- for attachment in message['prompt_attachments'] %}
        {{- attachment['processed_content'] + '\n\n' }}
    {%- endfor %}
    {{- message['content'] | trim }}
    {{- '<|eot_id|>' }}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}

  12. What are chat templates and system messages?

    main

    LLMs natively process plain text and do not inherently distinguish between user input and model output. Chat templates convert a conversation history into the specific plain-text format a model expects. Using the correct template is critical for model performance.

    A system message is a specific type of message used to control the LLM's behavior across the entire conversation (e.g., "Speak like a pirate"). While many models support system messages, not all do.

    When to customize:

    • You are sideloading a model that lacks a built-in template.
    • You need more granular control over the input than a system message allows.