DOCSEARCHbyAlgolia
Submit Docs

ChatGLM3 Conversational LLMs

repository·main·Indexed 11 days ago

https://github.com/zai-org/chatglm3

A series of open-source conversational large language models developed by Zhipu AI and Tsinghua University. Features include a web demo with Chat, Tool, and Code Interpreter modes, support for function calling and long-context reasoning, and comprehensive guides for fine-tuning ChatGLM3-6B using SFT, LoRA, and P-TuningV2. Includes instructions for OpenVINO deployment and model conversion to IR format.

Tokens
22.1K
Snippets
69
Records
88
Agent score
96%

What's inside ChatGLM3

  1. Overview of ChatGLM3

    main

    ChatGLM3 is a conversational pre-trained model series released by Zhipu AI and Tsinghua University's KEG Lab. The ChatGLM3-6B is the open-source version, designed to be high-performing while maintaining low deployment barriers.

    Key features include:

    • Stronger Base Model: ChatGLM3-6B-Base is optimized for semantic, mathematical, reasoning, code, and knowledge tasks, performing at a high level for models under 10B parameters.
    • Advanced Functionality: Supports a new Prompt format, native Function Calling (tool use), Code Interpreter, and Agent tasks.
    • Diverse Model Sequence: Includes the standard ChatGLM3-6B, the Base model, and long-context versions (32K and 128K sequence lengths).

  2. Overview of ChatGLM3-6B

    main

    ChatGLM3-6B is an open-source dialogue model developed by Zhipu AI and Tsinghua KEG. It is designed for smooth dialogue with a low deployment threshold. Key features include:

    • Stronger Base Model: The ChatGLM3-6B-Base model is optimized for semantics, mathematics, reasoning, code, and knowledge, outperforming other base models under 10B parameters.
    • Function Support: Natively supports tool invocation (Function Call), code execution (Code Interpreter), and Agent tasks in complex scenarios using a specific Prompt format.
    • Open-source Series: The series includes various versions for different needs:
      • ChatGLM3-6B: Standard dialogue model.
      • ChatGLM3-6B-Base: The foundational base model.
      • ChatGLM3-6B-32K: Optimized for long-text dialogue (32k sequence length).
      • ChatGLM3-6B-128K: Optimized for very long-text understanding (128k sequence length).

    Weights are fully open for academic research and free commercial use is allowed after registration via a questionnaire.

  3. Accelerate ChatGLM3-6B deployment on Intel devices

    main

    This directory provides tools and examples to accelerate the deployment of the ChatGLM3-6B model on Intel hardware using different acceleration frameworks.

    Supported Hardware

    • Intel CPU Series: Includes consumer CPUs as well as Server/Workstation CPUs.
    • Intel Arc Discrete GPUs: Including models such as the Arc A770.
    • Intel CPU Integrated Graphics (iGPU).
    • Other Intel toolkits that support OpenVINO acceleration.

  4. Format multi-turn conversations for dialogue fine-tuning

    main

    To fine-tune the model's dialogue capabilities (without tool use), organize your data into a JSON array of objects where each object contains a conversations list. Each entry in the list must have a role (system, user, or assistant) and content.

    Note: Using this format extensively with many training steps may impact the model's ability to perform tool calls.

    [
  {
    "conversations": [
      {
        "role": "system",
        "content": "<system prompt text>"
      },
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      }
    ]
  }
]

  6. Format datasets for dialogue and tool-calling fine-tuning

    main

    To fine-tune both dialogue and tool-calling capabilities, include a tools field in the data object. The tool role is used to represent the tool call, which is automatically converted during preprocessing.

    Key Rules:

    • The tools field is used to automatically generate the system prompt describing the tools.
    • The tool role includes name, parameters, and observation (the tool's return value).
    • The system role is optional but must appear before any user role and can only appear once per dialogue.
    • You can add a loss field (boolean) to any role to specify if that content should participate in loss calculation. By default, system and user roles do not calculate loss.
    [
  {
    "tools": [
      // available tools, format is not restricted
    ],
    "conversations": [
      {
        "role": "system",
        "content": "<system prompt text>"
      },
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant thought to text>"
      },
      {
        "role": "tool",
        "name": "<name of the tool to be called",
        "parameters": {
          "<parameter_name>": "<parameter_value>"
        },
        "observation": "<observation>"
      },
      {
        "role": "assistant",
        "content": "<assistant response to observation>"
      }
    ]
  }
]

  7. Understand the ChatGLM3 conversation format

    main

    ChatGLM3 uses a specific conversation format designed to prevent injection attacks and unify inputs for tasks like Code Interpreter, Tool use, and Agents.

    Conversations consist of multiple turns, where each turn contains a header and content.

    Header Format

    Headers occupy a full line and follow the pattern: <|role|>{metadata}. The <|role|> part uses special tokens that cannot be encoded as plain text by the tokenizer to ensure security. The metadata part is optional plain text.

    Roles

    • <|system|>: System instructions. While designed to be intercalated, currently it is only permitted at the very beginning of the conversation.
    • <|user|>: User input. Multiple <|user|> messages cannot appear consecutively.
    • <|assistant|>: The AI assistant. An <|assistant|> message must always be preceded by a <|user|> message.
    • <|observation|>: External results (e.g., tool outputs or code execution results). This must always follow an <|assistant|> message.
    <|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
Hello, I'm ChatGLM3. What can I assist you today?

  8. Format data for dialogue and tool-use fine-tuning

    main

    To fine-tune both dialogue and tool-calling capabilities, include a tools field in your data object. The conversations list will then include a tool role to represent tool execution.

    Key details:

    • System Prompt: You do not need to manually insert tool descriptions into the system prompt; the pre-processing step automatically formats the tools field and inserts it as the first system message.
    • Role Transformation: The tool role is not native to ChatGLM3; during pre-processing, it is converted into an assistant role with tool-call metadata and an observation role for the tool's return value.
    • Loss Calculation: You can add a bool field named loss to any role to specify if that content should participate in loss calculation. By default, system and user roles do not calculate loss, while other roles do.
    • Constraints: A system role is optional but must appear before any user role, and can only appear once per conversation.
    [
  {
    "tools": [
      // available tools, format is not restricted
    ],
    "conversations": [
      {
        "role": "system",
        "content": "<system prompt text>"
      },
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant thought to text>"
      },
      {
        "role": "tool",
        "name": "<name of the tool to be called",
        "parameters": {
          "<parameter_name>": "<parameter_value>"
        },
        "observation": "<observation>"
      },
      {
        "role": "assistant",
        "content": "<assistant response to observation>"
      }
    ]
  }
]

  9. Understand ChatGLM3 Demo modes

    main

    The ChatGLM3 Demo supports three distinct operational modes:

    1. Chat: Standard conversational mode. Users can adjust model behavior using parameters in the sidebar such as top_p, temperature, and System Prompt.
    2. Tool: An augmented mode where the model can perform actions via registered tools. It also includes a Manual mode where you can specify a tool list via YAML and manually feed outputs back to the model.
    3. Code Interpreter: A mode where the model executes code within a Jupyter environment. This allows the model to perform complex tasks like plotting charts or symbolic computation by executing multiple code blocks sequentially until the task is complete.

  10. Understand the ChatGLM3 dialogue format

    main

    ChatGLM3 uses a specific dialogue format to prevent injection attacks and unify inputs for tasks like Code Interpreter, Tool Calling, and Agents. The dialogue consists of multiple turns, where each turn contains a header and content.

    Role Tokens

    Each role is represented by a special token that cannot be encoded from plain text to prevent injection. The format is <|role|>{metadata}.

    • <|system|>: System instructions. Currently, these only appear at the very beginning of the dialogue.
    • <|user|>: User input. Multiple <|user|> messages cannot appear consecutively.
    • <|assistant|>: AI assistant response. A <|user|> message must always precede an <|assistant|> message.
    • <|observation|>: External return results (e.g., tool outputs or code execution results). This must always follow an <|assistant|> message.
    <|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
Hello, I'm ChatGLM3. What can I assist you today?

  11. Define tools for the System Prompt

    main

    Tools must be defined as a list of dictionaries following a specific schema. Each tool requires a name, description, and a parameters object (JSON Schema format).

    To use these tools, include them in a dictionary with role: "system" and pass this as the initial entry in your history list.

    tools = [
    {
        "name": "track",
        "description": "追踪指定股票的实时价格",
        "parameters": {
            "type": "object",
            "properties": {
                "symbol": {
                    "description": "需要追踪的股票代码"
                }
            },
            "required": ['symbol']
        }
    }
]
system_info = {"role": "system", "content": "Answer the following questions as best as you can. You have access to the following tools:", "tools": tools}

  12. Format multi-turn dialogue datasets for conversational fine-tuning

    main

    To fine-tune only conversational capabilities (without tool calling), use a JSON array of objects where each object contains a conversations list. Each entry in the list must have a role (system, user, or assistant) and content.

    Note: Extensive fine-tuning with this format may impact the model's tool-calling abilities.

    [
  {
    "conversations": [
      {
        "role": "system",
        "content": "<system prompt text>"
      },
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      }
    ]
  }
]