gpt-oss

repository·main·Indexed 12 days ago

https://github.com/openai/gpt-oss

A series of open-weight models from OpenAI designed for high-reasoning and agentic tasks, featuring gpt-oss-20b and gpt-oss-120b. The project provides reference inference implementations in PyTorch, Triton, and Metal, as well as tools for building agentic workflows, MCP servers, and support for deployment via vLLM, Hugging Face Transformers, Ollama, and LM Studio. All models require the Harmony response format to function correctly.

Tokens
10.3K
Snippets
25
Records
41
Agent score
97%

What's inside gpt-oss

  1. Overview of gpt-oss models

    main

    The gpt-oss series consists of open-weight models from OpenAI designed for reasoning and agentic tasks. There are two primary versions:

    • gpt-oss-120b: A 117B parameter model (5.1B active) designed for production and high-reasoning use cases. It is optimized to fit into a single 80GB GPU (e.g., NVIDIA H100 or AMD MI300X) using MXFP4 quantization.
    • gpt-oss-20b: A 21B parameter model (3.6B active) designed for lower latency, local deployment, or specialized use cases. It can run within 16GB of memory using MXFP4 quantization.

    Critical Requirement: Both models must be used with the Harmony response format to function correctly.

  2. Explore gpt-oss tools and response formats

    main

    Response Formats

    • OpenAI Harmony: The standard response format used by gpt-oss models.

    Tooling Examples

    • Python Tool: An example implementation of a Python tool for gpt-oss (located in ./gpt_oss/tools/python_docker/).
    • Browser Tool: An example implementation of a simple browser tool for gpt-oss (located in ./gpt_oss/tools/simple_browser/).
  3. Generate system prompts using build-system-prompt.py

    main
    The build-system-prompt.py script is used to automatically discover tools and construct a system prompt identical to the one generated by reference-system-prompt.py. This is useful for understanding how to programmatically build system prompts for Harmony based on discovered MCP tools.
  4. Run the reference PyTorch implementation

    main

    The PyTorch implementation in gpt_oss/torch/model.py is a non-optimized reference for educational purposes. It requires at least 4× H100 GPUs because it upcasts weights to BF16 and lacks advanced optimizations. It includes tensor parallelism for MoE to allow the larger model to run on multiple GPUs (e.g., 4xH100 or 2xH200).

    # Install dependencies
    pip install -e ".[torch]"
    
    # Run on 4xH100
    torchrun --nproc-per-node=4 -m gpt_oss.generate gpt-oss-120b/original/
  5. Install the Reference Triton implementation

    main

    The Triton implementation uses an optimized MoE kernel supporting MXFP4 and can run gpt-oss-120b on a single 80GB GPU. This requires installing triton and torch from source.

    Note: If you encounter torch.OutOfMemoryError while loading weights, set the environment variable PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to use the expandable allocator.

    # Install triton from source
    git clone https://github.com/triton-lang/triton
    cd triton/
    pip install -r python/requirements.txt
    pip install -e . --verbose --no-build-isolation
    pip install -e python/triton_kernels
    
    # Install the gpt-oss triton implementation
    pip install -e ".[triton]"
    
    # Run on 1xH100
    export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    python -m gpt_oss.generate --backend triton gpt-oss-120b/original/
  6. Run gpt-oss models locally

    main

    You can run gpt-oss models on your own hardware using several different local inference engines and platforms:

    • Ollama: Provides easy local deployment for gpt-oss models.
    • LM Studio: Supports both gpt-oss-20b and gpt-oss-120b models.
    • Hugging Face Transformers: Use the standard Transformers library to run models directly.
    • llama.cpp: Optimized for CPU/GPU inference, including support for Unsloth GGUFs.
    • TVM: Allows for compiling and running gpt-oss for optimized execution.

    Hardware Support

    • NVIDIA: Optimized for RTX GPUs.
    • AMD: Supported on Ryzen AI Processors, Radeon Graphics Cards, and via Lemonade for STX Halo and Radeon dGPUs.
  7. Use the Browser tool with gpt-oss

    main

    The browser tool allows gpt-oss models to search for phrases, open specific pages, and find content on a page.

    Warning: The provided SimpleBrowserTool is for educational purposes only. For production, implement your own backend by extending YouComBackend or ExaBackend.

    To enable the tool, you must include its definition in the system message of your Harmony-formatted prompt using either .with_browser_tool() or .with_tools(browser_tool.tool_config).

    Implementation Details:

    • Backends: Supports YouComBackend (default, requires YDC_API_KEY) and ExaBackend (set BROWSER_BACKEND=exa and provide EXA_API_KEY).
    • Context Management: The tool uses a scrollable text window to manage context.
    • Caching: The tool caches requests to allow revisiting page parts without reloading. Important: Create a new browser instance for every request to ensure proper behavior.
    import datetime
    import os
    from gpt_oss.tools.simple_browser import SimpleBrowserTool
    from gpt_oss.tools.simple_browser.backend import ExaBackend, YouComBackend
    from openai_harmony import SystemContent, Message, Conversation, Role, load_harmony_encoding, HarmonyEncodingName
    
    encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
    
    # Configure backend via environment variables
    tool_backend = os.getenv("BROWSER_BACKEND", "youcom")
    if tool_backend == "youcom":
        backend = YouComBackend(source="web")
    elif tool_backend == "exa":
        backend = ExaBackend(source="web")
    else:
        raise ValueError(f"Invalid tool backend: {tool_backend}")
    
    browser_tool = SimpleBrowserTool(backend=backend)
    
    # Setup system message with tool enabled
    system_message_content = SystemContent.new().with_conversation_start_date(
        datetime.datetime.now().strftime("%Y-%m-%d")
    )
    
    # Enable the tool in the prompt
    system_message_content = system_message_content.with_browser_tool()
    
    system_message = Message.from_role_and_content(Role.SYSTEM, system_message_content)
    
    # Construct conversation
    messages = [system_message, Message.from_role_and_content(Role.USER, "What's the weather in SF?")]
    conversation = Conversation.from_messages(messages)
    
    # Render for inference
    token_ids = encoding.render_conversation_for_completion(conversation, Role.ASSISTANT)
    
    # ... (perform inference) ...
    
    # Handle tool call in the response
    # Assuming 'output_tokens' is the result from inference
    parsed_messages = encoding.parse_messages_from_completion_tokens(output_tokens, Role.ASSISTANT)
    last_message = parsed_messages[-1]
    
    if last_message.recipient.startswith("browser"):
        response_messages = await browser_tool.process(last_message)
        parsed_messages.extend(response_messages)
  8. Configure Codex to use gpt-oss

    main

    You can use codex as a client for gpt-oss. To use the 20b version with a local provider like ollama, configure ~/.codex/config.toml to point to the local server and enable reasoning content visibility.

    # ~/.codex/config.toml
    disable_response_storage = true
    show_reasoning_content = true
    
    [model_providers.local]
    name = "local"
    base_url = "http://localhost:11434/v1"
    
    [profiles.oss]
    model = "gpt-oss:20b"
    model_provider = "local"

    To run:

    1. Start the provider (e.g., ollama)

    ollama run gpt-oss:20b

    2. Run codex with the oss profile

    codex -p oss

  9. Deploy gpt-oss using vLLM

    main

    vLLM can be used to spin up an OpenAI-compatible web server. It is recommended to use uv for dependency management. The following command installs the necessary specialized vLLM wheels and starts a server for gpt-oss-20b.

    uv pip install --pre vllm==0.10.1+gptoss \
        --extra-index-url https://wheels.vllm.ai/gpt-oss/ \
        --extra-index-url https://download.pytorch.org/whl/nightly/cu128 \
        --index-strategy unsafe-best-match
    
    vllm serve openai/gpt-oss-20b
  10. Configure providers in compatibility-test

    main

    Before running the compatibility tests, you must define the target API in providers.ts.

    • To test Chat Completions, add your provider configuration under the chat key.
    • To test the Responses API, add your provider configuration under the responses key.
    • Replace the default vllm placeholder with the specific provider name you are testing.
  11. Run gpt-oss locally with Ollama or LM Studio

    main

    For consumer hardware, use Ollama or LM Studio.

    Ollama commands:

    # gpt-oss-20b
    ollama pull gpt-oss:20b
    ollama run gpt-oss:20b
    
    # gpt-oss-120b
    ollama pull gpt-oss:120b
    ollama run gpt-oss:120b

    LM Studio commands:

    # gpt-oss-20b
    lms get openai/gpt-oss-20b
    # gpt-oss-120b
    lms get openai/gpt-oss-120b
  12. Run gpt-oss via server-side inference engines

    main

    For high-throughput or production-grade serving, use the following server-side frameworks:

    • vLLM: Highly efficient serving engine with specific recipes for gpt-oss.
    • NVIDIA TensorRT-LLM: Optimized for maximum performance on NVIDIA hardware.
    • AMD ROCm: Support for running models on AMD AI hardware ecosystems.