Seed-Coder Documentation

repository·master·Indexed 20 days ago

https://github.com/bytedance-seed/seed-coder

A family of lightweight 8B parameter open-source code LLMs, including Base, Instruct, and Reasoning models. Seed-Coder supports context lengths up to 32K or 64K depending on the model version. Documentation covers deployment using the Hugging Face transformers library and high-throughput inference via vLLM, including multi-GPU distributed serving with tensor parallelism.

Tokens: 1.2K
Snippets: 3
Records: 4
Agent score: 23%

What's inside Seed-Coder

Deploy Seed-Coder-8B-Instruct with vLLM

master

For high-throughput inference, Seed-Coder-8B-Instruct is fully supported by vLLM. You can perform offline batched inference by initializing the LLM class and defining SamplingParams for decoding hyperparameters like temperature, top_p, and max_tokens.

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
# Initialize the tokenizer
tokenizer = AutoTokenizer.from_pretrained("ByteDance-Seed/Seed-Coder-8B-Instruct")

# Pass the default decoding hyperparameters of Seed-Coder-8B-Instruct
# max_tokens is for the maximum length for generation.
sampling_params = SamplingParams(temperature=0.6, top_p=0.8, repetition_penalty=1.05, max_tokens=512)

# Input the model name or path. Can be GPTQ or AWQ models.
llm = LLM(model="ByteDance-Seed/Seed-Coder-8B-Instruct")

# Prepare your prompts
prompt = "#write a quick sort algorithm."

# generate outputs
outputs = llm.generate([prompt], sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}\n\nGenerated content: {generated_text!r}")

Enable Multi-GPU Distributed Serving with vLLM
master
To increase serving throughput and handle long-context inputs (up to 32K tokens), you can use tensor parallelism in vLLM. This helps mitigate GPU memory limitations by distributing the model across multiple devices using the tensor_parallel_size parameter.
```
llm = LLM(model="ByteDance-Seed/Seed-Coder-8B-Instruct", tensor_parallel_size=8)
```
https://github.com/bytedance-seed/seed-coder/blob/master/README.md

Deploy Seed-Coder-8B-Instruct with transformers

master

You can use the Hugging Face transformers library to deploy Seed-Coder models. This method is suitable for standard single-device inference. Ensure you use trust_remote_code=True when loading the tokenizer and model. It is recommended to use torch.bfloat16 for optimal performance.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "ByteDance-Seed/Seed-Coder-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)

messages = [
    {"role": "user", "content": "Write a quick sort algorithm."},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt",
    add_generation_prompt=True,  
).to(model.device)

outputs = model.generate(input_ids, max_new_tokens=512)
response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

Available Seed-Coder Models
master
Seed-Coder is a family of lightweight 8B code LLMs. The available models include:
- Seed-Coder-8B-Base: Pretrained on model-centric code data (32K context length).
- Seed-Coder-8B-Instruct: Instruction-tuned for user intent alignment (32K context length).
- Seed-Coder-8B-Reasoning: RL trained for enhanced reasoning (64K context length).
- Seed-Coder-8B-Reasoning-bf16: RL trained for enhanced reasoning in bfloat16 (64K context length).
https://github.com/bytedance-seed/seed-coder/blob/master/README.md