SGLang provides a /v1/score endpoint that allows an LLM to act as a classification engine. Instead of generating a sequence of tokens, it evaluates the probability distribution of the very next token. This is significantly faster than standard generation because it requires only a single forward pass regardless of the desired output length.
Key Implementation Details:
- Query Construction: The
query should contain the prompt up to the point immediately before the answer is expected. - Label Tokens: You must provide the token IDs of the possible labels (e.g., 'A', 'B', 'C') in the
label_token_ids field. The model will return the scores for these specific tokens. - Single-Token Constraint: Each label must be represented by a single token in the model's vocabulary. If a category name (e.g., "Noise Complaint") consists of multiple tokens, you must map it to a single-token symbol (like 'A') in the prompt and handle the mapping in your application logic.
- Tokenizer Verification: Token IDs vary by model. You must use the specific model's tokenizer to verify the correct IDs for your labels.
import requests
requests.post(f"{BASE_URL}/v1/score", json={
"model": MODEL,
"query": prompt, # Prompt ending just before the answer
"items": [""],
"label_token_ids": [32, 33, 34], # Example IDs for 'A', 'B', 'C'
"apply_softmax": True,
})
