lm-format-enforcer

lm-format-enforcer provides several advantages over other constrained generation libraries:

• Broad Compatibility: Works with any Python language model and tokenizer. Supports transformers, LangChain, LlamaIndex, llama.cpp, vLLM, Haystack, NVIDIA TensorRT-LLM, and ExLlamaV2.
• Advanced Generation Support: Supports batched generation and beam searches, where each input/beam can have different tokens filtered.
• Flexible Formats: Supports JSON Schema (including nested fields, arrays, dictionaries, and optional fields), JSON Mode (schemaless), and Regular Expressions.
• Natural LLM Behavior: Unlike libraries that force a rigid structure, LMFE allows the model to control whitespace and field ordering. This reduces hallucinations by letting the model generate tokens in its most natural style.
• Non-Intrusive: Does not modify the high-level loop of the transformers API.

You can use LM Format Enforcer with the vLLM inference server in two ways:

1. Global Server Configuration

Set the --guided-decoding-backend flag when starting the vLLM OpenAI API server to use lm-format-enforcer for all requests.

2. Per-Request Configuration

Pass guided_decoding_backend as an extra_body parameter in your OpenAI client calls. You can also specify guided_regex or guided_json for specific constraints.

# Global configuration via CLI
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.2 \
  --guided-decoding-backend lm-format-enforcer

# Per-request configuration via Python client
completion = client.chat.completions.create(
  model="mistralai/Mistral-7B-Instruct-v0.2",
  messages=[
    {"role": "user", "content": "Classify this sentiment: LMFE is wonderful!"}
  ],
  extra_body={
    "guided_regex": "[Pp]ositive|[Nn]egative",
    "guided_decoding_backend": "lm-format-enforcer"
  }
)

To enforce a specific JSON structure using Hugging Face transformers, follow these steps:

1. Define your schema using a pydantic.BaseModel.
2. Initialize a transformers pipeline.
3. Create a JsonSchemaParser using the Pydantic schema.
4. Use build_transformers_prefix_allowed_tokens_fn to create a prefix function that links the tokenizer and the parser.
5. Pass this function to the pipeline via the prefix_allowed_tokens_fn argument.

Note: If running in Google Colab with a T4 GPU, you may need additional dependencies:

from pydantic import BaseModel
from lmformatenforcer import JsonSchemaParser
from lmformatenforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn
from transformers import pipeline

# 1. Define schema
class AnswerFormat(BaseModel):
    first_name: str
    last_name: str
    year_of_birth: int
    num_seasons_in_nba: int

# 2. Create pipeline
hf_pipeline = pipeline('text-generation', model='TheBloke/Llama-2-7b-Chat-GPTQ', device_map='auto')
prompt = f'Here is information about Michael Jordan in the following json schema: {AnswerFormat.schema_json()} :\n'

# 3. & 4. Create parser and prefix function
parser = JsonSchemaParser(AnswerFormat.schema())
prefix_function = build_transformers_prefix_allowed_tokens_fn(hf_pipeline.tokenizer, parser)

# 5. Call pipeline with prefix function
output_dict = hf_pipeline(prompt, prefix_allowed_tokens_fn=prefix_function)

# Extract results
result = output_dict[0]['generated_text'][len(prompt):]
print(result)

While lm-format-enforcer guarantees output will match the specified format, it does not guarantee semantic correctness and may increase hallucinations if the enforcement is too aggressive.

To diagnose how much the enforcer is overriding the model's natural preferences, pass output_scores=True and return_dict_in_generate=True in the kwargs to generate_enforced(). This returns a token-by-token dataframe containing:

• generated_token: The token selected by the enforcer.
• generated_score: The score of the selected token.
• leading_token: The token the model would have chosen without enforcement.
• leading_score: The score of the token the model originally preferred.

If generated_score is significantly lower than leading_score, the enforcer is forcing the model into a low-probability state, which may require prompt engineering to better align the model's natural output with the required format.

When using generate_enforced(), the output contains enforced_scores. This is a dictionary (convertible to a pandas.DataFrame) that provides diagnostic information about how the enforcer intervened.

Each row represents a timestep where the enforcer changed the token selection. Key columns include:

• generated_token: The token the enforcer forced the model to pick.
• generated_token_idx: The index of the forced token.
• generated_score: The post-softmax score of the forced token.
• leading_token: The token the model wanted to generate.
• leading_token_idx: The index of the model's preferred token.
• leading_score: The post-softmax score of the model's preferred token.

High differences between leading_score and generated_score indicate aggressive enforcement, which can increase the likelihood of hallucinations. This information is useful for improving prompt engineering to reduce the need for heavy enforcement.