LLM-Blender Documentation

Enhance response quality by sampling $N$ responses from an LLM and selecting the one ranked highest by the reward model using blender.best_of_n_generate.

Parameters:

• model: The LLM model instance (e.g., a HuggingFace AutoModelForCausalLM).
• tokenizer: The corresponding tokenizer.
• prompts: A list of formatted prompt strings.
• n: The number of samples to generate per prompt.

Workflow:

1. Load a ranker (e.g., blender.loadranker("llm-blender/PairRM")).
2. Pass the model, tokenizer, and prompts to best_of_n_generate.

import llm_blender
from transformers import AutoTokenizer, AutoModelForCausalLM

# Setup model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceH4/zephyr-7b-beta", device_map="auto")

# Prepare prompts (example using chat template)
inputs = ["can you tell me a joke about OpenAI?"]
# ... (apply chat template to create 'prompts') ...

# Best-of-N sampling
blender = llm_blender.Blender()
blender.loadranker("llm-blender/PairRM")
outputs = blender.best_of_n_generate(model, tokenizer, prompts, n=10)
print(outputs[0])

Training is managed via the train_ranker.sh script. You can customize the training process by setting various environment variables or script parameters.

Key parameters for train_ranker.sh include:

• TORCHRUN_CMD: The path to your torchrun command (e.g., torchrun within a specific conda environment).
• dataset: The dataset to use for training.
• backbone_type: The architecture of the ranker (deberta or roberta).
• backbone_name: The specific model name (e.g., microsoft/deberta-v3-large or roberta-large).
• ranker: The type of ranker to use (PairRanker, Summaranker, or SimCLS).
• candidate_model: The model used to generate candidates (e.g., flan-t5-xxl or alpaca-native).
• candidate_decoding_method: The decoding strategy (e.g., top_p_sampling).
• n_candidates: The number of candidates to generate.
• using_metrics: The metrics used to train the signal (e.g., rouge1,rouge2,rougeLsum,bleu).
• do_inference: Set to False for training, or True for inference.
• inference_mode: When do_inference=True, set this to bubble or full to select the pairwise inference model.
• max_train_data_size, max_eval_data_size, max_predict_data_size: Limits the data size for each stage. Use -1 for no limit.
• checkpoint_trained_dataset: When performing inference on a new dataset (dataset=<A>) using a model trained on a different dataset, specify the original training dataset here (checkpoint_trained_dataset=<B>).

# Example of setting parameters for training
TORCHRUN_CMD=torchrun
dataset="my_dataset"
backbone_type="deberta"
backbone_name="microsoft/deberta-v3-large"
ranker="PairRanker"
do_inference=False

To compare multi-turn conversations, use the tokenize_conv_pair logic which transforms conversation histories into a format suitable for the PairRM model.

Conversation Format: Conversations must be a list of dictionaries with role and content keys. The format must strictly alternate: USER turns must be at even indices (0, 2, ...) and ASSISTANT turns must be at odd indices (1, 3, ...).

Logic:

1. The function takes convAs and convBs (lists of conversations).
2. It constructs a prompt by joining USER turns and appending an instruction: "Finish the following conversation in each i-th turn by filling in <Response i> with your response."
3. It formats the candidates by joining the ASSISTANT turns from each conversation with a <Response i>: prefix.
4. The resulting strings are then passed to the tokenize_pair function.

def tokenize_conv_pair(convAs: List[List[dict]], convBs: List[List[dict]]):
    # ... implementation details for formatting USER/ASSISTANT turns ...
    # 1. Validate even turns are USER and odd are ASSISTANT
    # 2. Construct instruction-based input strings
    # 3. Construct candidate strings with <Response i> prefixes
    # 4. Call tokenize_pair(inputs, cand1_texts, cand2_texts)
    pass

To convert a standard PairRM model into a Hugging Face compatible format, you must initialize a DebertaV2PairRM model with a specific configuration, add special tokens for source and candidates, and then load the original weights using safetensors.

Key steps:

1. Initialize DebertaV2Config from a base model (e.g., microsoft/deberta-v3-large).
2. Add special tokens: <|source|>, <|candidate1|>, <|candidate2|>, and <|candidate|>.
3. Set n_tasks, source_prefix_id, cand1_prefix_id, cand2_prefix_id, and cand_prefix_id in the config.
4. Resize token embeddings to match the new tokenizer length.
5. Load the .safetensors checkpoint into the model.
6. Use the Hugging Face Trainer to save the final checkpoint.

from llm_blender.pair_ranker.pairrm import DebertaV2PairRM
from transformers import DebertaV2Config, AutoTokenizer

config = DebertaV2Config.from_pretrained('microsoft/deberta-v3-large')
tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-v3-large')

source_prefix = "<|source|>"
cand1_prefix = "<|candidate1|>"
cand2_prefix = "<|candidate2|>"
cand_prefix = "<|candidate|>"
tokenizer.add_tokens([source_prefix, cand1_prefix, cand2_prefix, cand_prefix])

config.n_tasks = 1
config.source_prefix_id = 128001
config.cand1_prefix_id = 128002
config.cand2_prefix_id = 128003
config.cand_prefix_id = 128004
config.drop_out = 0.05

pairrm = DebertaV2PairRM(config)
pairrm.pretrained_model.resize_token_embeddings(len(tokenizer))

# Load weights
import safetensors
safetensors.torch.load_model(pairrm, "./PairRM/model.safetensors")

Use the Blender.rank method to rank multiple candidate responses for a given set of inputs.

1. Initialize Blender.
2. Load a ranker checkpoint using loadranker(checkpoint_path).
3. Call rank(inputs, candidates_texts, ...).

Parameters:

• inputs: A list of input strings.
• candidates_texts: A list of lists, where each inner list contains the candidate responses for the corresponding input.
• return_scores: Boolean (default False). If True, returns scores instead of integer ranks.
• batch_size: Integer determining how many inputs to process at once.

Output: Returns ranks, a list (or array) where ranks[i][j] represents the rank of candidate j for input i (e.g., rank 1 is the best).

import llm_blender
blender = llm_blender.Blender()
blender.loadranker("llm-blender/PairRM") # load ranker checkpoint

inputs = ["hello, how are you!", "I love you!"]
candidates_texts = [["get out!", "hi! I am fine, thanks!", "bye!"], 
                    ["I love you too!", "I hate you!", "Thanks! You're a good guy!"]]

ranks = blender.rank(inputs, candidates_texts, return_scores=False, batch_size=1)

If you do not want to install the full llm-blender package, you can use the DebertaV2PairRM model directly from Hugging Face. This is useful for custom development or lightweight integration.

Implementation Details:

• The model requires specific prefixes for tokenization: <|source|> for the input, <|candidate1|> for the first candidate, and <|candidate2|> for the second candidate.
• The comparison result is determined by whether the output logit is greater than 0.

import os
from llm_blender.pair_ranker.pairrm import DebertaV2PairRM
from transformers import AutoTokenizer

# Load model and tokenizer
pairrm = DebertaV2PairRM.from_pretrained("llm-blender/PairRM-hf", device_map="cuda:0").eval()
tokenizer = AutoTokenizer.from_pretrained('llm-blender/PairRM-hf')

# Tokenization requires manual concatenation of prefixes:
# source_prefix = "<|source|>"
# cand1_prefix = "<|candidate1|>"
# cand2_prefix = "<|candidate2|>"

# After tokenization and forward pass:
# outputs = pairrm(**encodings)
# comparison_results = outputs.logits > 0