ChemBench

ChemBench is a benchmarking suite designed to evaluate Large Language Models (LLMs) on chemistry tasks. It contains over 2,700 question-answer pairs covering knowledge, intuition, and reasoning.

Key features include:

• Diverse Question Types: Supports more than simple Multiple Choice Questions (MCQs), including floating-point answers and scientific notation.
• Specialized Prompting: Provides separate prompting strategies for Completion models, Instruction-tuned models, and special string decorations like SMILES.
• Advanced Evaluation: Includes refusal detection (identifying when a model refuses to answer), confidence prompting (asking models to estimate their own certainty), and scientifically sound parsing for numerical values.
• Extensibility: Uses a pipeline system that allows users to add or customize prompts, models, extraction methods, and evaluation methods.
• Unified Model Interface: Supports LLMs, VLLMs, and other models, often utilizing LiteLLM as a backend.

Every question must include specific keywords to facilitate filtering and evaluation. All keywords must be lowercase and use - as a separator.

Required Keyword Pools

• Requirement Pool (Must include at least one):
  • requires-knowledge: Relies on pre-existing scientific facts/formulas.
  • requires-reasoning: Requires logical inference or data interpretation.
  • requires-calculation: Requires mathematical/quantitative processing.
  • requires-tooluse: Requires external tools (calculators, code environments).
  • requires-ocr: (Multimodal only) Requires extracting text from images.
• Difficulty Pool (Must include at least one):
  • difficulty-basic: Undergraduate level concepts.
  • difficulty-advanced: Graduate level or multi-subject concepts.

Optional Keywords

• Domain/Specialization: e.g., material-science, solid-state-battery.
• Task Type: multiple-correct-answer, requires-multiple-steps.

ChemBench uses a multi-step parsing workflow to extract answers from model completions to ensure robust and fair evaluation. The strategy differs based on the model type:

1. Prompting Strategy

• Instruction-tuned models: Prompted to use a specific [ANSWER][\ANSWER] environment to report the final answer.
• Completion models: Standard completion prompting is used.
• Consistency: The workflow post-processes LaTeX syntax, chemical symbols, equations, and physical units to ensure consistent parsing.

2. Parsing Workflow

• Regex-based Extraction: The primary method uses regular expressions to identify the answer environment and extract enumeration letters (for MCQs) or numbers (for numeric tasks).
• Scientific Notation: Regex is specifically engineered to handle various forms of scientific notation and convert word-based numbers to floats.
• LLM Fallback: If hard-coded regex parsing fails, the system uses the same LLM being tested as a fallback to parse its own completion.

3. Refusal Detection

To handle cases where a model refuses to answer, ChemBench employs a pipeline combining:

• LLM Guard: Regular expression-based detection.
• BERT Model: A fine-tuned distilroberta-base-rejection-v1 model.
• Retries: If a refusal is detected, the system can perform up to n retry attempts (where n is user-defined).

ChemBench evaluates model capabilities in chemistry using a primary metric called Fraction Correct. This metric is an aggregation of scores from two types of questions: Multiple-Choice Questions (MCQ) and Numeric questions.

Scoring Rules

• Binary Scoring: For any single question, the score is either 0 or 1 (represented as all_correct in the benchmark).
• MCQ (Multiple-Choice): A question is correct if the Hamming Loss is 0. For single-answer MCQs, this means the predicted label matches the true label.
• Numeric: A question is correct if the Mean Absolute Error (MAE) is within a 1% threshold of the target value: $$ ext{MAE}_j < 0.01 imes | ext{Target}_j|$$

Final Score Calculation

The final score is the mean of the scores of all questions across all topics: $$ ext{Fraction Correct} = rac{1}{n} imes ext{Total Correct Questions}$$

Key Limitations

• No Partial Credit: There is no partial scoring for open-ended or complex answers; a question is strictly correct or incorrect.
• Equal Weighting: All questions are treated with equal weight regardless of difficulty level.
• Threshold Sensitivity: Numeric accuracy is highly dependent on the 1% MAE threshold.

\text{Fraction Correct} = \frac{1}{n} \left( \sum_{i \in \text{MCQ}} (1 - \text{HammingLoss}_i) + \sum_{j \in \text{Numeric}} \mathbb{1}(\text{MAE}_j < 0.01 \times |\text{Target}_j|) \right)

Benchmarking vision models follows the same pattern as text models but requires handling multi-modal inputs.

Implementation Details:

• Message Transformation: Since ChemBench uses a standard message format (similar to LiteLLM), you may need a helper method (e.g., _transform_message) to convert these messages into the specific format your model's processor expects (e.g., handling image_url or base64 data).
• Prompt Formatting: You must implement logic to process both text and image/video inputs using your model's specific processor (e.g., AutoProcessor from HuggingFace).
• Standard Message Format: ChemBench provides messages in a format where content can be a list of objects containing type: "text" or type: "image_url" (with base64 encoded data).

Workflow:

1. Implement a class with a .generate(prompts: List[str], **kwargs) method.
2. Use PromterBuilder.from_model_object(model=your_vision_model).
3. Run benchmark.bench(prompter).

Note: For multi-modal benchmark datasets, use jablonkagroup/MaCBench when initializing the benchmark.

from chembench.types import Generation, Generations
from chembench.evaluate import ChemBenchmark
from chembench.prompter import PrompterBuilder

class QwenVL:
    # ... implementation of _transform_message, format_prompt, and generate_text ...

    def generate(self, prompts: List[str], **kwargs):
        generations = []
        for prompt in prompts:
            generation = self.generate_text(prompt, **kwargs)
            generations.append([Generation(text=generation)])
        return Generations(generations=generations)

benchmark = ChemBenchmark.from_huggingface()
model = QwenVL()
prompter = PrompterBuilder.from_model_object(model=model)
results = benchmark.bench(prompter)