MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents

When using the Bespoke-MiniCheck-7B model, you can enable Automatic Prefix Caching (APC) to significantly increase throughput and reduce latency. APC works by caching the KV cache of a document; if you check multiple different claims against the same document, the model reuses the cached document prefix instead of recomputing it.

To enable this, set enable_prefix_caching=True during initialization.

# Enable APC for faster inference when reusing the same document
scorer = MiniCheck(model_name='Bespoke-MiniCheck-7B', enable_prefix_caching=True, cache_dir='./ckpts')

To use the OpenAI API for generating synthetic data in the C2D or D2C pipelines, you must manually add your OPENAI_API_KEY to the synthetic_data_gen/prompt_utils.py file.

# Edit synthetic_data_gen/prompt_utils.py to include:
OPENAI_API_KEY = 'your-api-key-here'

C2D (Claim-to-Document) data generation starts with a human-written claim. The process generates synthetic documents designed to require multi-sentence, multi-fact reasoning to verify the claim. This produces (doc, claim, label) triples.

Run the C2D_gen.py script with the --claim flag to specify the input claim.

cd synthetic_data_gen
python C2D_gen.py --claim "By this date, over 5,000 members of the caravan were staying at the Tijuana Stadium — a structure with a capacity of 3,000."

You can install the core MiniCheck package and its dependencies using pip. If you intend to use the Bespoke-MiniCheck-7B model for inference, you should install the [llm] extra, which includes the vllm dependency (note: vllm support is Linux only).

# Standard installation
pip install "minicheck @ git+https://github.com/Liyan06/MiniCheck.git@main"

# Installation with vllm support for Bespoke-MiniCheck-7B (Linux only)
pip install "minicheck[llm] @ git+https://github.com/Liyan06/MiniCheck.git@main"

D2C (Document-to-Claim) data generation starts with human-written documents. The process generates claims paired with specific portions of the document, requiring multi-sentence, multi-fact reasoning to check them. This produces (doc, claim, label) triples.

Note: The generation process assumes a document length of approximately 500 words. Longer documents increase costs due to the "leave-one-sent-out" document augmentation method used.

Run the D2C_gen.py script with the --doc_path flag to specify the path to the input document.

cd synthetic_data_gen
python D2C_gen.py --doc_path D2C-doc-example.txt

When using LLMCheck, the system uses a technique called SentenceFusion to handle multi-sentence claims:

1. Chunking: The document is split into chunks of size chunk_size.
2. Sentence Splitting: The claim is split into individual sentences.
3. Matrix Construction: For every document chunk and every claim sentence, a probability is calculated, forming a matrix of shape (num_chunks x num_sentences).
4. Max-over-Chunks: For each sentence, the maximum probability across all document chunks is selected: max_prob_per_sentence = np.max(prob_matrix, axis=0).
5. Min-over-Sentences: The final score is the minimum of these maximum values: final_score = np.min(max_prob_per_sentence). This ensures that if any part of the claim is unsupported, the overall score reflects that lack of support.

The minicheck package requires the NLTK punkt tokenizer. The package attempts to verify the existence of this resource on import and automatically downloads it via nltk.download('punkt') if it is missing. If you encounter issues with tokenization, ensure your environment has internet access to allow NLTK to fetch the required data.

import nltk

try:
    nltk.data.find('tokenizers/punkt')
except LookupError:
    nltk.download('punkt')

The LLM-AggreFact benchmark is available on HuggingFace. It aggregates 11 datasets for factual consistency evaluation. The dataset contains the following fields:

from datasets import load_dataset
dataset = load_dataset("lytang/LLM-AggreFact")

If you are using Bespoke-MiniCheck-7B on a GPU with low VRAM, you may encounter the following error:

ValueError: The model's max seq len (XXXX) is larger than the maximum number of tokens that can be stored in KV cache (YYYY). Try increasing 'gpu_memory_utilization' or decreasing 'max_model_len' when initializing the engine.

Solution: When initializing the MiniCheck class, explicitly set max_model_len to a smaller value than the default (32768).

Use the scorer.score() method to evaluate whether claims are supported by grounding documents.

Parameters:

• docs: An array/list of grounding documents.
• claims: An array/list of claims to be checked.

Returns:

• pred_label: Binary labels (1 or 0) indicating if the claim is supported. Note: The internal logic converts raw probabilities to 1/0 using a threshold of 0.5.
• raw_prob: The raw probability scores.
• Two additional return values (ignored in standard usage).

Performance Note: Using Bespoke-MiniCheck-7B (via vLLM) on a single NVIDIA A6000 (48GB VRAM) provides a throughput of > 500 docs/min.

# docs and claims should be arrays of strings
pred_label, raw_prob, _, _ = scorer.score(docs=docs, claims=claims)