FActScore

FActScore evaluates factual precision through a multi-step pipeline:

1. Atomic Fact Generation: The input text (generation) is broken down into individual, verifiable atomic facts using an LLM (typically davinci-003).
2. Retrieval: For each atomic fact, the system retrieves relevant context from a knowledge source (like Wikipedia) based on the topic.
3. Verification: An LLM (like ChatGPT) is prompted with the retrieved context and the atomic fact to determine if the fact is True or False based on that context.
4. Scoring: The individual verification results are averaged. If a gamma penalty is applied, the score is adjusted based on the number of atomic facts found, penalizing responses that are too brief to be informative.

If npm (Neural Probability Model) is enabled in the model_name, an additional probability check is performed to verify the support of the fact.

If you download the pre-computed unlabeled predictions from the provided Google Drive link, you can verify them against the statistics reported in the original paper (Table 5 and Figure 3) using the following verification script.

Each file in the unzipped directory corresponds to a subject LM. Each line in the file is a JSON dictionary containing:

• prompt: The initial prompt.
• facts: Atomic facts decomposed by the model.
• LLAMA+NP_labels: Labels for the facts verified by LLAMA+NP.
• ChatGPT_labels: Labels for the facts verified by ChatGPT.

Note: The number of lines may be less than 500 because cases where the model abstains are excluded. The response ratio is calculated as # of lines / 500.

import os
import json
import numpy as np

dirname = "factscore-unlabeled-predictions"
for fn in os.listdir(dirname):
    chatgpt_fs = []
    llama_fs = []
    n_facts = []
    with open(os.path.join(dirname, fn)) as f:
        for line in f:
            dp = json.loads(line)
            n_facts.append(len(dp["facts"]))
            if "ChatGPT_Labels" in dp:
                chatgpt_fs.append(np.mean([l=="S" for l in dp["ChatGPT_Labels"]]))
            llama_fs.append(np.mean([l=="S" for l in dp["LLAMA+NP_Labels"]]))
    print ("Model=%s\t(%.1f%% responding, %.1f facts/response)\tFactScore=%.1f (ChatGPT)\t%.1f (LLAMA)" % (
        fn.split(".")[0], len(n_facts)*100/500, np.mean(n_facts), np.mean(chatgpt_fs)*100, np.mean(llama_fs)*100
    ))

If you want to use a knowledge source other than the default Wikipedia dump, you must register it using register_knowledge_source.

Requirements:

• The source must be a .jsonl file where each line is a dictionary containing title and text.
• text can be a string or a list of strings (e.g., sections).

Note: Creating the database from a large file (like the 18GB English Wikipedia) can take significant time (e.g., ~8 hours). Once the db_path is created, you can reuse it by specifying it directly in future calls.

Usage:

from factscore.factscorer import FactScorer

fs = FactScorer()
fs.register_knowledge_source(
    name_of_your_knowledge_source,
    data_path=path_to_jsonl_file,
    db_path=path_to_output_db_file
)

from factscore.factscorer import FactScorer

fs = FactScorer()

# Registering a custom source
fs.register_knowledge_source(
    "my_custom_source",
    data_path="path/to/my_data.jsonl",
    db_path="path/to/output.db"
)

Use the FactScorer class to programmatically evaluate model generations against topics.

Usage:

1. Initialize FactScorer with your OpenAI key.
2. Provide a list of topics (strings representing Wikipedia titles) and generations (strings representing model outputs).
3. Call get_score() to retrieve the metrics.

Returned Dictionary Keys:

• score: The FActScore (factual precision).
• init_score: FActScore without the length penalty.
• respond_ratio: Percentage of responses that did not abstain.
• num_facts_per_response: Average number of atomic facts per response.

from factscore.factscorer import FactScorer

fs = FactScorer(openai_key="...")

# topics: list of strings (human entities used to generate bios)
# generations: list of strings (model generations)
topics = ["Albert Einstein"]
generations = ["Albert Einstein was a physicist..."]

out = fs.get_score(topics, generations, gamma=10)
print(out["score"]) # FActScore
print(out["init_score"]) # FActScore w/o length penalty
print(out["respond_ratio"]) # % of responding
print(out["num_facts_per_response"]) # average atomic facts

To compute the FActScore for a set of model generations, use the fs.get_score() method. You must provide a list of topics and a list of model generations. You can also specify a custom knowledge_source to use for the evaluation.

Parameters:

• topics: A list of strings representing the human entities used to generate the bios.
• generations: A list of strings representing the model's generated responses.
• knowledge_source (optional): The name of the knowledge source to use for verification.

Return Value: A dictionary containing the following keys:

• score: The computed FActScore.
• respond_ratio: The percentage of responses where the model did not abstain (e.g., did not say "I don't know").
• num_facts_per_response: The average number of atomic facts found per response.

# Specify knowledge source to use
out = fs.get_score(topics, generations, knowledge_source=name_of_your_knowledge_source)

print (out["score"]) # FActScore
print (out["respond_ratio"]) # % of responding (not abstaining from answering)
print (out["num_facts_per_response"]) # average number of atomic facts per response

Execute FActScore on a dataset provided in .jsonl format. Each line in the input file must contain a topic (Wikipedia title) and an output (the model generation).

Command Syntax:

python -m factscore.factscorer --input_path {input_path} --model_name {estimator_name} --openai_key {openai_key}

Arguments:

• --input_path: Path to the .jsonl file containing topic and output keys.
• --model_name: Supported values include retrieval+ChatGPT and retrieval+llama+npm (recommended). Other options include retrieval+ChatGPT+npm or retrieval+llama.
• --openai_key: Path to a file containing your OpenAI API Key.

Optional Flags:

• --data_dir: Directory containing knowledge source (default: .cache/factscore).
• --model_dir: Directory containing Inst-LLAMA weights (skip if model_name does not include llama).
• --cache_dir: Directory for API/model cache (default: .cache/factscore).
• --use_atomic_facts: Uses pre-released atomic facts instead of generating new ones (useful for reproducing paper results with low cost; cannot be used with new model generations).
• --gamma: Length penalty hyperparameter (default: 10). Set to 0 to disable length penalty.
• --n_samples: Run on a subset of the data.
• --verbose: Show progress bar.
• --print_rate_limit_error: Print OpenAI API rate limit errors.
• --cost_estimate: Estimation type for OpenAI API cost ("consider_cache" or "ignore_cache").
• --abstain_detection: Enables automatic detection of abstained responses. Supported detectors: "generic", "perplexity_ai".
• --knowledge_source: Specify a custom knowledge source name (must be preprocessed via register_knowledge_source).

python -m factscore.factscorer --input_path data/unlabeled/InstructGPT.jsonl --model_name retrieval+ChatGPT --openai_key my_key.txt

Use is_response_abstained(generation, fn_type) to determine if a model's generation should be excluded from evaluation due to low quality, refusal, or specific patterns associated with certain AI providers.

Supported fn_type values:

• "perplexity_ai": Uses logic tailored for Perplexity AI responses, checking for specific refusal phrases (e.g., "I could not find any information") and handling citation removal.
• "generic": Uses a simple check for common refusal patterns like starting with "I'm sorry" or containing "provide more".

If an unknown fn_type is provided, the function returns False.

from factscore.abstain_detection import is_response_abstained

# Example for Perplexity AI style responses
response = "I could not find any information regarding the subject."
should_abstain = is_response_abstained(response, "perplexity_ai") # Returns True

# Example for generic refusal detection
response = "I'm sorry, I cannot answer that."
should_abstain = is_response_abstained(response, "generic") # Returns True

The Retrieval class handles finding the most relevant document passages for a given topic and question using either keyword-based (BM25) or dense vector-based (GTR) methods.

Initialization

• db: An instance of DocDB.
• cache_path: Path to a JSON file for caching retrieval results.
• embed_cache_path: Path to a pickle file for caching embeddings.
• retrieval_type: The retrieval algorithm to use. Supported values:
  • "bm25"
  • Any string starting with "gtr-" (e.g., "gtr-t5-large").
• batch_size: Required if using a gtr- retrieval type. Determines the batch size for the encoder.

Key Methods

• get_passages(topic, question, k): The primary entry point. It constructs a retrieval_query by combining the topic and the question, then returns the top k most relevant passages.
• save_cache(): Persists the retrieval cache and embedding cache to the paths specified during initialization.

Retrieval Types

1. BM25: Uses rank_bm25.BM25Okapi for keyword-based scoring. It is faster but less semantically aware.
2. GTR (Generalizable T5 Retrieval): Uses a SentenceTransformer model (e.g., sentence-transformers/gtr-t5-large) to perform dense retrieval via cosine similarity (inner product).

from factscore.retrieval import DocDB, Retrieval

db = DocDB(db_path="my_database.db")
retriever = Retrieval(
    db=db,
    cache_path="retrieval_cache.json",
    embed_cache_path="embed_cache.pkl",
    retrieval_type="gtr-t5-large",
    batch_size=32
)

# Get top 5 passages for a question about a topic
passages = retriever.get_passages("Albert Einstein", "When was he born?", k=5)

# Save caches to disk
retriever.save_cache()