DOCSEARCHbyAlgolia
Submit Docs

OpenFactVerification (Loki)

repository·main·Indexed 22 days ago

https://github.com/libr-ai/openfactverification

Loki is an open-source automated fact-verification tool and pipeline that dissects long texts into claims, assesses their importance, and verifies them using multimodal inputs (string, text, speech, image, and video). It features a modular architecture allowing customization of evidence retrieval, language models (OpenAI, Anthropic, and local models), and prompts. The pipeline consists of five stages: Decomposer, Checkworthy, Query Generator, Evidence Retriever, and ClaimVerify.

Tokens
3.6K
Snippets
17
Records
20
Agent score
77%

What's inside openfactverification

  1. Overview of OpenFactVerification (Loki)

    main

    OpenFactVerification (also known as Loki) is an open-source fact-checking pipeline. It uses state-of-the-art language models to verify the veracity of textual claims. The pipeline is designed with modularity in mind, allowing users to customize three core components:

    1. Evidence Retrieval: How the system finds supporting or contradicting information.
    2. Language Model: The LLM used to perform the verification.
    3. Prompt: The specific instructions used to guide the fact-checking process.

  2. How the Loki fact-checking pipeline works

    main

    Loki uses a modular pipeline located in factcheck/core/ to verify textual claims. The process consists of five main stages:

    1. Decomposer: Breaks large texts into independent claims and provides a mapping between the original text and the claims.
    2. Checkworthy: Filters claims to remove vague, ambiguous, or opinion-based statements (e.g., "MBZUAI has a vast campus" is rejected due to the ambiguity of "vast").
    3. Query Generator: Converts check-worthy claims into precise search queries.
    4. Evidence Retriever: Fetches relevant evidence from the internet (currently uses Google Search via Serper API).
    5. ClaimVerify: Compares each piece of evidence against the claim to determine if it is supporting, refuting, or irrelevant.

    The pipeline relies on two core utilities:

    • Language Model (LLM): Used by the Decomposer, Checkworthy, Query Generator, and ClaimVerify components. Supported clients are in factcheck/core/utils/llmclient/.
    • Prompt: Optimized for specific LLMs and located in factcheck/core/utils/prompt/.

  3. Optimize and test prompts

    main

    Prompts are located in factcheck/utils/prompt/. To optimize a prompt for a specific LLM:

    1. Modify the prompt file in factcheck/utils/prompt/.
    2. Run the minimal test suite using script/minimal_test.py to verify changes.
    3. To ensure long-term reliability, add new test cases to script/minimal_test.json.
    python script/minimal_test.py

  4. Install Loki (OpenFactVerification)

    main

    You can install Loki using either Poetry or pip. Ensure you have Python 3.9 or newer installed.

    Option 1: Using Poetry

    1. Install Poetry.
    2. Run poetry install in the project directory.

    Option 2: Using pip

    1. Create and activate a Python 3.9+ environment.
    2. Run pip install -r requirements.txt in the project directory.
    # Using Poetry
poetry install

# Using pip
pip install -r requirements.txt

  5. Prepare code style with pre-commit

    main

    Loki uses black and flake8 to enforce code style. Before submitting a pull request, you should install and run pre-commit to ensure your code is properly formatted and aligned with the project's standards.

    pip install pre-commit
pre-commit install
pre-commit run --all-files

  6. Support a new LLM client

    main

    To add support for a new Large Language Model, you must create a new client in factcheck/core/utils/llmclient/ that inherits from BaseClient (defined in factcheck/core/utils/llmclient/base.py).

    Requirements:

    • Implement the _call method: It must accept a single string input and return a string output.
    • Output Format: To maintain pipeline sanity, the LLM output must be a string representation of compiled Python code (e.g., a stringified list or dict) so it can be parsed using Python's eval() method.
    • Post-processing: If the LLM provides structured output (like ChatGPT's json_mode), you may need to implement post-processing to ensure it conforms to the expected format.
    # Conceptual implementation
from factcheck.core.utils.llmclient.base import BaseClient

class MyNewLLMClient(BaseClient):
    def _call(self, input_str: str) -> str:
        # Implementation that returns a stringified list or dict
        return "['result1', 'result2']"

  7. Verify Multimodal Inputs

    main

    Loki unifies different modalities (text, speech, image, and video) by converting them into text before running the standard verification pipeline. Use the --modal flag to specify the input type.

    Supported modalities:

    • string: Raw text string.
    • text: Path to a text file.
    • speech: Path to an audio file (e.g., .mp3).
    • image: Path to an image file (e.g., .webp).
    • video: Path to a video file (e.g., .m4v).
    # String input
python -m factcheck --modal string --input "Your text"

# Text file input
python -m factcheck --modal text --input demo_data/text.txt

# Speech input
python -m factcheck --modal speech --input demo_data/speech.mp3

# Image input
python -m factcheck --modal image --input demo_data/image.webp

# Video input
python -m factcheck --modal video --input demo_data/video.m4v

  8. Switch Between LLM Models and Clients

    main

    Loki supports OpenAI, Anthropic, and local models. Use --model to specify the version and --client to specify the provider type (primarily for local models).

    Model Type--model--client
    OpenAIgpt-VERSIONNone
    Anthropicclaude-VERSIONNone
    LocalMODEL_NAMElocal_openai

    Note: When using local_openai, you must provide LOCAL_API_KEY and LOCAL_API_URL.

    # OpenAI
python -m factcheck --modal string --input "text" --model gpt-4-turbo

# Anthropic
python -m factcheck --modal string --input "text" --model claude-3-opus-20240229

# Local
python -m factcheck --modal string --input "text" --client local_openai --model wizardlm2

  9. Customize Prompts

    main

    You can override the default prompts used for claim decomposition, checkworthiness, query generation, and verification by providing a custom YAML or JSON file via the --prompt flag.

    Refer to factcheck/config/sample_prompt.yaml for the expected structure.

    python -m factcheck --input "Your text" --prompt PATH_TO_PROMPT/custom_prompt.yaml

  10. Support a new search engine (Retriever)

    main

    To implement a new evidence retrieval mechanism, create a new retriever in factcheck/core/Retriever/ that inherits from EvidenceRetriever (defined in factcheck/core/Retriever/base.py).

    You must implement the retrieve_evidence method to handle the search logic.

    # Conceptual implementation
from factcheck.core.Retriever.base import EvidenceRetriever

class MyNewRetriever(EvidenceRetriever):
    def retrieve_evidence(self, query: str):
        # Implementation to fetch evidence
        pass

  11. Switch Between Search Engines

    main

    You can choose between different search engines for evidence retrieval using the --retriever argument. Supported options are serper and google.

    # Use Serper
python -m factcheck --modal string --input "text" --retriever serper

# Use Google
python -m factcheck --modal string --input "text" --retriever google

  12. Install OpenFactVerification

    main

    You can install the project using either Poetry or pip.

    Option 1: Poetry

    1. Install Poetry.
    2. Run poetry install in the project directory.

    Option 2: pip

    1. Ensure you have Python 3.9 or newer.
    2. Create and activate a virtual environment.
    3. Run pip install -r requirements.txt in the project directory.
    # Clone the repository
git clone https://github.com/Libr-AI/OpenFactVerification.git
cd OpenFactVerification

# Option 1: Poetry
poetry install

# Option 2: pip
pip install -r requirements.txt