PIKE-RAG Framework

PIKE-RAG uses a standardized QA data protocol to ensure consistency across different datasets. Every QA entry is represented as a dictionary (Dict) containing a unique ID, the question, a list of correct answer labels, and a question type. Metadata is stored in a metadata dictionary to preserve dataset-specific information like original IDs, difficulty levels, and supporting facts.

qa: Dict = {
    # str: The unique qa id generated with uuid.uuid4().hex.
    "id": "a unique qa id",

    # str: The question to be answered.
    "question": "the question to be answered",

    # List[str]: A list of correct answers, could be a list of single value for some datasets.
    "answer_labels": ["a list of correct answers", "there could be only one answer for some datasets"],

    # Literal["yes_no", "undefined"]: may be extended in the future.
    #   "yes_no" indicates the answer should be in ["yes", "no"].
    "question_type": "undefined",

    # Dict[str, Union[str, List]]: The set of metadata information, varied among different datasets.
    "metadata": {
        # Union[str, int] if exists: The qa id defined in the original dataset.
        "original_id": "the id of type str or int if exists",

        # str if exists: The question type defined in the original dataset. Values varied among different datasets.
        "original_type": "bridge",

        # Literal["easy", "median", "hard"] if exists: The difficulty level defined in the original dataset.
        #   Currently, it only exists for HotpotQA.
        "original_level": "hard",

        # List[Dict]: The supporting facts that are useful to answer the question, if exists.
        #   It corresponds to the "supporting_facts"/"evidence_span"/"long_answers"/... in the original datasets.
        #   Refers to the introduction of each dataset for more details.
        "supporting_facts": [
            {
                # Literal["wikipedia", "wikidata", "BingSearch"]: The type of the supporting fact,
                #   may be extended in the future.
                #   Each type of supporting fact corresponds to a specific set of valid keys in the dict.
                "type": "wikipedia",

                # str: The supporting wikipedia title.
                "title": "wikipedia example 1",

                # str: The supporting wikipedia contents. May be a long string with multiple sentences.
                "contents": "wikipedia contents example 1",
            },
            {
                "type": "wikidata",

                # str: The supporting wikidata title.
                "title": "wikidata title example 1",

                # str: The supporting wikidata section.
                "section": "wikidata section example 1",

                # str: The supporting wikidata contents in the specified section.
                "contents": "wikidata contents example 1",
            },
        ],

        # List[Dict]: The retrieval contexts provided by the original dataset, if exists.
        #   It corresponds to the "contexts"/... in the original datasets.
        #   Refer to the introduction of each dataset for more details.
        "retrieval_contexts": [
            {
                # Literal["wikipedia", "wikidata", "BingSearch"]: The type of the retrieval contexts,
                #   may be extended in the future. Each type of retrieval contexts correspond to a specific set of valid keys in the dict.
                "type": "wikipedia",

                # str: The wikipedia title of the contexts.
                "title": "wikipedia example 2",

                # str: The wikipedia contexts. May be a long string with multiple sentences.
                "contents": "wikipedia contexts example 2",
            },
            {
                "type": "wikidata",

                # str: The wikidata title of the contexts.
                "title": "wikidata title example 2",

                # str: the wikidata section of the contexts.
                "section": "wikidata section example 2",

                # str: The wikidata contents in the specified section.
                "contents": "wikidata contents example 2",
            },
            {
                "type": "BingSearch",

                # str: The title of the returned item.
                "title": "Bing search title example 1",

                # str: The url of the returned item.
                "url": "http://example.url.com",

                # str: The simple description showed in the search page.
                "description": "Bing search description example 1",

                # str: The contents of the returned item.
                "contents": "Bing search contents example 1",

                # int: The search rank of the item in the search results.
                "rank": 1,
            },
        ],
        "reasoning_logics": [
            {
                "type": "wikidata",

                # str: The supporting wikidata title.
                "title": "wikidata title example 3",

                # str: The supporting wikidata section.
                "section": "wikidata section example 3",

                # str: The supporting wikidata contents in the specified section.
                "contents": "wikidata contents example 3",
            },
        ]
    },
}

The PIKE-RAG open benchmarks follow a standardized dataset protocol for Question Answering (QA). Every QA entry is structured with a unique ID, a question, a list of correct answers, and a metadata dictionary.

Core Schema Fields:

• str: The unique qa_id generated via uuid.uuid4().hex.
• str: The question to be answered.
• List[str]: A list of correct answers.
• Literal["yes_no", "undefined"]: Indicates if the answer format is constrained to ["yes", "no"].
• Dict[str, Union[str, List]]: The metadata object containing dataset-specific information.

Common Metadata Keys:

• original_id (str): The ID from the source dataset.
• original_type (str): The question type defined in the original dataset.
• original_level (Literal["easy", "median", "hard"]): The difficulty level.
• supporting_facts (List[Dict]): A list of dictionaries containing the evidence used to answer the question. Each dictionary typically includes type, title, and contents.

You can split documents into chunks using several methods by configuring a YAML file and running examples/chunking.py.

Methods:

1. Context-aware chunking: Uses LLMs to split markdown files (see examples/biology/configs/chunking.yml).
2. RecursiveSentenceSplitter: A lighter splitter that does not require LLM calls.
3. Third-party Splitters: Supports existing splitters like langchain.text_splitter.TextSplitter.

To run the chunking task:

python examples/chunking.py PATH-TO-YAML-CONFIG

# Example: Using RecursiveSentenceSplitter
splitter:
    module_path: pikerag.document_transformers
    class_name: RecursiveSentenceSplitter
    args:
        ...  # Configure according to your need

# Example: Using LangChain Splitter
splitter:
    module_path: langchain.text_splitter
    class_name: TextSplitter
    args:
        ...  # Configure according to your need

The AzureMetaLlamaClient allows you to manage multiple Llama models by defining pairs of endpoint names and API keys in your .env file. When initializing the client in your YAML configuration, specify the names of these pairs.

Option 1: Single Model

Use the default variable names LLAMA_ENDPOINT and LLAMA_API_KEY. You must update these every time you want to switch models.

LLAMA_ENDPOINT = "YOUR-LLAMA-ENDPOINT"
LLAMA_API_KEY = "YOUR-API-KEY"

Option 2: Multiple Models

Define custom pairs for different models (e.g., LLAMA3_8B_...).

LLAMA3_8B_ENDPOINT = "..."
LLAMA3_8B_API_KEY = "..."

LLAMA3_70B_ENDPOINT = "..."
LLAMA3_70B_API_KEY = "..."

# Option 1: Set only one pair in one time, update these variables every time you want to change the LlaMa model.
LLAMA_ENDPOINT = "YOUR-LLAMA-ENDPOINT"
LLAMA_API_KEY = "YOUR-API-KEY"

# Option 2: Add multiple pairs into the .env file, for example:
LLAMA3_8B_ENDPOINT = "..."
LLAMA3_8B_API_KEY = "..."

LLAMA3_70B_ENDPOINT = "..."
LLAMA3_70B_API_KEY = "..."