DOCSEARCHbyAlgolia
Submit Docs

Graph Chain-of-Thought (Graph-CoT)

repository·main·Indexed 43 days ago

https://github.com/petergriffinjin/graph-cot

A framework designed to augment Large Language Models with structured knowledge from textual graphs, enabling iterative, step-by-step reasoning through graph interaction. It includes GRBench, a benchmark for augmenting LLMs with external textual graphs, and supports inference for GPT-3.5-turbo, GPT-4, and open-source LLMs using direct, Text RAG, and Graph RAG methods.

Tokens
21.7K
Snippets
52
Records
67
Agent score
54%

What's inside Graph-CoT

  1. How Graph Chain-of-Thought (Graph-CoT) works

    main

    Graph-CoT is an iterative framework that enables LLMs to traverse a graph step-by-step to find key information instead of receiving a whole subgraph as context. Each iteration consists of three sub-steps:

    1. Reasoning: The LLM proposes a conclusion based on current information and identifies what further information is needed from the graph.
    2. Interaction: The LLM generates specific interactions required to fetch information (e.g., finding nodes, checking neighbors).
    3. Execution: The requested interactions are executed on the graph, and the resulting information is returned to the LLM.

    This process repeats until the LLM reaches a final conclusion in the reasoning step.

  2. Run Graph RAG Inference

    main

    To perform Graph RAG inference, execute run_llm_rag.sh with the following configuration requirements:

    • Set hop to 1.
    • max_len: Sets the maximum input length for the LLM.
    • graph_max_len: Truncates the retrieved context to ensure the total input does not exceed max_len. If you receive warnings that the total sequence length exceeds the LLM's maximum length, increase/adjust graph_max_len.
    cd scripts
bash run_llm_rag.sh

  3. Download raw datasets for Graph CoT

    main

    The project utilizes several external datasets for graph reasoning tasks. You can manually download the raw data from the following sources to prepare your environment:

    • Amazon: Metadata, user deduplication, and item deduplication files.
    • DBLP: Academic citation/author data.
    • Goodreads: Book metadata, authors, series, and genres.
    • Legal: CourtListener bulk data including citation maps, courts, dockets, opinion clusters, opinions, and parentheticals.
    • MAPLE: Specialized dataset from Zenodo.
    • Biology: Hetionet v1.0 JSON data.
    ### Amazon
http://snap.stanford.edu/data/amazon/productGraph/metadata.json.gz
http://snap.stanford.edu/data/amazon/productGraph/user_dedup.json.gz
http://snap.stanford.edu/data/amazon/productGraph/item_dedup.json.gz

### DBLP
https://originalfileserver.aminer.cn/misc/dblp_v14.tar.gz

### Goodreads
https://datarepo.eng.ucsd.edu/mcauley_group/gdrive/goodreads/goodreads_books.json.gz
https://datarepo.eng.ucsd.edu/mcauley_group/gdrive/goodreads/goodreads_book_authors.json.gz
https://datarepo.eng.ucsd.edu/mcauley_group/gdrive/goodreads/goodreads_book_series.json.gz
https://datarepo.eng.ucsd.edu/mcauley_group/gdrive/goodreads/goodreads_book_genres_initial.json.gz

### Legal
https://storage.courtlistener.com/bulk-data/citation-map-2023-08-31.csv.bz2
https://storage.courtlistener.com/bulk-data/courts-2023-08-31.csv.bz2
https://storage.courtlistener.com/bulk-data/dockets-2023-08-31.csv.bz2
https://storage.courtlistener.com/bulk-data/opinion-clusters-2023-08-31.csv.bz2
https://storage.courtlistener.com/bulk-data/opinions-2023-08-31.csv.bz2
https://storage.courtlistener.com/bulk-data/parentheticals-2023-08-31.csv.bz2

### MAPLE
https://zenodo.org/records/7611544/files/MAPLE.zip

### Biology
https://github.com/hetio/hetionet/blob/main/hetnet/json/hetionet-v1.0.json.bz2

  4. Run Graph RAG Inference for GPT-3.5-turbo / GPT-4

    main

    To perform Graph Retrieval-Augmented Generation (RAG) inference, navigate to the scripts directory and execute run_GPT_rag.sh.

    Configuration Requirements:

    • Set hop to 1.
    • max_len: Sets the maximum input length for the LLM.
    • graph_max_len: Truncates the retrieved context to ensure the total input does not exceed the LLM's max_len. If you receive warnings regarding total sequence length exceeding the LLM limit, increase/adjust graph_max_len accordingly.
    cd scripts
bash run_GPT_rag.sh

  5. Run Direct Inference for GPT-3.5-turbo / GPT-4

    main

    To perform direct inference using GPT-3.5-turbo or GPT-4, navigate to the scripts directory and execute the run_GPT.sh shell script.

    Note: You must replace the DATASET placeholder in the script with your specific domain name before running.

    cd scripts
bash run_GPT.sh

  6. Configure environment variables for Open-source LLM inference

    main

    Before running inference scripts, you must configure the following environment variables:

    • TRANSFORMERS_CACHE: Set this to your Hugging Face cache directory.
    • MODEL_NAME: Set this to your target model (e.g., llama-13b or mixtral 8*7b).
    • DATASET: Set this to your specific domain name.

  8. Install Graph Chain-of-Thought

    main

    The project requires Python 3.8. It is recommended to use a Conda virtual environment. Follow these steps to set up the environment and install dependencies:

    1. Create and activate a Conda environment named graphcot.
    2. Install PyTorch with CUDA 11.3 support.
    3. Install faiss-gpu version 1.7.4.
    4. Install remaining dependencies via requirements.txt.
    conda create --name graphcot python==3.8
conda activate graphcot
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
conda install -c pytorch -c nvidia faiss-gpu=1.7.4
pip3 install -r requirements.txt

  9. Run Graph-CoT and Baselines

    main

    The repository is organized into directories for different models. To run them, you must navigate to the specific directory and refer to its local README.md for detailed instructions.

    Important: You must update the bash variables within the scripts before execution.

    • Graph-CoT: Located in Graph-CoT/.
    • Open-source LLMs (with RAG): Located in LLM/.
    • GPT models (with RAG): Located in GPT/.

  10. Run Text RAG Inference for GPT-3.5-turbo / GPT-4

    main

    To perform Text Retrieval-Augmented Generation (RAG) inference, navigate to the scripts directory and execute run_GPT_rag.sh.

    Configuration Requirements:

    • Set hop to 0.
    • max_len: Sets the maximum input length for the LLM.
    • graph_max_len: Truncates the retrieved context to ensure the total input does not exceed the LLM's max_len. If you receive warnings regarding total sequence length exceeding the LLM limit, increase/adjust graph_max_len accordingly.
    cd scripts
bash run_GPT_rag.sh

  11. Construct GRBench from raw data

    main

    If you need to build the GRBench dataset from scratch, follow these steps:

    1. Download Raw Data

    Navigate to the specific dataset directory in data/raw_data/ and use wget to download the required web_data_dir. Specific directory paths are available in the repository's data/raw_data folder.

    cd data/raw_data/{data_name}
wget web_data_dir

    2. Process Raw Data

    Run the cells in raw_process.ipynb located within the data/raw_data/{data_name} directory. This script processes the raw data and outputs a graph.json file.

    3. Design Question Answering Pairs

    Once the graph is processed, generate the QA pairs:

    • Manual Design: Run design_questions.ipynb in data/processed_data/{data_name} to use manually designed question templates and automatic answer generation.
    • Diversification: Run generate_sample.ipynb (located in the parent directory of processed_data) to diversify question expressions using GPT-4.
    cd data/raw_data/{data_name}
wget web_data_dir

  12. Run Text RAG Inference

    main

    To perform Text RAG inference, execute run_llm_rag.sh with the following configuration requirements:

    • Set hop to 0.
    • max_len: Sets the maximum input length for the LLM.
    • graph_max_len: Truncates the retrieved context to ensure the total input does not exceed max_len. If you receive warnings that the total sequence length exceeds the LLM's maximum length, increase/adjust graph_max_len.
    cd scripts
bash run_llm_rag.sh