DOCSEARCHbyAlgolia
Submit Docs

FrugalGPT

repository·main·Indexed 42 days ago

https://github.com/stanford-futuredata/frugalgpt

A framework for building LLM applications that balance high quality with low cost. It provides tools to optimize LLM usage, evaluate performance and cost tradeoffs across different models, and implement LLM cascades using trained strategies. The framework includes utilities for batch completion via LLMforAll, performance scoring with compute_score, and support for multiple providers including OpenAI, Anthropic, Gemini, and Cohere.

Tokens
16.4K
Snippets
54
Records
57
Agent score
53%

What's inside FrugalGPT

  1. Quickstart with FrugalGPT via Google Colab

    main

    If you want to experience FrugalGPT without setting up a local environment or managing API keys, you can use the official Google Colab Notebook. This is the fastest way to see the framework in action.

    https://colab.research.google.com/drive/1LM-Wq-u87VI4TKM4thpnwepnOxTAtWaM?authuser=1#scrollTo=a95a1eec

  2. Install FrugalGPT locally

    main

    To install FrugalGPT on your local machine, you need to clone the repository, install the package via pip, and download the required strategy and database files.

    Follow these steps:

    1. Clone the repository.
    2. Install the package using pip install from the GitHub URL.
    3. Download the HEADLINES.zip file, unzip it into the strategy/ directory, and remove the zip file.
    4. Download the required .sqlite database files into the db/ directory.
    git clone https://github.com/stanford-futuredata/FrugalGPT
cd FrugalGPT
pip install git+https://github.com/stanford-futuredata/FrugalGPT
wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.1/HEADLINES.zip
unzip HEADLINES.zip -d strategy/
rm HEADLINES.zip
wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.1/HEADLINES.sqlite
wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.1/qa_cache.sqlite

  3. Use the Strategy class to manage LLM budgets and evaluations

    main

    The Strategy class is the base class for defining how LLM operations are optimized and evaluated. It manages a budget (defaulting to 100) and provides methods for evaluating performance against labels.

    Key methods:

    • setbudget(budget): Sets the maximum allowable budget for the strategy.
    • evaluate(responses, labels): Evaluates a batch of responses against ground truth labels using utils.evaluate_batch.
    • loadstrategy(strategy_path): Loads a previously saved strategy from a JSON file. It populates self.thres, self.model_ids, and self.quantile based on the provided budget.
    • savestrategy(strategy_path): Persists the current strategy (thresholds, model lists, and quantiles) to a JSON file.
    from FrugalGPT.llmchain import Strategy

strategy = Strategy(metric="em_mc")
strategy.setbudget(10)
# ... perform training or loading ...
strategy.savestrategy("my_strategy.json")

  4. Train a FrugalGPT LLM Cascade strategy

    main

    The FrugalGPT.LLMCascade class is used to train a strategy that optimizes for a specific budget. The training process involves learning which models to call in a cascade to maximize accuracy while staying under a cost threshold.

    Key Parameters for LLMCascade.train:

    • train_data: The dataset used for training.
    • budget: The target cost budget.
    • service_names: A list of available LLM services (e.g., ['provider/model']).
    • no_scorer_train: Boolean. If True, skips training the scorer (used for subsequent budget evaluations).
    • prefix: Prompt prefix.
    • cascade_depth: The maximum number of models in the cascade.
    • score_test_size: The fraction of data used for scoring during training.

    Workflow:

    1. Initialize LLMCascade with db_path and batch_build=True.
    2. Call .train() to optimize the cascade for a specific budget.
    3. Call .save() to persist the trained strategy.
    4. Use .load(loadpath=..., budget=...) to load a specific strategy for evaluation.
    MyCascade = FrugalGPT.LLMCascade(db_path=f"db/{dataname}.sqlite", batch_build=True)

# Training the cascade for a specific budget
MyCascade.train(
    train_data=train_data,
    budget=0.00017,
    service_names=service_names_train,
    no_scorer_train=False,
    prefix=prefix,
    cascade_depth=3,
    score_test_size=0.45
)

# Save the trained strategy
MyCascade.save(savepath=f"strategy/{name}/")

  5. Configure API keys for LLM services

    main

    If you intend to use FrugalGPT for your own queries (rather than just running the provided analysis notebooks), you must set the environment variables for the LLM providers you wish to use.

    Supported providers include:

    • OPENAI_API_KEY
    • AI21_STUDIO_API_KEY
    • COHERE_STUDIO_API_KEY
    • TEXTSYNTH_API_SECRET_KEY
    • ANTHROPIC_API_KEY
    • TOGETHER_API_KEY
    • GEMINI_API_KEY

    You only need to set keys for the specific providers you plan to leverage.

    import os
os.environ['OPENAI_API_KEY'] = 'OPENAI_API_KEY'
os.environ['AI21_STUDIO_API_KEY'] = 'AI21_STUDIO_API_KEY'
os.environ['COHERE_STUDIO_API_KEY'] = 'COHERE_STUDIO_API_KEY'
os.environ['TEXTSYNTH_API_SECRET_KEY'] = 'TEXTSYNTH_API_SECRET_KEY'
os.environ['ANTHROPIC_API_KEY'] = 'ANTHROPIC_API_KEY'
os.environ['TOGETHER_API_KEY'] = 'TOGETHER_API_KEY'
os.environ['GEMINI_API_KEY'] = 'GEMINI_API_KEY'

  6. Install FrugalGPT and required data assets

    main

    To use the FrugalGPT framework for generation analysis, you must clone the repository, install the package via pip, and download specific datasets and strategy files required for the analysis.

    Note: The installation process involves downloading specific .zip and .sqlite files from external releases to populate the FrugalGPT/strategy/ and FrugalGPT/db/ directories.

    # Clone and install
! git clone https://github.com/stanford-futuredata/FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT

# Setup directories and download assets
!mkdir -p FrugalGPT/strategy/
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.3/HEADLINES_Model2024_New.zip
! unzip HEADLINES_Model2024_New.zip -d FrugalGPT/strategy/HEADLINES_Model2024_New
! rm HEADLINES_Model2024_New.zip
! wget -P FrugalGPT/db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.3/HEADLINES.sqlite

%cd FrugalGPT

  7. Initialize LLMCascade for cost-effective model cascading

    main

    The LLMCascade class implements strategies to use cheaper LLMs first and only escalate to more expensive models when a quality threshold is not met.

    To initialize, provide a metric (e.g., 'em' for exact match), a path to your database, and optional configuration for noise injection or batch building.

    from FrugalGPT.llmcascade import LLMCascade
from service.modelservice import GenerationParameter

# Initialize the cascade engine
cascade = LLMCascade(
    metric="em",
    db_path='db/HEADLINES.sqlite',
    score_noise_injection=False,
    batch_build=False
)
    from FrugalGPT.llmcascade import LLMCascade

cascade = LLMCascade(
    metric="em",
    db_path='db/HEADLINES.sqlite',
    score_noise_injection=False,
    batch_build=False
)

  8. Install FrugalGPT and setup environment

    main

    To use FrugalGPT, clone the repository, install the package via pip, and set up the necessary directory structure for strategies and data. This example also demonstrates downloading specific model weights and datasets required for the OVERRULING evaluation.

    Note: It is highly recommended to use accelerated hardware (GPU/TPU) when running FrugalGPT workflows.

    # set up the environment
! git clone https://github.com/stanford-futuredata/FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p FrugalGPT/strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.2/OVERRULING_Model20241128.zip
! unzip OVERRULING_Model20241128.zip -d FrugalGPT/strategy/OVERRULING_Model20241128
! rm OVERRULING_Model20241128.zip
! wget -P FrugalGPT/db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.3/OVERRULING.sqlite
!mkdir -p FrugalGPT/data/OVERRULING
!wget -P FrugalGPT/data/OVERRULING https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/OVERRULING_train.csv
!wget -P FrugalGPT/data/OVERRULING https://github.com/stanford-futuredata/FrugalGPT/releases/download/0.0.1/OVERRULING_test.csv

%cd FrugalGPT

  9. Configure API keys for LLM providers

    main

    FrugalGPT requires API keys to interact with various LLM providers. If you are only using specific providers (e.g., OpenAI and AI21), you only need to set those keys. Set the following environment variables:

    • OPENAI_API_KEY
    • AI21_STUDIO_API_KEY
    • COHERE_STUDIO_API_KEY
    • TEXTSYNTH_API_SECRET_KEY
    • ANTHROPIC_API_KEY
    • TOGETHER_API_KEY
    • GEMINI_API_KEY
    import os
os.environ['OPENAI_API_KEY'] = 'YOUR_OPENAI_KEY'
os.environ['AI21_STUDIO_API_KEY'] = 'YOUR_AI21_KEY'
os.environ['COHERE_STUDIO_API_KEY'] = 'YOUR_COHERE_KEY'
os.environ['TEXTSYNTH_API_SECRET_KEY'] = 'YOUR_TEXTSYNTH_KEY'
os.environ['ANTHROPIC_API_KEY'] = 'YOUR_ANTHROPIC_KEY'
os.environ['TOGETHER_API_KEY'] = 'YOUR_TOGETHER_KEY'
os.environ['GEMINI_API_KEY'] = 'YOUR_GEMINI_KEY'

  10. Train and save a custom LLMCascade model

    main

    To build a custom LLMCascade for your specific data:

    1. Load Data: Use FrugalGPT.loadcsvdata(path) to load your dataset and FrugalGPT.formatdata(data, prefix) to format it for the model.
    2. Train: Call MyCascade.train(data, budget, service_names) where budget is the target budget per query and service_names is a list of LLM services to include in the cascade.
    3. Save: Use MyCascade.save(savepath) to persist the trained strategy.
    4. Load: Use MyCascade.load(loadpath, budget) to reuse the strategy later.
    # 1. Load and format
dev = FrugalGPT.loadcsvdata("data/HEADLINES/train.csv")
prefix = open('config/prompt/HEADLINES/prefix_e8.txt').read()
data = FrugalGPT.formatdata(dev, prefix)

# 2. Train
MyCascade = FrugalGPT.LLMCascade()
service_names = ['openaichat/gpt-3.5-turbo', 'openaichat/gpt-4', 'ai21/j1-large']
MyCascade.train(data, budget=100, service_names=service_names)

# 3. Save
MyCascade.save(savepath="strategy/TEST/")

# 4. Load later
new_cascade = FrugalGPT.LLMCascade()
new_cascade.load(loadpath="strategy/TEST/", budget=100)

  11. Install FrugalGPT and prepare the environment

    main

    To use FrugalGPT, clone the repository, install the package via pip, and set up the necessary strategy and database files. Note that running performance/cost tradeoff evaluations is highly recommended to be done on accelerated hardware (GPU/TPU).

    # Clone and install
git clone https://github.com/stanford-futuredata/FrugalGPT
cd FrugalGPT
pip install git+https://github.com/stanford-futuredata/FrugalGPT

# Setup directories and sample data
mkdir -p strategy
wget https://github.com/lchen001/DataHolder/releases/download/v0.0.2/HEADLINES_Model2024.zip
unzip HEADLINES_Model2024.zip -d strategy/
rm HEADLINES_Model2024.zip

# Setup database
mkdir -p db/
wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.2/HEADLINES.sqlite
    # set up the environment
%%capture
! git clone https://github.com/stanford-futuredata/FrugalGPT
%cd FrugalGPT
! pip install git+https://github.com/stanford-futuredata/FrugalGPT
!mkdir -p strategy
! wget  https://github.com/lchen001/DataHolder/releases/download/v0.0.2/HEADLINES_Model2024.zip
! unzip HEADLINES_Model2024.zip -d strategy/
! rm HEADLINES_Model2024.zip
! wget -P db/ https://github.com/lchen001/DataHolder/releases/download/v0.0.2/HEADLINES.sqlite

  12. Install and Setup FrugalGPT

    main

    To use FrugalGPT, ensure the src/ directory is in your Python path. You must also configure environment variables for the supported LLM providers to enable actual API queries. Supported providers include OpenAI, AI21, Cohere, TextSynth, and Anthropic.

    Required environment variables:

    • OPENAI_API_KEY
    • AI21_STUDIO_API_KEY
    • COHERE_STUDIO_API_KEY
    • TEXTSYNTH_API_SECRET_KEY
    • ANTHROPIC_API_KEY
    import sys
sys.path.append("src/")

import os
os.environ['OPENAI_API_KEY'] = 'YOUR_KEY'
os.environ['AI21_STUDIO_API_KEY'] = 'YOUR_KEY'
os.environ['COHERE_STUDIO_API_KEY'] = 'YOUR_KEY'
os.environ['TEXTSYNTH_API_SECRET_KEY'] = 'YOUR_KEY'
os.environ['ANTHROPIC_API_KEY'] = 'YOUR_KEY'