FastChat

When running test_classification.py, the script performs a comparative evaluation of different embedding models. It trains classifiers based on the following models and reports their respective accuracies:

• vicuna-7b (Local model)
• text-similarity-ada-001 (OpenAI model)
• text-embedding-ada-002 (OpenAI model)

The Nginx gateway acts as a reverse proxy and security layer for Gradio servers. Its primary functions include:

• Security: Protects Gradio servers by acting as a firewall and reducing the attack surface by requiring only a single public port to be exposed.
• Management: Facilitates dynamic mounting/unmounting of Gradio servers and provides load balancing.
• Traffic Control: Offers additional security features like total connection limits.

MT-bench supports three grading modes via the --mode flag in gen_judgment.py:

1. Single-answer grading (Default): GPT-4 grades each answer directly on a scale of 10.
2. pairwise-baseline: Compares models against a baseline model (defaults to gpt-3.5-turbo). Results are saved to data/mt_bench/model_judgment/gpt-4_pair.jsonl.
3. pairwise-all: Performs pairwise comparisons between all possible pairs of models in the --model-list.

# Example: Pairwise comparison against baseline
python gen_judgment.py --mode pairwise-baseline --model-list vicuna-13b-v1.3 alpaca-13b --parallel 2
python show_result.py --mode pairwise-baseline

# Example: Pairwise comparison between all pairs
python gen_judgment.py --mode pairwise-all --model-list vicuna-13b-v1.3 alpaca-13b --parallel 2
python show_result.py --mode pairwise-all

FastChat provides pre-generated model answers and judgments for certain models. You can download them using download_mt_bench_pregenerated.py and view them locally using the qa_browser.py tool.

# Download pre-generated data
python3 download_mt_bench_pregenerated.py

# View data locally
python3 qa_browser.py --share

Use upload_hf_dataset.py to upload the processed dataset to the Hugging Face Hub.

Input: The final processed JSON file (e.g., $BASE.s1.s2.json).

export BASE=clean_conv_20230809_100k_pii
python3 upload_hf_dataset.py --in $BASE.s1.s2.json

To enable efficient 4-bit LLM inference in FastChat, you must install the llm-awq package and its CUDA kernels. Follow these steps to set up a dedicated environment:

1. Create and activate a new Conda environment.
2. Install FastChat in editable mode.
3. Clone the llm-awq repository and install it.
4. Compile and install the AWQ CUDA kernels.

Note: Ensure you have the necessary build tools and CUDA environment configured as per the llm-awq requirements.

conda create -n fastchat-awq python=3.10 -y
conda activate fastchat-awq
# cd /path/to/FastChat
pip install --upgrade pip
pip install -e .

git clone https://github.com/mit-han-lab/llm-awq repositories/llm-awq
cd repositories/llm-awq
pip install -e .

cd awq/kernels
python setup.py install

To perform data cleaning tasks such as HTML conversion, language filtering, and conversation splitting, you must install the following Python dependencies:

For HTML to Markdown conversion:

• bs4
• markdownify

For language detection and filtering:

• polyglot
• pyicu
• pycld2

pip3 install bs4 markdownify
pip3 install polyglot pyicu pycld2

You can run a local instance of the Chatbot Arena (side-by-side battle UI) by configuring API-based models (like OpenAI, Anthropic, or Gemini) in a JSON file and launching the Gradio multi-server.

1. Create an api_endpoint.json file defining your models. For OpenAI, use "api_type": "openai". For Anthropic, use "api_type": "anthropic_message". For Gemini, use "api_type": "gemini".
2. Launch the server using the gradio_web_server_multi module pointing to your config file.

{
    "gpt-4o-2024-05-13": {
        "model_name": "gpt-4o-2024-05-13",
        "api_base": "https://api.openai.com/v1",
        "api_type": "openai",
        "api_key": "[Insert API Key]",
        "anony_only": false
    }
}

python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file api_endpoint.json

The playground/test_embedding directory contains scripts for evaluating various machine learning tasks using text embeddings. You can use these scripts to:

• Evaluate text similarity: See test_sentence_similarity.py.
• Build a classifier: See test_classification.py.
• Perform semantic search: See test_semantic_search.py.

Prerequisites

1. Dataset: Download the Amazon Fine Food Reviews dataset from Kaggle.
2. OpenAI API Key: Required for comparing results against OpenAI models.

Execution

To run the classification test, navigate to the directory and execute the script using Python 3:

cd playground/test_embedding
python3 test_classification.py