PromptBench

The Prompt Engineering Module in promptbench evaluates various prompting methods across multiple datasets. It is designed to compare different prompting strategies like Chain-of-Thought (CoT), Zero-shot CoT, Expert Prompting, Emotion Prompt, and Least-to-Most prompting.

Currently supported models include:

• GPT-3.5-turbo
• GPT-4-1106

PromptBench implements four levels of adversarial attacks to evaluate LLM robustness:

• Character-level: Uses TextBugger and DeepWordBug to introduce typos or errors (adding, deleting, repeating, replacing, or permuting characters).
• Word-level: Uses BertAttack and TextFooler to replace words with synonyms or contextually similar words.
• Sentence-level: Uses StressTest and CheckList to append distracting information:
  • StressTest: Appends and true is true, and false is not true, or and true is true five times.
  • CheckList: Appends 50 random sequences of alphabets and digits (length 10 each).
• Semantic-level: Simulates linguistic nuances by translating prompts into 6 languages (Chinese, French, Arabic, Spanish, Japanese, and Korean) and back to English.

DyVal (Graph-informed Dynamic Evaluation of Large Language Models) is a dynamic evaluation protocol for LLMs. It measures model performance across various reasoning tasks categorized by complexity levels.

Key reasoning tasks evaluated include:

• Arithmetic
• Linear Equation
• Boolean Logic
• Deductive Logic
• Abductive Logic
• Reachability
• Max Sum Path

If you wish to have your model's results included in the PromptBench DyVal leaderboard, you must contact the project maintainers.

PromptBench provides a leaderboard to evaluate the adversarial robustness of Large Language Models (LLMs) against various prompt attacks. The leaderboard tracks how different models perform across multiple datasets when subjected to adversarial prompts.

For detailed methodology, refer to the paper: PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts.

To quickly start evaluating models, install the promptbench package using pip. Note that the pip version may lag behind the latest updates available on GitHub.

pip install promptbench

For a user-friendly interface to interact with adversarial prompts, you can use the PromptBench Streamlit application hosted on Hugging Face Spaces instead of using the raw markdown files.

https://huggingface.co/spaces/March07/PromptBench

To add a custom dataset, follow these two steps:

1. Implement a New Dataset Class: Create a class in dataload/dataset.py that inherits from the Dataset class. Implement the __init__ method to handle your dataset loading. It is recommended to organize data samples as dictionaries to simplify the input process.
2. Register the Interface: Register your new dataset class in the DataLoader class within dataload.py to make it accessible via the framework.

To make changes to the codebase or experiment with the source, follow these steps to clone the repository and install dependencies manually:

1. Clone the repository:

   git clone git@github.com:microsoft/promptbench.git
   cd promptbench

2. Create a Conda environment (recommended):

   conda create --name promptbench python=3.9

3. Install the required Python packages:

   pip install -r requirements.txt

Note: This only installs basic Python packages. If you intend to use Prompt Attacks, you must also install textattacks.

PromptBench provides several specialized notebooks for different evaluation tasks:

• Basic Model Evaluation: Construct an evaluation pipeline for existing benchmarks using examples/basic.ipynb.
• Multi-modal Evaluation: Evaluate multi-modal models and datasets using examples/multimodal.ipynb.
• Prompt Attack Robustness: Examine how models handle adversarial attacks using examples/prompt_attack.ipynb.
• Dynamic Evaluation (DyVal): Construct DyVal datasets to mitigate test data contamination using examples/dyval.ipynb.
• Efficient Multi-prompt Evaluation (PromptEval): Use the PromptEval method for efficient evaluation using examples/efficient_multi_prompt_eval.ipynb.