DOCSEARCHbyAlgolia
Submit Docs

PromptWizard

repository·main·Indexed 26 days ago

https://github.com/microsoft/promptwizard

A task-aware prompt optimization framework that uses a self-evolving mechanism to iteratively generate, critique, and refine instructions and in-context learning examples. It features instruction mutation, reasoning enrichment via Chain-of-Thought, and sequential optimization to improve LLM performance. The framework supports OpenAI and Azure OpenAI APIs and provides the GluePromptOpt class for managing the optimization process and DatasetSpecificProcessing for custom dataset integration.

Tokens
4.3K
Snippets
8
Records
20
Agent score
85%

What's inside PromptWizard

  1. Understand how PromptWizard works

    main

    PromptWizard (PW) is a task-aware prompt optimization framework that uses the following mechanisms to improve LLM performance:

    • Instruction Mutation & Refinement: Generates variations of task descriptions and instructions by prompting LLMs to mutate them. It uses a critique component to provide feedback for iterative refinement.
    • In-Context Example Optimization: Selects a diverse set of positive and negative examples from training data. Negative examples are used to inform further prompt refinements.
    • Sequential Optimization: Optimizes examples and instructions sequentially, using critiques to generate synthetic examples that address current prompt weaknesses.
    • Reasoning Enrichment: Generates detailed reasoning chains via Chain-of-Thought (CoT).
    • Alignment: Integrates task intent and expert personas to align prompts with human reasoning, improving both performance and interpretability.

  2. Configure LLM API environment variables

    main

    Before using PromptWizard, ensure your LLM API credentials (such as OpenAI or Azure OpenAI) are set in a .env file in your project root. Use load_dotenv(override=True) to load these variables into your environment.

    from dotenv import load_dotenv
load_dotenv(override = True)

  4. Best practices for PromptWizard configuration

    main

    When tuning promptopt_config.yaml, consider these findings from experimental results:

    • Iteration Counts: For mutate_refine_iterations and refine_task_eg_iterations, values of 3 or 5 typically yield the best performance.
    • Dataset Scaling: The seen_set_size can be increased to 50 to improve results.
    • Few-Shot Tuning: The few_shot_count should be adjusted based on your specific use case.
    • Example Strategy: Experiment with both synthetic in-context examples and in-context examples from the training set to determine which approach works best for your task.
    • Human Supervision: While generated prompts are detailed, manual user supervision is recommended to fine-tune the final output for specific tasks.

  5. Configure API environment variables

    main

    PromptWizard requires API access via either OpenAI or Azure OpenAI. Configure these settings in a .env file.

    If using OpenAI: Set USE_OPENAI_API_KEY="True" and provide OPENAI_API_KEY and OPENAI_MODEL_NAME.

    If using Azure OpenAI: Set USE_OPENAI_API_KEY="False" and provide AZURE_OPENAI_ENDPOINT, OPENAI_API_VERSION, and AZURE_OPENAI_CHAT_DEPLOYMENT_NAME.

  6. Configure optimization hyperparameters in promptopt_config.yaml

    main

    Use promptopt_config.yaml to define the optimization behavior. Key parameters include:

    • task_description: Description of the task for the prompt.
    • base_instruction: The starting instruction (e.g., "Lets think step by step.").
    • answer_format: Instructions for the LLM to format its answer (e.g., using <ANS_START> and <ANS_END> tags).
    • seen_set_size: Number of training samples used for optimization (recommended 20-50).
    • few_shot_count: Number of in-context examples to include.
    • generate_reasoning: Boolean to enable/disable step-by-step reasoning in examples.
    • generate_expert_identity: Boolean to help make the prompt task-relevant.
    • generate_intent_keywords: Boolean to help make the prompt task-relevant.

    Global Hyperparameters:

    • run_without_train_examples: Use when no training samples are available.
    • generate_synthetic_examples: Use to generate synthetic data when no training samples exist.
    • use_examples: Use to optimize prompts using existing training data.

  7. Configure PromptWizard hyperparameters

    main

    Hyperparameters for the prompt optimization process are defined in promptopt_config.yaml. Key configuration keys include:

    KeyDescription
    mutate_refine_iterationsNumber of iterations for mutating task descriptions followed by instruction refinement.
    mutation_roundsNumber of mutation rounds performed when generating different styles.
    refine_task_eg_iterationsNumber of iterations for refining task descriptions and in-context examples.
    style_variationNumber of thinking style variations used in prompt mutation.
    questions_batch_sizeNumber of questions sent to the LLM in a single batch during the training step.
    min_correct_countMinimum number of question batches correctly answered for a prompt to be considered 'good'.
    max_eval_batchesMaximum number of mini-batches used to evaluate a prompt.
    top_nNumber of top best prompts considered from the scoring stage for the next stage.
    seen_set_sizeNumber of samples from the trainset used for training.
    few_shot_countNumber of in-context examples required in the final prompt.

  8. Optimize prompts using synthetic or existing training data

    main

    To optimize prompts using a dataset (either synthetic or real), provide the dataset_jsonl path and your custom data_processor to the GluePromptOpt instance.

    Configure the following in promptopt_config.yaml for best results:

    • few_shot_count: Number of examples to include in the prompt.
    • generate_reasoning: Boolean to enable reasoning generation.
    • mutate_refine_iterations: Number of refinement rounds.
    • seen_set_size: Size of the set used for optimization.

    Call get_best_prompt with use_examples=True and run_without_train_examples=False.

    # Initialize with dataset and processor
gp = GluePromptOpt(
    promptopt_config_path,
    setup_config_path,
    dataset_jsonl="train_synthetic.jsonl",
    data_processor=my_custom_processor
)

# Run optimization with examples
best_prompt, expert_profile = gp.get_best_prompt(
    use_examples=True,
    run_without_train_examples=False,
    generate_synthetic_examples=False
)

  10. Optimize prompts without training data or in-context examples

    main

    If you have no training data and do not want in-context examples in the final prompt, configure your promptopt_config.yaml with a task_description, base_instruction, and mutation_rounds. Then, call get_best_prompt with the following arguments:

    • use_examples=False
    • run_without_train_examples=True
    • generate_synthetic_examples=False
    # Configuration example
config_dict = {
    "task_description": "You are a mathematics expert. You will be given a mathematics problem which you need to solve",
    "base_instruction": "Lets think step by step.",
    "mutation_rounds": 5
}
# update_yaml_file(file_path, config_dict) # Helper to update config

# Optimization call
best_prompt, expert_profile = gp.get_best_prompt(
    use_examples=False,
    run_without_train_examples=True,
    generate_synthetic_examples=False
)

  11. Create a custom dataset for PromptWizard

    main

    PromptWizard expects datasets in .jsonl format. Both training and testing sets must follow this structure. Each line in the .jsonl file must be a JSON object containing:

    1. question: The complete question/input to be asked to the LLM.
    2. answer: The ground truth answer (can be verbose or concise).

  12. Evaluate an optimized prompt

    main

    After generating an optimal prompt and expert profile, you can evaluate its performance on a test dataset using the evaluate method. You must first assign the optimized results back to the GluePromptOpt instance.

    1. Set gp.EXPERT_PROFILE = expert_profile.
    2. Set gp.BEST_PROMPT = best_prompt.
    3. Call gp.evaluate(test_file_name) where test_file_name is the path to your test .jsonl file.
    # Assign optimized results
gp.EXPERT_PROFILE = expert_profile
gp.BEST_PROMPT = best_prompt

# Evaluate performance
accuracy = gp.evaluate(test_file_name)
print(f"Final Accuracy: {accuracy}")