prompt-ops

PDO (Prompt Duel Optimizer) is a dueling bandit optimization strategy designed to evolve and select high-performing prompts. It operates through the following lifecycle:

1. Prompt Pool Management: Maintains a pool of different prompt variations.
2. Dueling: Runs "duels" between prompts to compare their performance on a specific task.
3. Thompson Sampling: Uses Thompson sampling to intelligently select which prompts to test in subsequent rounds.
4. Evolution: Generates new prompt variations from the top-performing prompts in the pool.
5. Ranking: Uses the Copeland ranking method to determine the final best prompts.

This strategy is useful for tasks requiring complex reasoning, such as the "Web of Lies" logical reasoning task, where the goal is to find the most effective instruction set to improve model accuracy.

A standard project created via prompt-ops create follows this directory structure:

• .env: Environment variables (e.g., API keys).
• README.md: Project documentation.
• config.yaml: Configuration for the optimization task.
• data/dataset.json: The dataset used for evaluation.
• prompts/prompt.txt: The system prompt to be optimized.
• results/: Directory where optimized prompts and performance metrics are saved.

The dataset must be in JSON format. Each entry requires a fields object containing the input (e.g., a customer message) and an answer field containing the expected ground truth JSON output.

{
  "fields": {
    "input": "Subject: Urgent HVAC Repair Needed\n\nHi ProCare Support Team..."
  },
  "answer": "{\"categories\": {\"routine_maintenance_requests\": false, ...}, \"sentiment\": \"positive\", \"urgency\": \"high\"}"
}

prompt-ops supports multiple inference providers and endpoints. You can configure your infrastructure to use:

• OpenRouter: Cloud-based API.
• vLLM: Local deployment.
• NVIDIA NIMs: Optimized containers.

Optimization requires a JSON file containing query-response pairs (at least 50 examples recommended).

Standard Format

If your data follows this structure, prompt-ops will automatically use the StandardJSONAdapter:

[
    {
        "question": "Your input query here",
        "answer": "Expected response here"
    },
    {
        "question": "Another input query",
        "answer": "Another expected response"
    }
]

Custom Formats

If your data does not match this structure, you must create a custom dataset adapter by extending the DatasetAdapter class.

You can switch between different LLM backends by modifying the model section in your YAML configuration files. prompt-ops uses LiteLLM as a unified API client, which automatically detects the provider based on the model name prefix (e.g., openrouter/, groq/, together_ai/) and maps it to the corresponding environment variable.

Basic model configuration structure:

model:
  name: "provider/model-name"
  temperature: 0.0
  max_tokens: 4096

model:
  name: "openrouter/meta-llama/llama-3.1-8b-instruct"
  temperature: 0.0
  max_tokens: 40960

Dataset adapters determine how prompt-ops reads your input data. Choose an adapter based on your data structure:

Metrics are used to compare model predictions against ground truth. Select a metric based on your expected output format:

The ms-marco-pdo directory follows this structure:

• config.yaml: Contains the PDO optimization configuration.
• prompts/prompt.txt: The initial prompt template used to seed the pool.
• dataset/ms_marco_description.json: The MS MARCO-style QA samples used for evaluation.
• results/: Directory where optimization results (rankings, best prompts, etc.) are generated.
• MSMARCO_PDO_eval.ipynb: An optional notebook for evaluating and analyzing the optimization results.

ms-marco-pdo/
├── config.yaml                 # PDO optimization configuration
├── prompts/
│   └── prompt.txt             # Initial prompt template
├── dataset/
│   └── ms_marco_description.json  # MS MARCO-style QA samples
├── results/                   # Optimization results (generated)
└── MSMARCO_PDO_eval.ipynb     # Optional: evaluation/analysis notebook

prompt-ops uses LiteLLM as a unified API client. It automatically detects the provider from the model name string. For example:

• openrouter/model-name requires OPENROUTER_API_KEY.
• groq/model-name requires GROQ_API_KEY.

Supported providers include OpenRouter, vLLM (local), and NVIDIA NIMs.

Select a metric based on your data format and evaluation requirements: