repository·main·Indexed 25 days ago
https://github.com/hegelai/prompttoolsAn open-source, self-hostable toolkit for testing, experimenting with, and evaluating LLMs, vector databases, and prompts. It provides abstractions like Experiments and Harnesses to evaluate providers including OpenAI, Anthropic, Google, and HuggingFace, as well as vector databases like ChromaDB, Weaviate, and Pinecone. Features include a local Streamlit playground, RAG evaluation, and support for integrating evaluations into CI/CD workflows.
PromptTools provides tools for testing and experimenting with prompts, allowing developers to evaluate prompts using code and notebooks. There are two primary workflows:
The prompttools package is organized into several specialized subpackages for managing prompt engineering workflows:
prompttools.experiment: Tools for running and managing prompt experiments.prompttools.harness: Frameworks for testing and evaluating prompts.prompttools.mock: Utilities for mocking LLM responses and behaviors.prompttools.prompttest: Specific tools for testing prompt quality and consistency.prompttools.requests: Utilities for handling LLM requests.prompttools.utils: General purpose utility functions.The prompttools library uses two primary abstractions for managing LLM evaluations:
prompttools.utils module provides built-in functions for evaluating the outputs of LLM experiments. These utilities can be used directly in your evaluation scripts or integrated into a CI/CD system using prompttest.The prompttools library uses two primary abstractions for testing and evaluating LLM outputs:
Use an Experiment when you need fine-grained control over the specific parameters being varied across multiple API calls.
Use prompttools to evaluate vector database performance and Retrieval Augmented Generation (RAG) workflows. Supported experiments include:
To execute your prompttests (similar to how you would run standard unittest suites), you must call the main() method from the prompttools.prompttest module within your script's entry point.
if __name__ == "__main__":
prompttest.main()To install from source, clone the GitHub repository to your local machine and run the installation command from the root of the repository.
pip install .To run experiments in a notebook, use PromptTemplateExperimentationHarness to manage prompt templates and user inputs. You can define a custom evaluation function (eval_fn) to score results, run the harness, and then use .evaluate() to apply your metric and .visualize() to display the results as a table in the notebook.
For built-in evaluation functions, you can use prompttools.utils.similarity for semantic similarity comparisons.
from prompttools.harness import PromptTemplateExperimentationHarness
from typing import Dict
def eval_fn(prompt: str, results: Dict, metadata: Dict) -> float:
# Your logic here, or use a built-in one such as `prompttools.utils.similarity`.
pass
prompt_templates = [
"Answer the following question: {{input}}",
"Respond to the following query: {{input}}"
]
user_inputs = [
{"input": "Who was the first president?"},
{"input": "Who was the first president of India?"}
]
harness = PromptTemplateExperimentationHarness("text-davinci-003",
prompt_templates,
user_inputs)
harness.run()
harness.evaluate("metric_name", eval_fn)
harness.visualize() # The results will be displayed as a table in your notebookUse the @prompttest.prompttest decorator to transform a completion function into a unit test (referred to as a prompttest). This allows you to execute and evaluate experiments as part of your testing suite or CI/CD workflow.
The decorator requires metric_name, an eval_fn, and a list of prompts.
Your evaluation function (eval_fn) must accept one of the following parameter signatures:
input_pair: Tuple[str, Dict[str, str]], results: Dict, metadata: Dictprompt: str, results: Dict, metadata: Dictmessages: List[Dict[str,str]], results: Dict, metadata: Dictimport prompttools.prompttest as prompttest
import os
@prompttest.prompttest(
metric_name="is_valid_json",
eval_fn=validate_json.evaluate,
prompts=[create_json_prompt()],
)
def json_completion_fn(prompt: str):
response = None
if os.getenv("DEBUG", default=False):
response = mock_openai_completion_fn(**{"prompt": prompt})
else:
response = openai.completions.create(prompt)
return response.choices[0].textIn prompttools, unit tests for prompts are called prompttests. You can transform a completion function into an efficient unit test using the @prompttest.prompttest decorator. This decorator executes and evaluates experiments, allowing you to test prompts over time against specific metrics.
To use it, provide the following arguments to the decorator:
metric_name: A string name for the metric being tested.eval_fn: The evaluation function used to validate the output.prompts: A list of prompt templates or strings to be tested.Example usage:
import prompttools.prompttest as prompttest
@prompttest.prompttest(
metric_name="is_valid_json",
eval_fn=validate_json.evaluate,
prompts=[create_json_prompt()],
)
def json_completion_fn(prompt: str):
response = None
if os.getenv("DEBUG", default=False):
response = mock_openai_completion_fn(**{"prompt": prompt})
else:
response = openai.completions.create(prompt)
return response.choices[0].textimport prompttools.prompttest as prompttest
@prompttest.prompttest(
metric_name="is_valid_json",
eval_fn=validate_json.evaluate,
prompts=[create_json_prompt()],
)
def json_completion_fn(prompt: str):
response = None
if os.getenv("DEBUG", default=False):
response = mock_openai_completion_fn(**{"prompt": prompt})
else:
response = openai.completions.create(prompt)
return response.choices[0].textThere are two ways to initialize an Experiment class:
list objects and pass them directly into the __init__ method of the specific experiment class (e.g., OpenAIChatExperiment).initialize method: Define two dictionaries—one containing the parameters you want to test (varying) and one containing the parameters you want to keep constant (frozen). Pass these dictionaries to the initialize class method.