PromptSource Documentation

To install the promptsource module in editable mode for creating new prompts:

1. Download the repository.
2. Navigate to the root directory.
3. Run pip install -e ..

To use PromptSource for applying existing prompts to datasets, install it via pip:

pip install promptsource

If you intend to create new prompts using the web-based GUI, you must install the repository locally in editable mode. Note that for stability, a Python 3.7 environment is currently recommended, though you can remove this constraint in setup.py if you only intend to use the prompts and not the creation interface.

To launch the local web-based GUI for sourcing, viewing, or aggregating prompt metrics, run the following command from the root directory of the repository:

streamlit run promptsource/app.py

You can use DatasetTemplates to load prompts for a specific dataset or subset and apply them to examples from the Hugging Face datasets library. The apply() method returns a tuple containing the formatted input and the target output.

from datasets import load_dataset
from promptsource.templates import DatasetTemplates

# Load a dataset example
dataset = load_dataset("ag_news", split="train")
example = dataset[1]

# Load prompts for the dataset
# For subsets, use the syntax: DatasetTemplates("dataset_name/subset_name")
ag_news_prompts = DatasetTemplates('ag_news')

# Select a prompt by its name
prompt = ag_news_prompts["classify_question_first"]

# Apply the prompt to the example
# result[0] is the INPUT, result[1] is the TARGET
result = prompt.apply(example)
print("INPUT: ", result[0])
print("TARGET: ", result[1])

The PromptSource application can be run using Streamlit. You can run it in standard mode or in read-only mode using the --read-only or -r flag.

To run in read-only mode (which limits available modes to 'Helicopter view' and 'Prompted dataset viewer'), use:

streamlit run promptsource/app.py -- -r

Or:

streamlit run promptsource/app.py -- --read-only

OS X Darwin Errors

If you encounter warnings or errors related to Darwin on macOS, try downgrading PyArrow to version 3.0.0.

Connection Refused

If you see ConnectionRefusedError: [Errno 61] Connection refused, try restarting the application.

The TemplateCollection class provides access to all prompts available in PromptSource by wrapping multiple DatasetTemplates instances.

Methods:

• get_dataset(dataset_name, subset_name): Returns the DatasetTemplates object for the specified dataset_name and optional subset_name.
• get_templates_count(): Returns the total count of templates across all datasets (note: subsets are included in the dataset count).

Every Template instance has a metadata attribute (an instance of the Metadata class) containing the following attributes:

• original_task (Bool): Indicates if the prompt asks the model to perform the original task designed for the dataset.
• choices_in_prompt (Bool): Indicates if answer choices are included in the template (for classification tasks).
• metrics (List[str]): A list of strings denoting metrics used for evaluation.

The Template class wraps a prompt and its metadata. Use the apply method to generate a prompted version of a dataset example.

Methods:

• apply(example, truncate=True, highlight_variables=False): Creates a prompted example.
  • example (Dict): The dataset example.
  • truncate (Bool): If True, truncates fields to TEXT_VAR_LENGTH.
  • highlight_variables (Bool): If True, highlights added variables.
• get_id(): Returns the prompt's UUID.
• get_name(): Returns the prompt's name.
• get_reference(): Returns bibliographic or additional info.
• get_answer_choices_list(example): Returns a list of answer choices if applicable.

The DatasetTemplates class manages all Template instances for a specific dataset/subset. To access prompts, instantiate it using a template_key formatted as "{dataset_name}/{subset_name}" (or just "{dataset_name}" if no subset is specified).

Key attributes/methods:

• len(prompts): Returns the number of prompts for the dataset.
• prompts.all_template_names: Returns a sorted list of all template names for the dataset.

Use TemplateCollection to retrieve all available prompts across all datasets in PromptSource. The datasets_templates attribute returns a dictionary where keys are (dataset_name, subset_name) tuples and values are DatasetTemplates instances.

from promptsource.templates import TemplateCollection

collection = TemplateCollection()
# Returns a dict: {(dataset_name, subset_name): DatasetTemplates}
print(collection.datasets_templates)