DOCSEARCHbyAlgolia
Submit Docs

EDSL (Expected Parrot Domain-Specific Language)

repository·main·Indexed 19 days ago

https://github.com/expectedparrot/edsl

A domain-specific language for computational social science and market research. EDSL enables developers to design, run, and replicate surveys and experiments using multiple AI agents and LLMs simultaneously. It includes features for creating declarative question types and a service framework to expose EDSL workflows as web APIs using decorators like @edsl_service, @input_param, and @output_schema.

Tokens
69.2K
Snippets
245
Records
296
Agent score
67%

What's inside edsl

  1. Debug Answer validation errors

    main

    Exceptions subclassed from QuestionAnswerValidationError indicate issues with how a question was constructed or answered (e.g., receiving a string when a list was expected, or receiving None for an unanswered question).

    To debug these, check the Settings class for the Questions model. The following default settings govern answer and question constraints:

    • MAX_ANSWER_LENGTH: 2000
    • MAX_EXPRESSION_CONSTRAINT_LENGTH: 1000
    • MAX_NUM_OPTIONS: 200
    • MIN_NUM_OPTIONS: 2
    • MAX_OPTION_LENGTH: 10000
    • MAX_QUESTION_LENGTH: 100000

  2. Build and configure a Survey

    main

    A Survey is a collection of questions administered together. You can control how much context (memory) the model has regarding previous questions in the survey:

    • Targeted Memory: Provide the answer to a specific previous question to the current question using survey.add_targeted_memory(q_target, q_source).
    • Full Memory Mode: Provide the full history of all prior questions and answers to every subsequent question using survey.set_full_memory_mode(). Note that this is token-intensive.

    To create a survey, pass a list of question objects to the Survey constructor.

    from edsl import Survey

questions = [q1, q2, q3, q4, q5]
survey = Survey(questions)

# To add memory:
# survey = survey.add_targeted_memory(q2, q1)
# survey = survey.set_full_memory_mode()

  3. Use the `edsl humanize` command group

    main

    The humanize command group in the EDSL CLI is used for managing human-in-the-loop survey workflows, including schema validation, previewing, creating surveys, and managing responses.

    Available subcommands include:

    • edsl humanize schema (for validation and previewing)
    • edsl humanize list (for listing items)
    • edsl humanize create (for creating surveys/jobs)
    • edsl humanize status (for checking status)
    • edsl humanize responses (for retrieving results)
    • edsl humanize qr (for generating QR codes)

  4. Choose between local and remote survey execution

    main

    EDSL supports two execution modes:

    1. Remote Inference: Surveys run on the Expected Parrot server. This enables remote caching and automatic storage of survey results. You can toggle this on or off in your Account Settings.
    2. Local Execution: Surveys run on your own machine. You must provide your own API keys from service providers (e.g., Anthropic, OpenAI, Google) to power the language models.

  5. How EDSL calculates estimated costs

    main

    EDSL uses a specific heuristic to estimate costs for each question in a survey:

    1. Input Tokens:
      • Counts characters in user_prompt and system_prompt (including Agent/Scenario data).
      • If a prompt has Jinja braces (piping an answer from a previous question), a multiplier of 2 is applied to the character count.
      • Converts characters to tokens using a factor of 4 characters per token (rounded down).
    2. Output Tokens:
      • Calculated as 0.75 * input tokens (rounded up).
    3. Pricing:
      • Applies rates for the specific model and inference service.
      • If the model is unknown, it uses a default provider price.
      • If both are unknown, it falls back to USD 1.00 per 1M input tokens and USD 1.00 per 1M output tokens.
    4. Credit Conversion:
      • Total cost in credits = total cost in USD * 100 (rounded up to the nearest 1/100th credit).

  6. Access answer commentary for non-free text questions

    main

    When using question types other than QuestionFreeText (such as QuestionMultipleChoice), EDSL automatically includes a comment field. This allows the AI agent to provide unstructured reasoning or commentary alongside its structured response.

    If a question is named pid3, the commentary field will be accessible via comment.pid3_comment.

  7. Understand memory scaling expectations

    main
    Memory performance tests in EDSL are designed to confirm that memory usage scales efficiently. A successful test demonstrates that memory usage per interview decreases as the number of interviews (job size) increases. Typical expected improvements range from 25% to 80% memory reduction per interview when comparing small jobs to large jobs. Because memory measurements vary by environment, tests use relative comparisons rather than fixed thresholds.

  8. Working with survey results

    main

    Once surveys are complete, EDSL provides tools to manage and analyze the output:

    • Dataset: Use the Dataset class to work with survey results as tabular data.
    • Estimating & Tracking Costs: Tools to monitor the financial cost of running surveys.
    • Exceptions & Debugging: Methods to identify and handle errors during execution.
    • Token usage: Monitoring tools for tracking language model token limits and usage.

  9. Use the Request/Response Pattern for Distributed Firecrawl Processing

    main

    For large-scale scraping or searching tasks, avoid running everything in a single process. Instead, use the request/response pattern to create serializable request objects. These objects can be sent to distributed workers or APIs and then executed later using execute_request.

    This pattern is essential for managing memory and scaling operations across multiple machines.

    from edsl.scenarios.firecrawl_scenario import (
    create_scrape_request,
    create_search_request,
    execute_request
)

# Create serializable requests (can be sent to workers, APIs, etc.)
requests = [
    create_scrape_request("https://example1.com"),
    create_scrape_request("https://example2.com"),
    create_search_request("machine learning tutorials")
]

# Execute requests (potentially on different machines/processes)
results = []
for request in requests:
    result = execute_request(request)
    results.append(result)

  10. Add question memory to a survey

    main

    Instead of using placeholders in the question text, you can add "memory" of prior questions and answers to the context of a follow-on question. This provides the model with the full context (both the question text and the answer) without requiring explicit placeholders in the question_text.

    You can use the .add_targeted_memory(target_question, source_question) method on a Survey object to achieve this.

    from edsl import QuestionNumerical, QuestionYesNo, Survey

q1 = QuestionNumerical(
    question_name = "random_number",
    question_text = "Pick a random number between 1 and 1,000."
)

q2 = QuestionYesNo(
    question_name = "prime",
    question_text = "Is the number you picked a prime number?"
)

# Adds memory of q1 to the context of q2
survey = Survey([q1, q2]).add_targeted_memory(q2, q1)

results = survey.run()

  11. Accessing Language Models and Managing API Keys

    main

    EDSL supports many language model providers (e.g., Anthropic, Azure, Bedrock, Google, OpenAI, etc.). To run surveys, you must provide API keys. There are two primary ways to handle this:

    1. Expected Parrot API Key: Use your Expected Parrot API key to access all available models. This key is used by default if no private key is provided for a selected model. You will need credits on your Expected Parrot account to cover token costs.
    2. Provider API Keys: Provide your own API keys from the specific service providers (e.g., your own OpenAI API key).

    You can manage your keys via the Keys page on the Expected Parrot platform, where you can add keys, share them with users, and set their priority.

  12. How to use scenarios with humanized surveys

    main

    To present different content to different respondents (e.g., different policy descriptions), use ScenarioList.

    Important Rules:

    • You must call .humanize() on a Jobs object (created via .by(scenarios)) rather than directly on the Survey.
    • You must specify a scenario_list_method.
    • You must attach scenarios if you specify a method.

    Scenario List Methods:

    • "randomize": Each respondent gets a random scenario (with replacement).
    • "ordered": Scenarios are assigned sequentially to respondents.
    • "loop": The survey is expanded; every question is repeated for every scenario.
    • "single_scenario": Exactly one scenario is used for all respondents (the list must have length 1).
    from edsl import Survey, QuestionFreeText, QuestionLinearScale, Scenario, ScenarioList

scenarios = ScenarioList([
    Scenario({"policy": "Four-day workweek with no change in pay"}),
    Scenario({"policy": "Flexible start times between 7am and 11am"}),
    Scenario({"policy": "Monthly remote-work stipend of $100"}),
])

q1 = QuestionLinearScale(
    question_name="support",
    question_text="How much do you support this policy: '{{ scenario.policy }}'?",
    question_options=[1, 2, 3, 4, 5],
    option_labels={1: "Strongly oppose", 5: "Strongly support"},
)
q2 = QuestionFreeText(
    question_name="reasoning",
    question_text="What is the main reason for your rating?",
)

survey = Survey([q1, q2])

# Call .humanize() on the Jobs object returned by .by()
info = survey.by(scenarios).humanize(
    human_survey_name="Workplace Policy Reactions",
    scenario_list_method="randomize",
)