DOCSEARCHbyAlgolia
Submit Docs

SkillWeaver

repository·main·Indexed 41 days ago

https://github.com/osu-nlp-group/skillweaver

A skill-centric framework for web agents to self-improve by discovering, synthesizing, and honing reusable skills as APIs through iterative exploration. It includes modules for task attempts, autonomous website exploration, and benchmark evaluations using WebArena, with support for OpenAI, Azure OpenAI, and Browser-Use integration.

Tokens
1.9K
Snippets
6
Records
9
Agent score
50%

What's inside SkillWeaver

  1. Structure of the debug_prompt template

    main

    The debug_prompt.md template is used to instruct an LLM to act as a Python debugging expert specializing in Playwright web automation. It provides the model with the task description, the original failing code, the error information, and a truncated version of the current webpage's HTML structure.

    When using this template for API calls, the following constraints are enforced via the prompt:

    • The input parameter for the API must include page (a Playwright Page object).
    • Do not include the browser object or define the browser in the code.
    • Do not use the goto() method, as the page is assumed to already be at the correct state.
    • Do not use await page.wait_for_selector('selector') as it is explicitly flagged as problematic.
    • The output must be ONLY the fixed, runnable API code with necessary imports, without explanations or markdown formatting within the code block, and without a main() function.

  2. Run parallel evaluations with Managed Containers

    main

    SkillWeaver can orchestrate multiple Docker containers to run experiments in parallel. This requires an Orchestrator Server running outside of Docker.

    1. Start the Orchestrator Server:

      python -m skillweaver.containerization.serve
# Or specify a custom port:
ORCHESTRATOR_PORT=5128 python -m skillweaver.containerization.serve

    2. Networking Note: Containers are routed to ports 8000, 8001, etc. If using a cloud environment, ensure these ports are accessible and set the IP variable in your .env file to the server's IP instead of 127.0.0.1.

    python -m skillweaver.containerization.serve

  3. Install SkillWeaver

    main

    To install SkillWeaver, it is recommended to use a Conda virtual environment with Python 3.10. You must also install the Playwright dependencies.

    conda create -n skillweaver python=3.10
conda activate skillweaver
pip install -r requirements.txt
playwright install

  4. Configure OpenAI and Azure OpenAI environment variables

    main

    SkillWeaver requires API keys for LLM access. You can use standard OpenAI or Azure-hosted OpenAI models.

    Standard OpenAI: Set OPENAI_API_KEY.

    Azure OpenAI: Set AZURE_OPENAI=1, AZURE_OPENAI_gpt-4o_ENDPOINT, and AZURE_OPENAI_gpt-4o_API_KEY.

    # OpenAI API
export OPENAI_API_KEY=<your_openai_api_key>

# Azure-hosted OpenAI models
export AZURE_OPENAI=1
export AZURE_OPENAI_gpt-4o_ENDPOINT=<endpoint>
export AZURE_OPENAI_gpt-4o_API_KEY=<endpoint API key>

  5. Run single evaluations with Existing Containers

    main

    If you have already manually set up Docker containers for WebArena, you can evaluate against them by setting CONTAINER_SETUP=manual in your .env file and providing the hostnames for the websites.

    Required .env settings:

    SHOPPING=(hostname)
SHOPPING_ADMIN=(hostname)
REDDIT=(hostname)
GITLAB=(hostname)
MAP=(hostname)
CONTAINER_SETUP=manual

  6. Explore a website to synthesize skills

    main

    Use skillweaver.explore to autonomously discover and synthesize reusable skills (APIs) for a specific website.

    Arguments:

    • website: The URL or a WebArena name (shopping, shopping_admin, map, reddit, gitlab).
    • out_dir: Directory to save results. Exploration will not start if the directory already exists.
    • --iterations [niter]: Number of iterations (default: 10).
    • --agent-lm-name [lm_name]: LLM for the agent (default: gpt-4o).
    • --api-synthesis-lm-name [lm_name]: LLM for synthesizing APIs.
    • --success-check-lm-name [lm_name]: LLM for checking success (default: gpt-4o).
    • --explore-schedule: Format test_probability:X or explore:X,test:Y.
    • --allow-recovery: Allow patching APIs that throw exceptions (default: enabled). Use --no-allow-recovery to disable.
    python -m skillweaver.explore reddit logs/explore-reddit-gpt4o --agent-lm-name gpt-4o --api-synthesis-lm-name gpt-4o --iterations 160

  7. Attempt a task with skillweaver.attempt_task

    main

    Use the attempt_task module to have an agent try to complete a specific task on a website. You can use special prefixes like __REDDIT__ to load addresses from environment variables and perform required login steps.

    Arguments:

    • start-url: The URL or special prefix (e.g., __REDDIT__, __MAP__, __SHOPPING__, __SHOPPING_ADMIN__, __GITLAB__).
    • task: A string describing the task.
    • --agent-lm-name [lm_name]: The LLM name (default: gpt-4o).
    • --max-steps: Action limit (default: 10).
    • --knowledge-base-path-prefix: Path to synthesized APIs (without _code.py).
    • --headless: Run in headless mode.
    # Example: Task on reddit using a knowledge base
python -m skillweaver.attempt_task __REDDIT__ "Post to the gaming forum to ask about the best games of the year" --knowledge-base-path-prefix skill_library/reddit/reddit_kb_post

# Example: Task without a knowledge base (baseline)
python -m skillweaver.attempt_task __REDDIT__ "Post to the gaming forum to ask about the best games of the year"

  8. Run benchmark evaluations

    main

    Use skillweaver.evaluate_benchmark to run evaluations on a specific website using the WebArena benchmark.

    Arguments:

    • website: Name of the website (shopping, shopping_admin, reddit, gitlab, or map).
    • out_dir: Directory for results (must not already exist).
    • --time-limit [time_limit]: Action limit per task (default: 10).
    • --knowledge-base-path-prefix [prefix]: Prefix of the knowledge base to use (e.g., /path/to/dir/kb_post).
    • --lm-name [lm_name]: LLM for the agent (default: gpt-4o-2024-08-06).
    • --pool-size [pool_size]: Number of subprocesses/containers (default: 8).
    • --use-debugger-eval: Use modified WebArena debugger (default: True).
    • --allow-recovery: Allow agent to patch APIs (default: True).
    • --reduced-set: Use one test case per unique intent template (default: True).
    • --allow-unverified-apis: Allow use of unexecuted APIs (default: False).
    • --selected-tasks [task1,task2,...] OR reduced_set: Specific task indices or reduced_set.

  9. Attempt a task with the Browser-Use version

    main

    The attempt_task_browser_use command is an experimental version that uses the Browser-Use agent. It converts the SkillWeaver knowledge base into a Browser-Use Controller object to extend the agent's action space.

    Arguments:

    • start-url: The URL or special prefix.
    • task: A string describing the task.
    • --agent-lm-name [lm_name]: The LLM name (default: gpt-4o).
    • --knowledge-base-path-prefix: Path to synthesized APIs.
    • --headless: Run in headless mode.
    python -m skillweaver.attempt_task_browser_use __REDDIT__ "Post to the gaming forum to ask about the best games of the year" --knowledge-base-path-prefix skill_library/reddit/reddit_kb_post