LaVague Documentation

repository·main·Indexed 27 days ago

An open-source Large Action Model (LAM) framework for building AI Web Agents that automate web-based processes by translating high-level objectives into executable browser actions. The framework includes core components like WorldModel and ActionEngine, a Chrome Extension, an AgentServer for WebSocket communication, and specialized tools such as lavague-qa for generating pytest files from Gherkin descriptions and a test runner for automated website testing.

Tokens: 34.9K
Snippets: 118
Records: 159
Agent score: 88%

What's inside LaVague

Understand LaVague Agent Architecture and Workflow
main
LaVague operates as a loop consisting of four primary components that transform a high-level user goal into automated web actions:
1. Objective: The global goal provided by the user (e.g., "Log into my account and change my username to The WaveHunter.").
2. World Model: Analyzes the current webpage state (via screenshots and HTML) and the user's objective to generate the next specific Instruction.
3. Action Engine: Receives a text instruction and generates/executes the corresponding automation code.
4. Driver: A webdriver (like Playwright or Selenium) that executes the code and provides perception (screenshots and HTML source) back to the World Model.
The Workflow Loop:
- The World Model generates an instruction based on the objective and current state.
- The Action Engine converts that instruction into code and executes it via the Driver.
- The Driver captures the new state (updated screenshot/HTML).
- The World Model processes the new state to generate the next instruction.
- This repeats until the objective is met.
https://github.com/lavague-ai/lavague/blob/main/docs/docs/learn/architecture.md

Build and run a LaVague Web Agent

main

To create a Web Agent, you need to initialize a SeleniumDriver, a WorldModel, and an ActionEngine, then pass them to a WebAgent.

Note: By default, LaVague uses OpenAI's configuration. You must set the OPENAI_API_KEY environment variable in your local environment for the agent to function.

from lavague.core import  WorldModel, ActionEngine
from lavague.core.agents import WebAgent
from lavague.drivers.selenium import SeleniumDriver

selenium_driver = SeleniumDriver(headless=False)
world_model = WorldModel()
action_engine = ActionEngine(selenium_driver)
agent = WebAgent(world_model, action_engine)
agent.get("https://huggingface.co/docs")
agent.run("Go on the quicktour of PEFT")

# Launch Gradio Agent Demo
agent.demo("Go on the quicktour of PEFT")

Install LaVague
main
Install the core LaVague package using pip. Ensure you are using Python 3.10 or greater.
Note: By default, LaVague uses OpenAI models for embeddings, LLM, and Vision. You must set the OPENAI_API_KEY environment variable in your local environment for the default configuration to work.
```
pip install lavague
```
https://github.com/lavague-ai/lavague/blob/main/docs/docs/get-started/quick-tour.md
Initialize ActionEngine and WorldModel using OpenaiContext
main
After defining an OpenaiContext, you can instantiate the core LaVague components (ActionEngine and WorldModel) using the .from_context() method. This allows the components to inherit the configured LLM and embedding models.
https://github.com/lavague-ai/lavague/blob/main/docs/docs/integrations/openai.md
Launch the LaVague Gradio Agent Demo
main
You can launch an interactive chat interface to test LaVague agents in your browser using the agent.demo() method. This allows you to interact with the agent, view its progress, and test different objectives on various URLs through a Gradio-based UI.
To use this, initialize your WebAgent with a driver, an ActionEngine, and a WorldModel, then call .demo() on the agent instance.
```
driver = SeleniumDriver(headless=True)
action_engine = ActionEngine(driver)
world_model = WorldModel()

agent = WebAgent(world_model, action_engine)

# Set the target URL
agent.get("https://huggingface.co/docs")

# Launch the demo with a specific objective
agent.demo("Go on the quicktour of PEFT")
```
https://github.com/lavague-ai/lavague/blob/main/docs/docs/get-started/gradio.md

Use built-in Contexts to configure LaVague agents

main

A Context object defines the LLM, multi-modal LLM, and embedding models used by a LaVague agent. You can use pre-configured built-in Contexts to quickly set up agents with popular providers. To use a built-in context, initialize your WorldModel and ActionEngine using the from_context(my_context) method.

Available built-in Contexts:

Context	Pypi package	Default multi-modal LLM (World Model)	Default LLM (Action Engine)	Default embedding model (Action Engine)
Anthropic	`lavague-contexts-anthropic`	Claude 3.5 Sonnet	Claude 3.5 Sonnet	`text-embedding-3-small` (OpenAI)
Azure	`lavague-contexts-openai`	`gpt-4o`	No default	`text-embedding-3-small`
Fireworks	`lavague-contexts-fireworks`	`gpt-4o` (OpenAI)	`llama-v3p1-70b-instruct`	`nomic-embed-text-v1.5`
Gemini	`lavague-contexts-gemini`	`gemini-1.5-pro-latest`	`gemini-1.5-flash-latest`	`text-embedding-004`
OpenAI	`lavague-contexts-openai`	`gpt-4o`	`gpt-4o`	`text-embedding-3-small`

Handle CAPTCHAs via manual interaction
main
If running in non-headless mode, you can pause the agent's execution to manually resolve a CAPTCHA, pop-up, or login screen. Use time.sleep() or input() to create a pause, then proceed once the manual task is complete.
```
agent.get("https://www.bbc.co.uk")

import time
time.sleep(30) # Allows time for manual interaction

agent.run("What is the weather in Birmingham")
```
```
agent.get("https://www.bbc.co.uk")

import time
time.sleep(30)

agent.run("What is the weather in Birmingham")
```
https://github.com/lavague-ai/lavague/blob/main/docs/docs/get-started/troubleshoot.md

Implement a custom PromptsStore for production

main

By default, cached values are stored in YAML files and loaded into memory, which may cause high memory usage in production. To use an optimized storage system (like a database), implement the PromptsStore[str] abstract class and pass it to the cache wrapper via the store parameter.

Required methods to implement:

_get_for_prompt(self, prompt: str) -> str: Retrieve the output from your database using the prompt as a key.
_add_prompt(self, prompt: str, output: str): Store the prompt and its corresponding output in your database.

from lavague.contexts.cache import LLMCache
from lavague.contexts.cache.prompts_store import PromptsStore
from llama_index.llms.openai import OpenAI

class MyDataBaseStore(PromptsStore[str]):
  def _get_for_prompt(self, prompt: str) -> str:
    # return from DB with prompt key
    pass

  def _add_prompt(self, prompt: str, output: str):
    # store in DB
    pass

my_database_store = MyDataBaseStore()

llm = LLMCache(yml_prompts_file="llm.yml", fallback=OpenAI(model = "gpt-4o"), store=my_database_store)

from lavague.contexts.cache import LLMCache
from lavague.contexts.cache.prompts_store import PromptsStore
from llama_index.llms.openai import OpenAI

class MyDataBaseStore(PromptsStore[str]):
  def _get_for_prompt(self, prompt: str) -> str:
    # return from DB with prompt key
    pass

  def _add_prompt(self, prompt: str, output: str):
    # store in DB
    pass

my_database_store = MyDataBaseStore()

llm = LLMCache(yml_prompts_file="llm.yml", fallback=OpenAI(model = "gpt-4o"), store=my_database_store)

Use the LaVague QA CLI to generate pytest files
main
LaVague QA is a specialized tool designed to generate pytest files from Gherkin test descriptions. You can use the lavague-qa command-line interface to automate the creation of test scripts by providing a target URL and a .feature file containing Gherkin syntax.
```
lavague-qa --url https://amazon.fr/ --feature features/demo_amazon.feature
```
https://github.com/lavague-ai/lavague/blob/main/lavague-qa/README.md

Create a custom configuration file

main

To use a custom context, create a .py file in the lavague-tests/contexts folder. The file must define a context variable (an instance of Context) and a token_counter variable (an instance of TokenCounter).

The Context should be initialized with llm, mm_llm, and embedding models from llama_index.

from lavague.core.token_counter import TokenCounter
from llama_index.llms.openai import OpenAI
from llama_index.multi_modal_llms.openai import OpenAIMultiModal
from llama_index.embeddings.openai import OpenAIEmbedding
from lavague.core.context import Context

llm_name = "gpt-4o-mini"
mm_llm_name = "gpt-4o-mini"
embedding_name = "text-embedding-3-large"

token_counter = TokenCounter()

# Initialize models
llm = OpenAI(model=llm_name)
mm_llm = OpenAIMultiModal(model=mm_llm_name)
embedding = OpenAIEmbedding(model=embedding_name)

# Initialize context
context = Context(llm, mm_llm, embedding)

Navigate through Notion with LaVague

main

You can use LaVague to navigate Notion spaces and answer questions based on the content within nested documents. The agent uses a WorldModel to understand the page structure and an ActionEngine to control the browser via a driver (e.g., Selenium or Playwright) to perform interactions like clicking and navigating through pages.

from lavague.core import ActionEngine, WorldModel
from lavague.core.agents import WebAgent
from lavague.drivers.selenium import SeleniumDriver

selenium_driver = SeleniumDriver()
action_engine = ActionEngine(selenium_driver)
world_model = WorldModel()
agent = WebAgent(world_model, action_engine)

agent.get("https://maize-paddleboat-93e.notion.site/Welcome-to-ACME-INC-0ac66cd290e3453b93a993e1a3ed272f")
agent.run("What's the name of our Lead Developer ?")

Run LaVague with open-source or local models
main
LaVague agents utilize three models: a multi-modal model, the Action Engine's LLM, and an embedding model. You can replace any of these with any llama-index compatible alternative (including open-source models via local or remote inference) provided they have a sufficiently large context window.
Note: Performance varies significantly by model. Currently, finding open-source multi-modal LLMs that meet performance requirements is a known challenge.
https://github.com/lavague-ai/lavague/blob/main/docs/docs/get-started/FAQs.md