AgentQL Documentation

AgentQL is a suite of tools designed to connect LLMs and AI agents to the web by enabling data extraction and workflow automation on live websites. It uses an AI-powered natural language query language to pinpoint data and elements on any web page, including those with authenticated or dynamically generated content.

Key capabilities include:

• Natural Language Selectors: Find elements intuitively based on page content.
• Structured Output: Define the shape of your data directly within your queries.
• Self-healing Queries: Resilience to UI changes over time.
• Transforms: Apply data transformations directly within queries.
• Cross-site Compatibility: Use the same query across different websites with similar content structures.

AgentQL provides several interfaces for different development workflows:

• Python SDK: For running automation and scraping scripts in Python.
• JavaScript SDK: For running automation and scraping scripts in JavaScript.
• REST API: For executing queries directly without using an SDK.
• Integrations: Support for agent frameworks (e.g., Langchain), automation tools (e.g., Zapier), and an MCP server.
• Debugger Browser Extension: A Chrome extension to debug and refine queries in real-time on live sites.
• AgentQL Query Language: A natural language-based language for defining queries.
• Playground: A web-based environment to test queries and export Python scripts.

When dealing with infinite scroll, a single scrolling method might not work for all websites due to multiple scrollable areas or custom key mappings. You can experiment with different interaction patterns to find the one that works for your target site.

In the provided example, you can swap the scrolling mechanism by replacing the function call:

• End Key Method: Uses key_press_end_scroll(page) to simulate pressing the End key.
• Mouse Wheel Method: Use mouse_wheel_scroll(page) to simulate scrolling via the mouse wheel.

Testing these different methods is recommended when page.goto navigates to a site with complex scroll behaviors.

To lower the risk of being detected by anti-bot systems when using AgentQL, you can implement stealth mode techniques. This involves randomizing browser fingerprints and network characteristics to make automated requests appear more like organic human traffic.

Key techniques include:

• Randomizing HTTP Headers: Varying headers like User-Agent, Accept-Language, and Referer so consecutive requests do not look identical.
• Randomizing Window Size: Changing the browser viewport dimensions to avoid detection via static window sizes.
• Randomizing Timezone and Geolocation: Varying the browser's perceived location and time settings.
• Using a Proxy Server (Optional): Routing traffic through an external proxy provider (e.g., NetNut or BrightData) using a host, username, and password configuration.

To run the ecommerce data collection example, follow these steps:

1. Install the AgentQL SDK (refer to the official installation guide).
2. Save the provided JavaScript example file locally as main.js.
3. Execute the script from your terminal using Node.js:

node main.js

To run the provided login example script, follow these steps:

1. Install the AgentQL SDK.
2. Save the example Python file locally as main.py.
3. Execute the script using Python 3 from your project folder.

python3 main.py

This example demonstrates how to use AgentQL to extract news headlines from a selection of URLs and save the results to a CSV file. To run this example:

1. Install the AgentQL SDK.
2. Save the provided JavaScript file locally as main.js.
3. Execute the script using Node.js from your project folder.

Note: While the title mentions sentiment analysis, the core logic of this specific script focuses on returning news headlines from URLs.

node main.js