DOCSEARCHbyAlgolia
Submit Docs

Maxun Web Data Platform

website·Indexed 32 days ago

https://docs.maxun.dev

An open-source web data platform that converts websites into structured data for AI agents and workflows. It offers scraping, crawling, extraction, searching, and monitoring via a REST API, Node.js and Python SDKs, a CLI, and an MCP server, featuring AI-driven extraction from natural-language prompts and data retrieval behind logins.

Tokens
1.8K
Snippets
8
Records
14
Agent score
11%

What's inside Maxun

  1. Overview of Maxun Web Data Platform

    Maxun is an open-source web data platform (AGPLv3) that converts websites into structured data for AI agents and workflows. It uses real browsers to handle JavaScript-heavy sites, pagination, infinite scroll, and pages behind logins. The platform is available as a fully managed Cloud version or a self-hosted edition via Docker. All automations in Maxun are called 'robots', which are reusable, parameterized tasks that can be triggered via API, SDK, CLI, MCP, or a no-code interface.

  2. Maxun platform capabilities overview

    Maxun is a web data platform that converts websites into structured data for AI agents. Key capabilities include:

    • Scraping: Convert websites to clean HTML, LLM-ready Markdown, and screenshots.
    • Extraction: Build robots to extract structured JSON (lists, tables, details) using natural language prompts or manual configuration.
    • Automation: Turn websites into APIs via REST API or webhooks, and sync data to Google Sheets or Airtable.
    • Advanced Features: Support for extracting data behind logins, deep extraction, stealth mode, and custom proxy (BYOP) configurations.

  3. Use AI-powered extraction and Smart Queries

    Maxun offers two primary AI-driven capabilities for data retrieval:

    1. AI Mode Extraction: Describe the data you want in natural language (e.g., "Extract the first 20 product names and prices"), and an LLM builds the extractor. The SDK supports Anthropic and OpenAI providers.
    2. Smart Queries: Attach a natural-language prompt to a Scrape robot. After scraping the page, an LLM answers the prompt based on the page content.

  4. Install Maxun Node.js SDK

    The Maxun Node.js SDK requires Node.js 18 or higher. Install it via npm.
    npm install maxun-sdk

  5. Extract data from pages behind a login

    Maxun supports accessing authenticated pages using two methods:

    • Password-free (Cloud): Use the Maxun Chrome extension to sync an existing logged-in browser session, avoiding the need to share credentials.
    • Credential-based (Self-hosted): Store encrypted login credentials directly within your own self-hosted instance.

  6. Configure anti-bot and reliability settings

    Maxun employs several strategies to avoid being blocked by websites:

    • Stealth Mode: Enabled by default on Maxun Cloud to make browser activity appear human.
    • Managed Infrastructure: Maxun Cloud provides CAPTCHA bypass and automatic proxy rotation.
    • BYOP (Bring Your Own Proxy): Self-hosted users can configure their own HTTP or SOCKS proxies.

  7. Connect Maxun Cloud to an MCP-compatible client

    Maxun provides an MCP server to connect AI clients like Claude Desktop, Cursor, or Windsurf. If the client does not support native HTTP, use the mcp-remote bridge.
    // Native HTTP Support
{
  "mcpServers": {
    "maxun": {
      "url": "https://app.maxun.dev/api/mcp",
      "headers": { "x-api-key": "your_api_key_here" }
    }
  }
}

// Using mcp-remote bridge
{
  "mcpServers": {
    "maxun": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://app.maxun.dev/api/mcp", "--header", "x-api-key:your_api_key_here"]
    }
  }
}

  8. Understand Maxun Cloud credit costs

    Maxun Cloud operations are billed based on credits. Failed requests are not charged. Unused credits do not roll over unless upgrading plans mid-cycle.
    OperationCost
    Scrape (standard)1 credit per page
    Scrape with Smart Queries+2 credits per run
    Crawl (standard)1 credit per page
    Extract (Recorder mode)1 credit per 4 rows
    Extract (AI mode)1 credit per 3 rows
    Screenshot1 credit per screenshot
    Search (standard)1 credit per 5 results
    Search + Scrape1 credit per result

  9. Scrape web content using the Maxun Python SDK

    The Maxun Python SDK requires Python 3.8+. For self-hosted instances, set the MAXUN_BASE_URL environment variable; Cloud users can omit it.
    pip install maxun
    from maxun import Scrape, Config

scraper = Scrape(Config(api_key="your-api-key"))

robot = await scraper.create(
    "Content Scraper",
    "https://example.com/article",
    formats=["markdown", "html"],
)
result = await robot.run(timeout=30000)

  10. Manage robots via the Maxun REST API

    The Maxun REST API allows programmatic management of robots and their execution. A successful run returns status: "success" with extracted data located in run.data.
    GET /api/robots                      # List robots
GET /api/robots/{id}                 # Get a specific robot
POST /api/robots/{id}/duplicate      # Duplicate a robot with a new URL
POST /api/robots/{id}/runs          # Execute a robot
GET /api/robots/{id}/runs/{runId}   # Retrieve run results and extracted data

  11. Extract web data using the Maxun Node.js SDK

    The Maxun Node.js SDK allows for both AI-driven extraction using natural language prompts and selector-based extraction. AI extraction requires an LLM provider (e.g., Anthropic) and a corresponding API key.
    import { Extract, Scrape, Crawl, Search } from 'maxun-sdk';

const scraper = new Scrape({ apiKey: process.env.MAXUN_API_KEY });
const extractor = new Extract({ apiKey: process.env.MAXUN_API_KEY });

// AI extraction from a natural-language prompt
const robot = await extractor.extract('https://example.com', {
  prompt: 'Extract first 20 product names and prices',
  llmProvider: 'anthropic',
  llmApiKey: process.env.ANTHROPIC_API_KEY
});

// Selector-based extraction without an LLM
const productRobot = await extractor
  .create('Product Extractor')
  .navigate('https://example.com/products')
  .captureText({ productName: '.product-title', price: '.price' });

const result = await productRobot.run({ waitForCompletion: true, timeout: 60000 });

  12. Manage web robots via the Maxun CLI

    The Maxun CLI provides commands to authenticate, create robots for scraping, crawling, extraction, and searching, and manage robot runs.
    npm install -g maxun-cli

# Authentication
maxun login --api-key <your-api-key>                     # Cloud
maxun login --api-url http://localhost:8080 --api-key <key>  # Self-hosted

# Robot Creation
maxun robots scrape https://example.com -f markdown      # create a scrape robot
maxun robots crawl https://example.com -f markdown       # create a crawl robot
maxun robots extract -p "Extract all job titles and locations" -u https://example.com/jobs
maxun robots search "open source web scraping" -f markdown

# Management and Execution
maxun robots list --table
maxun run <robot-id>                                     # run and return JSON
maxun runs get <robot-id> <run-id> -f csv -o data.csv    # export a run as CSV
maxun credits                                            # check remaining credits