DOCSEARCHbyAlgolia
Submit Docs

Botasaurus Documentation

repository·master·Indexed 26 days ago

https://github.com/omkarcloud/botasaurus

An all-in-one web scraping framework for building undetected, human-like scrapers. It features browser automation via the @browser decorator, humane HTTP requests via the @request decorator, and tools to bypass Cloudflare challenges. The ecosystem includes botasaurus-api for task management, botasaurus-humancursor for simulating natural mouse movements, and pg-cache-storage for PostgreSQL-based caching.

Tokens
51.9K
Snippets
148
Records
279
Agent score
90%

What's inside Botasaurus

  1. Overview of Botasaurus Desktop Extractors

    master

    A Desktop Extractor is a standalone application built with Botasaurus that runs locally on a user's machine to extract data from websites, PDFs, Excel files, and other documents.

    Key benefits include:

    • Zero Cloud Costs: Runs on the user's machine instead of expensive cloud VMs.
    • Faster Performance: Eliminates cloud latency and utilizes the user's local CPU.
    • Cross-Platform Deployment: Allows creating production-ready installers for Windows (.exe), macOS (.dmg), and Linux (.deb/.rpm) quickly.
    • Built-in Features: Includes task management, data tables, data export (Excel, CSV, etc.), sorting, filtering, caching, and auto-updates.

  2. Run Botasaurus in Docker

    master

    Use the Botasaurus Starter Template to run your project in a containerized environment. This template includes the necessary Dockerfile and Docker Compose configurations.

    git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
cd my-botasaurus-project
docker-compose build && docker-compose up

  3. Obtain Windows Code Signing for international businesses

    master

    For businesses outside the US and Canada, you must obtain an Extended Validation (EV) Code Signing Certificate (not an Organization Validation/OV) from a Certification Authority (CA).

    Requirements

    Code signing is designed for registered companies. Standard requirements include:

    • Registered Business: You must be a legally registered company (sole proprietorships may not qualify).
    • Physical Address: A physical business address must be available (no P.O. boxes or virtual offices) and verifiable via public records like Google Maps or DUNS.
    • Professional Web Presence: You must have a business website and a company email address (e.g., name@yourcompany.com). Personal emails like Gmail or Outlook are not accepted.

    Implementation Best Practices

    • Use Cloud Signing: When purchasing, select the "Cloud Signing" or "Cloud HSM" option. This allows for automated signing in CI/CD environments like GitHub Actions and avoids the need for physical hardware tokens.
    • Recommended Provider: DigiCert is recommended due to its popularity (used by Electron), extensive documentation in tools like @electron/windows-sign, and a 30-day refund policy if validation fails.

  4. Create a Google Cloud VM for Botasaurus

    master

    Create a VM instance in Google Cloud with the following recommended settings for scraping:

    SectionSettingValue
    Machine configurationNameYour app name (e.g., yahoo-finance)
    Machine configurationRegionus-central1 (Must match the static IP region)
    Machine configurationSeriesE2
    Machine configurationMachine Typee2-medium (2 vCPU, 4 GB memory)
    OS and storageBoot Disk TypeStandard persistent disk
    OS and storageBoot Disk Size20 GB
    Data protectionBack up your dataNo backups
    NetworkingFirewall/Allow HTTP traffic✓
    NetworkingFirewall/Allow HTTPS traffic✓
    NetworkingFirewall/Allow Load Balancer Health Checks✓
    NetworkingNetwork Interfaces/External IPv4 addressSelect the static IP created previously
    ObservabilityOps AgentUncheck Install Ops Agent

    Tip: For non-critical/personal workloads, enable Spot VMs (under Advanced > Provisioning model) to save 60-91% on costs.

  6. Install Botasaurus Desktop API on a VM

    master

    Follow these steps to install your application on a Debian/Ubuntu-based VM:

    1. Install dependencies: Run the installation script to set up the Botasaurus CLI and Apache web server:

      curl -sL https://raw.githubusercontent.com/omkarcloud/botasaurus/master/vm-scripts/install-bota-desktop.sh | bash

    2. Install the app: Use the install-desktop-app command with your Debian installer URL (uploaded to S3):

      python3 -m bota install-desktop-app --debian-installer-url <YOUR_S3_DEBIAN_URL>

    Configuration Options

    The install-desktop-app command supports:

    • --port: Sets the app port (e.g., 8001).
    • --api-base-path: Adds a prefix to all API routes (e.g., /yahoo-finance).
    • --skip-apache-request-routing: Disables automatic Apache configuration (use if manually configuring Nginx/Load Balancers).

    Hosting Multiple APIs on one VM

    To host multiple apps, use unique --port and --api-base-path values for each:

    python3 -m bota install-desktop-app \
  --debian-installer-url https://amazon-invoice-extractor.s3.us-east-1.amazonaws.com/Amazon+Invoice+Extractor-amd64.deb \
  --port 8001 \
  --api-base-path /amazon-invoices

  7. Set up a PostgreSQL database on Supabase

    master

    To use a PostgreSQL database with Botasaurus, you can host it on Supabase.

    1. Sign up at supabase.com using GitHub.
    2. Click New project.
    3. Configure your project settings:
      • Name: Any name (e.g., Pikachu).
      • Database Password: A strong password (required for the connection string).
      • Region: Select the region closest to your server.
    4. Once the project is created, navigate to Project settings > Database to find and copy your connection string.

  8. Reserve a static IP address for your VM

    master

    To ensure your VM is always reachable at the same IP address, use the Botasaurus CLI to reserve a static IP via Google Cloud Shell.

    1. Open the Google Cloud Console and click the Cloud Shell button.
    2. Install the Botasaurus CLI and run the create-ip command.
    3. When prompted, provide a name for your VM (e.g., yahoo-finance).
    4. When prompted for a region, press Enter to accept the default (us-central1) for lower costs.
    python -m pip install bota --upgrade 
python -m bota create-ip # Create a static IP address for your VM

  9. Add filters to a scraper using `botasaurus-server/ui`

    master

    You can enhance your scraper's UI by providing a list of filters to Server.addScraper. Filters allow users to narrow down scraped data based on specific criteria like text search, numeric ranges, or boolean values.

    To use them, import filters from botasaurus-server/ui and pass an array of filter instances in the options object of Server.addScraper.

    import { Server } from "botasaurus-server/server"
import { filters } from "botasaurus-server/ui"
import { scrapeProductData } from "../src/scrapeProductData"

const allFilters = [
    new filters.SearchTextInput("name"),
    new filters.MinNumberInput("reviews", { label: "Minimum Reviews" }),
    new filters.SingleSelectDropdown("category", {
        options: [
            { value: "apparel", label: "Apparel" },
            { value: "electronics", label: "Electronics" }
        ]
    }),
    new filters.MultiSelectDropdown("tags", {
        options: [
            { value: "cotton", label: "Cotton" },
            { value: "casual", label: "Casual" },
            { value: "computer", label: "Computer" },
            { value: "portable", label: "Portable" }
        ]
    }),
    new filters.IsTrueCheckbox("is_available", { label: "Is Available" })
]

Server.addScraper(
    scrapeProductData,
    { filters: allFilters }
)

  10. Set up the Botasaurus Starter Template

    master

    To start a new UI-based scraping project using the recommended starter template, follow these steps:

    1. Clone the repository:
      git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
cd my-botasaurus-project
    2. Install dependencies:
      python -m pip install -r requirements.txt
python run.py install
    3. Run the scraper:
      python run.py

    The UI will be available at http://localhost:3000/.

    git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
cd my-botasaurus-project
python -m pip install -r requirements.txt
python run.py install
python run.py

  11. Enable Human Mode for Anti-Detection

    master

    Use driver.enable_human_mode() to perform realistic, human-like mouse movements. This helps bypass bot detection systems like Cloudflare Turnstile. You can disable it using driver.disable_human_mode() when no longer needed.

    driver.enable_human_mode()
# ... perform human-like interactions ...
driver.disable_human_mode()

  12. Enable Caching in Botasaurus

    master

    Caching allows Botasaurus to save scraper results. When a scraper is executed with the same input data, Botasaurus serves the cached result instead of re-running the scraper, saving time and compute resources.

    To enable caching globally for all scrapers, call Server.enableCache() in your server entry point (typically src/scraper/backend/server.ts).

    import { Server } from "botasaurus-server/server"

// Enable caching for all scrapers
Server.enableCache();