Google Maps Scraper

repository·main·Indexed 26 days ago

https://github.com/gosom/google-maps-scraper

A high-performance, open-source tool for extracting structured business data from Google Maps. It supports extracting over 33 data points, including business details, reviews, emails, and coordinates. The tool provides multiple interfaces including a CLI, Web UI, and REST API, and can be deployed on single machines or Kubernetes clusters. Key features include proxy rotation (SOCKS5, HTTP, HTTPS), flexible output formats (CSV, JSON, PostgreSQL, S3, LeadsDB), and a specialized grid scraping mode for bounding boxes.

Tokens
18.1K
Snippets
56
Records
123
Agent score
91%

What's inside google-maps-scraper

  1. Overview of Google Maps Scraper

    main

    Google Maps Scraper is an open-source tool designed to extract business leads and data from Google Maps at scale. It supports extracting over 33 data points, including business details, reviews, emails, phone numbers, websites, ratings, and coordinates.

    Key features include:

    • Multiple Interfaces: Use the tool via CLI, Web UI, or REST API.
    • High Performance: Capable of scraping approximately 120 places per minute using optimized concurrency.
    • Flexible Output: Export data to CSV, JSON, PostgreSQL, S3, LeadsDB, or via custom plugins.
    • Proxy Support: Built-in support for SOCKS5, HTTP, and HTTPS proxy rotation.
    • Scalability: Can be deployed on a single machine or scaled to Kubernetes clusters.
  2. Understand the Google Maps Scraper SaaS Merge Architecture

    main

    The google-maps-scraper project has merged the maps-scraper-pro SaaS layer into a single module. The resulting architecture combines existing core packages with new SaaS-specific packages under a single go.mod file.

    Core Packages (Unchanged/Extended):

    • gmaps/, runner/, web/, webdata/, deduper/, exiter/, leadsdb/, s3uploader/, tlmt/
    • postgres/: Now extended to include SaaS Connect/pgxpool functionality via pool.go.

    New SaaS Packages:

    • admin/: Admin dashboard with embedded templates and static assets.
    • api/: REST API with embedded Swagger documentation.
    • cli/: CLI prompt/UI utilities.
    • cryptoext/: Encryption utilities.
    • env/: Environment variable handling.
    • httpext/: HTTP utilities & middleware.
    • infra/: Cloud provisioning (DigitalOcean, Hetzner, PlanetScale, VPS, Cloudinit).
    • log/: Structured logging.
    • migrations/: SQL migrations (embedded *.sql files).
    • saas/: Root constants and environment variable names.
    • ratelimit/: Rate limiting and ratelimit/postgres.
    • rqueue/: River job queue integration.
    • scraper/: Scraper manager (provider, writer, lifecycle).

    New Binary:

    • cmd/gmapssaas/: The SaaS binary providing serve, worker, provision, update, and admin subcommands.
  3. Install Google Maps Scraper

    main

    Pull the official image which includes Playwright:

    docker pull gosom/google-maps-scraper

    Build from Source

    Requires Go 1.26.5+.

    git clone https://github.com/gosom/google-maps-scraper.git
    cd google-maps-scraper
    go mod download
    
    go build
    ./google-maps-scraper -input example-queries.txt -results results.csv -exit-on-inactivity 3m

    Note: The first run will download required Playwright browser libraries.

  4. Install the AI Agent Skill

    main

    To enable natural-language lead requests via an AI agent, install the skill using npx:

    npx skills add gosom/google-maps-scraper

    This allows you to prompt an agent with requests like Find me all dentists in Berlin with their emails. The skill handles Docker image checks, proxy selection, validation scrapes, and background monitoring.

  5. Run Google Maps Scraper via Command Line

    main

    You can run the scraper using Docker by mounting a volume for the Playwright cache, an input file containing queries, and an output directory.

    To run a basic scrape with a depth of 1 and an inactivity timeout of 3 minutes, use the following command structure:

    docker run \
      -v gmaps-playwright-cache:/opt \
      -v "$PWD/example-queries.txt:/queries.txt:ro" \
      -v "$PWD/gmaps-output:/out" \
      gosom/google-maps-scraper \
      -input /queries.txt \
      -results /out/results.csv \
      -depth 1 \
      -exit-on-inactivity 3m
  6. Use the NetNut Google Maps Places API

    main

    The NetNut Google Maps Places API retrieves structured JSON search results for businesses, points of interest, and venues. It handles proxy rotation, retries, and anti-bot bypass automatically.

    curl -X GET "https://serp-api.netnut.io/search?engine=google_places&q=coffee&hl=en&gl=us" \
      -u "username:password"
  7. Build Google Maps Scraper for macOS

    main

    To build the scraper manually on macOS, clone the repository, compile the binary, and organize it into a .app bundle structure.

    1. Clone and enter the directory:
    git clone https://github.com/gosom/google-maps-scraper.git
    cd google-maps-scraper
    1. Build the binary:

    go build -o google_maps_scraper

    1. Create the application structure:
    mkdir -p GoogleMapsScraper.app/Contents/MacOS
    mv google_maps_scraper GoogleMapsScraper.app/Contents/MacOS/
    go build -o google_maps_scraper
  8. Configure proxy credentials safely

    main

    To use an existing proxy, run the following command to enter your credentials securely in the terminal. This method ensures credentials are not passed through chat or logged in plain text.

    Important: The scraper uses the file path returned by this script via the --proxy-file flag. Do not attempt to pass credentials inline using -proxies.

    bash "$SKILL_DIR/scripts/configure-proxy.sh"
  9. Start the Web UI

    main

    To launch the browser-based interface, run the scraper via Docker and map a local data folder and port 8080:

    mkdir -p gmapsdata
    
    docker run \
      -v "$PWD/gmapsdata:/gmapsdata" \
      -p 8080:8080 \
      gosom/google-maps-scraper \
      -data-folder /gmapsdata

    Once running, access the UI at http://localhost:8080. Note that results may take at least 3 minutes to appear.

  10. Extract emails from business websites

    main

    To extract email addresses, add the -email flag to your command. Note that this process is slower than a standard scrape because the scraper must visit the business's website when available.

    docker run \
      -v gmaps-playwright-cache:/opt \
      -v "$PWD/example-queries.txt:/queries.txt:ro" \
      -v "$PWD/gmaps-output:/out" \
      gosom/google-maps-scraper \
      -input /queries.txt \
      -results /out/results.csv \
      -depth 1 \
      -email \
      -exit-on-inactivity 3m