DOCSEARCHbyAlgolia
Submit Docs

SiteOne Crawler

repository·main·Indexed 21 days ago

https://github.com/janreges/siteone-crawler

A high-performance Rust-based website crawler and QA toolkit for security, performance, SEO, and accessibility audits. Version 2.6.1 provides tools for offline cloning, Markdown export for AI training, sitemap generation, and cache warming. It features a quality gate for CI/CD integration, optional Chromium-based browser rendering for SPAs, and an AI assistant for SEO analysis. Available as a dependency-free binary with an interactive wizard, CLI, and desktop GUI.

Tokens
42.3K
Snippets
99
Records
162
Agent score
74%

What's inside siteone-crawler

  1. Overview of SiteOne Crawler

    main

    SiteOne Crawler is a high-performance website analyzer, cloner, and converter written in Rust. It is designed for developers, SEO specialists, and website owners to perform tasks such as security/performance auditing, site backups, and content conversion.

    Key Capabilities:

    • Crawling & Analysis: Deep crawling of pages and assets, error identification (404s, redirects), and sitemap generation.
    • Reporting: Generates interactive HTML audit reports with quality scoring (0.0-10.0), JSON for programmatic use, and text for terminal output. Supports automated email reports via SMTP.
    • Cloning & Conversion: Creates complete offline website clones for serverless browsing and converts websites into clean Markdown (ideal for AI training or documentation).
    • CI/CD Integration: Includes a quality gate (--ci) that can block deployments by exiting with code 10 if quality thresholds are not met.
    • Deployment Support: Can be used for cache warming by crawling a site immediately after deployment.

  2. Overview of SiteOne Crawler features

    main

    SiteOne Crawler is a high-performance, native Rust-based tool designed for website crawling, analysis, and content transformation. Key capabilities include:

    • Crawler: Powerful website crawling with device simulation (desktop/mobile/tablet) and robots.txt respect.
    • Dev/DevOps Assistant: Stress/load testing with configurable concurrency (--workers) and request rates (--max-reqs-per-sec).
    • Analyzer: Automated auditing for SEO, security, accessibility, and performance.
    • Reporter: Generates interactive HTML reports, structured JSON, and colored text output, including an SMTP mailer for HTML reports.
    • Offline Website Generator: Clones websites to local HTML files, supporting multi-domain clones with intelligent cross-linking.
    • Website to Markdown Converter: Exports sites to multi-file or single-file markdown (optimized for AI tools), with a built-in web server for viewing.
    • Sitemap Generator: Creates sitemap.xml and sitemap.txt files.
    • Quality Scoring: Automatic scoring (0.0-10.0) across Performance, SEO, Security, Accessibility, and Best Practices.
    • CI/CD Integration: Configurable quality gates that exit with code 10 on failure.
    • AI Assistant (Optional): LLM integration for SEO analysis and llms.txt generation.
    • Browser Rendering (Optional): Uses Chromium via CDP to crawl SPA/JavaScript-heavy sites and capture screenshots.

  3. Inspect SSL/TLS Certificate Information

    main

    The crawler performs a pure Rust SSL/TLS inspection (using rustls and x509-parser) to provide details about the primary host's certificate. This does not require openssl or sh on the host.

    Reported Info:

    • Issuer, Subject, Subject Alternative Names (SANs).
    • Valid from / Valid to (with status like VALID already 35 day(s)).
    • Serial number, Signature algorithm, Public key details.
    • SHA-256 fingerprint.
    • Supported protocols (e.g., TLSv1.2, TLSv1.3).
    • Trust (whether it is validated by the system CA store).

    Security Findings:

    • Critical/Warning: Expired/not-yet-valid certs, hostname mismatches, self-signed/untrusted roots, weak signature algorithms (SHA-1/MD5), weak public keys (RSA < 2048-bit), or obsolete protocols (SSLv3, TLS 1.0, TLS 1.1).
    • Informational: Missing Common Name (CN) or empty Subject.
    • Positive (OK): Trusted chains, strong signatures (SHA-256+), strong keys, and modern protocols.

    Note: If you use the --accept-invalid-certs (-aic) flag, certificate problems are reported as Warnings instead of Criticals.

  4. Understand the text output severity levels

    main

    The SiteOne Crawler text output uses specific severity levels to categorize findings. While the documentation shows bracketed labels like [CRITICAL] or [INFO], the actual output uses corresponding emoji characters to represent these levels:

    • CRITICAL (red circle): Serious issues requiring immediate attention (e.g., pages with critical security findings, skipped URLs).
    • WARNING (warning sign): Issues that should be addressed (e.g., missing Brotli support, missing ARIA labels, skipped heading levels).
    • INFO (fast-forward): Informational items (e.g., robots.txt status, external URL count, DNS IPv6 status, 404 notices).
    • OK (green check): Positive findings confirming correct configuration (e.g., valid SSL certificate, no redirects, all titles unique).
    • NOTICE (pin): Export notifications (e.g., text/JSON/HTML report save paths and timing).

  5. Understand the Progress Report format

    main

    The Progress Report displays real-time or final crawl status. The display mode depends on your terminal width:

    • Wide mode (>= 140 chars): Includes % (percentage processed) and Bar (visual progress indicator) columns.
    • Compact mode (< 140 chars): Omits % and Bar columns. You may see a message: Detected terminal width [X] < 140 chars - compact mode activated.

    Column Definitions:

    • Progress (X/Y): X is the current URL sequence number; Y is the total URLs found so far.
    • URL: The path or full URL being processed.
    • Status: HTTP status code (e.g., 200, 404, 301).
    • Type: Content type (e.g., HTML, JS, CSS, Image).
    • Time: Download duration.
    • Size: Downloaded content size.
    • Cache: Detected lifetime (e.g., 60 min, etag, none).
    • Access.: Accessibility issues summary (e.g., 3/1 means 3 OK / 1 warning; a single number like 7 indicates the count of findings in the most relevant severity).
    • Best pr.: Best practices issues summary (e.g., 1/6 means 1 OK / 6 warnings).
    | Progress | %   | Bar | URL                                                     | Status | Type | Time  | Size  | Cache  | Access. | Best pr. |
| :------- | :-- | :-- | :------------------------------------------------------ | :----- | :--- | :---- | :---- | :----- | :------ | :------- |
| 1/40     | 2%  | >   | /                                                       | 200    | HTML | 4 ms  | 50 kB| 60 min | 3/1     | 7        |

  6. Understand the SiteOne Crawler JSON output structure

    main

    The SiteOne Crawler generates a comprehensive JSON output file containing detailed metadata about the crawl process, individual URL results, quality scores, and aggregated analysis tables. This dataset is designed for deep website auditing and automation.

    Key Data Components:

    • Crawl Metadata (crawler): Details about the execution, including version, execution time, command used, hostname, and the final user agent.
    • Configuration (options): A complete record of all CLI configuration values used during the crawl.
    • Quality Scores (qualityScores): Overall and per-category scores (0-10) including details on deductions.
    • Visited URL Results (results): An array of objects for every visited URL, containing:
      • url: The address visited.
      • status: HTTP status code.
      • elapsed: Request performance (time elapsed).
      • size: Response body size.
      • content_type: Type of resource (HTML, CSS, JS, etc.).
      • cache: Caching information (flags, lifetime).
      • extras: Additional analysis results.
    • Aggregate Statistics (stats): Totals for URLs, sizes, timings, and status code counts.
    • Findings Summary (summary): A list of findings categorized by severity (OK, Warning, Critical, Info) used for scoring.
    • Analysis Tables (tables): Structured, aggregated data covering topics like redirects, 404s, SSL/TLS info, SEO, performance, and security headers.

  7. Understand JSON Output Data Types in Analysis Tables

    main

    When consuming the tables key in the SiteOne Crawler JSON output, be aware that all values in all table rows are strings.

    Even numeric values such as counts, times, and sizes are serialized as strings. For example:

    • A count of 51 is represented as "51".
    • A request time of 0.003 seconds is represented as "0.003".
    • Empty values are represented as an empty string "".

    This serialization applies to every table described in the documentation.

  8. Analyze Skipped URLs and Summaries

    main

    The crawler provides two ways to view URLs that were not crawled:

    1. Skipped URLs Summary: A high-level grouping by Reason (e.g., Not allowed host, Robots.txt, Max depth reached) and Domain, showing the count of Unique URLs for each combination.
    2. Skipped URLs List: A detailed list showing:
      • Reason: Why it was skipped.
      • Skipped URL: The specific URL.
      • Source: How it was discovered (e.g., <a href>, <img src>, CSS url()).
      • Found at URL: The parent URL where the link was located.

  9. Understand Quality Scoring metrics

    main

    The crawler calculates a quality score from 0.0 to 10.0 based on five weighted categories. This score is useful for assessing website health and is included in console, JSON, and HTML reports.

    Scoring Categories:

    • Security (25%): SSL/TLS certificates, security headers, unsafe protocols.
    • Performance (20%): Response times, slow URLs.
    • SEO (20%): Missing H1, title uniqueness, meta descriptions, 404s, redirects.
    • Accessibility (20%): Lang attribute, image alt text, form labels, ARIA, heading levels.
    • Best Practices (15%): Duplicate/large SVGs, deep DOM, Brotli/WebP support.

    Score Labels:

    • 9.0-10.0: Excellent (green)
    • 7.0-8.9: Good (blue)
    • 5.0-6.9: Fair (yellow)
    • 3.0-4.9: Poor (purple)
    • 0.0-2.9: Critical (red)

  10. Choose between Text and JSON output formats

    main

    SiteOne Crawler provides two primary output formats depending on your goal:

    1. Text Output (Default/Standard): Best for human readability, quick manual health checks, and simple command-line processing with tools like grep.
    2. JSON Output (--output-json-file): Best for programmatic consumption, detailed data analysis, integration with dashboards, or complex automated workflows. If you need to build a tool that reacts to crawl data, use the JSON format.

    For details on the structured schema, refer to the JSON Output Documentation.

  11. Review External, Redirected, and 404 URLs

    main

    The text output includes specific sections for tracking link integrity and external discovery:

    • External URLs: Lists URLs found that are outside the target domain. Shows the External URL, the number of Pages it appeared on, and up to 5 Found on URL examples.
    • Redirected URLs: Lists HTTP redirects. Shows Status (e.g., 301), URL from, URL to, and the Found at URL. If none exist, it displays No redirects found.
    • 404 URLs: Lists broken links. Shows Status (404), URL 404, and the Found at URL (the page containing the broken link).

  12. Generate an AI Executive Summary

    main

    The summary action provides a high-level synthesis of the entire website. It evaluates five areas (security, accessibility, SEO, performance, infrastructure) using aggregated crawl data to produce up to 15 prioritized, actionable recommendations.

    Cost Efficiency: The cost is fixed at 6 LLM calls regardless of site size, as it uses aggregated data rather than raw per-URL lists.

    Optimization Tip: You can use --ai-extra-body to disable thinking for the individual area evaluations and --ai-synthesis-extra-body to enable thinking specifically for the final synthesis step to improve quality without unnecessary cost.

    # Run only the summary action
./siteone-crawler --url=https://example.com/ \
  --ai-provider=openai --ai-model=gpt-5-mini --ai-actions=summary

# Optimized: disable thinking for evaluations, enable for synthesis
--ai-extra-body='{"chat_template_kwargs":{"enable_thinking":false}}' \
--ai-synthesis-extra-body='{"chat_template_kwargs":{"enable_thinking":true}}'