DOCSEARCHbyAlgolia
Submit Docs

Katana Web Crawling and Spidering Framework

repository·dev·Indexed 12 days ago

https://github.com/projectdiscovery/katana

A high-performance, configurable web crawling and spidering framework for security researchers and developers. Katana supports standard and headless browser modes for deep crawling of JavaScript-heavy websites, featuring ML-based page-type classification, automatic form filling, and advanced page content similarity (PCS) filtering. It provides granular control over crawl depth, rate-limits, and scope via regex or field-scope, with support for various output formats including JSONL.

Tokens
13K
Snippets
42
Records
59
Agent score
90%

What's inside Katana

  1. Overview of Katana features

    dev

    Katana is a next-generation crawling and spidering framework with the following capabilities:

    • Modes: Supports both Standard and Headless modes.
    • JavaScript Support: Capable of JavaScript parsing and crawling.
    • Form Interaction: Customizable automatic form filling.
    • Scope Control: Manage crawling scope using preconfigured fields or Regex.
    • Intelligence: Includes a Knowledge base with ML-based page-type and form classification (models are auto-downloaded).
    • Input Sources: Accepts input via STDIN, URL, and LIST.
    • Output Formats: Supports STDOUT, FILE, and JSON.

  2. Use Knowledge Base classification

    dev

    Katana can use machine-learning to classify crawled pages using the dit engine. When enabled via -kb (--knowledge-base), classification results are attached to the knowledgebase field in JSONL output.

    Key features:

    • Page Type Classification: Identifies if a page is a login, error, captcha, parked, etc.
    • Form Identification: Identifies forms on the page and their field types.
    • Secrets Extraction (-kb-secrets): Surfaces detected secrets (API keys, tokens) under the secrets key.
    • Endpoint Extraction (-kb-endpoints): Classifies requests as REST, GraphQL, SOAP, or XHR under the endpoints key.
    • Filtering (-fpt): Use -fpt <type> to filter results to specific page types (e.g., login,error). This automatically enables -kb.
    {
  "timestamp": "...",
  "request": { "...": "..." },
  "response": {
    "...": "...",
    "knowledgebase": {
      "PageType": "login",
      "Forms": [{
        "type": "login",
        "fields": {"username": "username or email", "password": "password"}
      }]
    }
  }
}

  3. Extend the Headless Crawler via Interfaces

    dev

    The headless engine is designed to be extensible through several key interfaces:

    • FingerprintStrategy: Allows users to plug in custom logic for SimHash or visual/screenshot-based fingerprinting.
    • ValueProvider & SiteAdapter: These interfaces can depend on PageState to make intelligent decisions about crawler actions.
    • Diagnostics Sink: A hook that receives PageState and the serialized Action graph, enabling offline visualization of the crawl.

  4. Understand the Headless Crawler State Management

    dev

    The headless crawler's state is managed by state.go. This component acts as the central 'state-manager' for the entire headless package, including browser wrappers, normalizers, graphs, and diagnostics.

    To ensure scalability, reliability, and effective de-duplication, the state manager is responsible for three core responsibilities:

    1. Fingerprinting: Building a reproducible fingerprint (state ID) for the current page.
    2. Metadata Persistence: Storing the surrounding metadata required to replay a specific state later.
    3. State Restoration: Providing deterministic, 'cheapest-first' logic to navigate back to any previously recorded state.

    This architecture allows the crawler to handle complex Single Page Applications (SPAs) and minimize redundant crawling by identifying similar page states.

  5. Compare Standard and Headless crawling modes

    dev

    Katana offers two primary crawling modalities:

    1. Standard Mode (Default): Uses the standard Go http library. It is significantly faster because it lacks browser overhead, but it only analyzes raw HTTP response bodies. It will not execute JavaScript or render the DOM, meaning it may miss endpoints generated by asynchronous calls or complex web applications.

    2. Headless Mode: Enabled via the -headless flag. It executes requests within a browser context. This provides better coverage by analyzing both raw responses and browser-rendered content (via JavaScript execution). It also ensures the HTTP fingerprint (TLS and User-Agent) matches a legitimate browser.

  6. How Page Fingerprinting and Deduplication Works

    dev

    Katana uses a two-tier fingerprinting strategy to identify pages and deduplicate states in the crawler graph. This allows the crawler to treat minor DOM variations (like ads or CSRF tokens) as the same state.

    Fingerprint Tiers

    • ExactHash: A SHA-256 hash of the strippedDOM.
    • FuzzyHash: A SimHash64 hash based on 4-word shingles of the strippedDOM.

    Equality Logic

    Two states are considered equal if:

    • Their ExactHash matches,
    • OR the Hamming distance between their FuzzyHash values is $\le 3$ bits.

    DOM Normalization Process

    Before hashing, the DOM is processed via domNormalizer to:

    • Strip <script> and <style> tags.
    • Remove dynamic IDs.
    • Remove transient event attributes (e.g., onclick, onmouseover).
    • Collapse all whitespace into single spaces.
    type PageState struct {
    ExactHash        string // always present
    FuzzyHash        uint64 // present if SimHash computed
    URL              string
    Title            string
    Depth            int
    StrippedDOM      string
    NavigationAction *Action // edge that produced this state
    Timestamp        time.Time
}

  7. Configure Page Content Similarity (PCS) filtering

    dev

    Katana provides an optional Layer-2 filtering mechanism to deduplicate pages based on content similarity after exact MD5 content deduplication. This helps skip parsing and enqueuing pages that are too similar to already processed ones.

    To use this, you must enable it with the -pcs flag and then select a mode using -pcsm.

    Available Modes (-pcsm):

    • simhash (default): Best for detecting near-duplicates or template clones. Use -pcsd to specify Hamming distance.
    • tfidf: Best for topical cosine similarity. Use -pcst to set the threshold.
    • bm25: Best for topical similarity with length normalization. Use -pcst to set the threshold.

    Key Flags:

    • -pcs: Enables Page Content Similarity filtering.
    • -pcsm <mode>: Sets the similarity mode (simhash, tfidf, or bm25).
    • -pcst <threshold>: Sets the similarity threshold (e.g., 0.85).
    • -pcsn <number>: Sets the cluster budget.
    # Near-duplicate detection (default mode)
katana -u https://example.com -pcs

# TF-IDF topical filtering
katana -u https://shop.example.com -pcs -pcsm tfidf -pcst 0.85 -pcsn 2

# BM25 mode
katana -u https://example.com -pcs -pcsm bm25

  8. How the Return-to-Origin Algorithm Works

    dev

    When the crawler needs to navigate from a current page back to a targetOriginID, it follows a prioritized three-step approach to find the most efficient path:

    1. Step 1: Element Re-use: If the NavigationAction contains a non-nil Element, the crawler attempts to locate it via XPath. It verifies the element is visible, interactable, and passes a canonicalized DOM equality check.
    2. Step 2: Browser History: The crawler attempts to walk back through the browser history using page.GetNavigationHistory(). It iterates back (up to a limit of 10 steps), calling WaitForRouteChange() after each back() command, until the URL and Title match the target.
    3. Step 3: Graph Shortest Path: If history fails, the crawler calculates the shortest path in the crawlerGraph from the currentID to the targetID. It executes each Action in the path, waiting for route changes after each step.

    If all steps fail, the crawler retries from an empty page (a fresh tab). If that also fails, it returns ErrNoNavigationPossible.

  9. Install Katana

    dev

    Katana can be installed via Go or by using Docker.

    Go Installation Requires Go 1.26+. It is recommended to use the latest version of Go.

    Docker Installation You can pull the latest image and run Katana directly in standard or headless mode.

    Ubuntu Prerequisites If running on Ubuntu, ensure you have zip, curl, wget, git, and snapd installed. For headless mode, you must also install google-chrome-stable.

    CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest

  10. Solve Captchas in Headless mode

    dev

    Katana can automatically detect and solve captchas during headless crawling. It supports reCAPTCHA v2, reCAPTCHA v3, reCAPTCHA Enterprise, Cloudflare Turnstile, and hCaptcha.

    Currently, the supported provider is capsolver.

    Configuration via CLI:

    • -csp <provider>: Specify the provider (e.g., capsolver).
    • -csk <key>: Specify the API key for the provider.

    Configuration via Environment Variables:

    • CAPTCHA_SOLVER_PROVIDER
    • CAPTCHA_SOLVER_KEY
    # Using CLI flags
katana -u https://example.com -headless -csp capsolver -csk YOUR_API_KEY

# Using Environment Variables
export CAPTCHA_SOLVER_PROVIDER=capsolver
export CAPTCHA_SOLVER_KEY=YOUR_API_KEY
katana -u https://example.com -headless

  11. Define and use custom fields with regex

    dev

    You can extract specific information (like emails or phone numbers) from page responses using custom regex rules defined in a YAML configuration file.

    1. Configuration: Create a YAML file (default location: $HOME/.config/katana/field-config.yaml) or specify a custom path using -flc.
    2. Attributes:
      • name (required): The identifier used as the value for the -f CLI option.
      • type (required): Currently only regex is supported.
      • part (optional): Where to look (response [default], header, or body).
      • group (optional): The specific regex match group to extract (e.g., group: 1).

    Example YAML structure:

    - name: email
  type: regex
  regex:
  - '([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'
    katana -u https://tesla.com -f email,phone

  12. Advanced filtering with DSL expressions

    dev

    Katana supports a Domain Specific Language (DSL) for complex matching and filtering based on response attributes. Use -mdc (match-condition) to include results or -fdc (filter-condition) to exclude them.

    Common DSL usage examples:

    • Status Code: status_code == 200 or status_code != 403
    • Content Matching: contains(endpoint, "default")
    • Technology Matching: contains(to_lower(technologies), "php")

    DSL functions can be applied to any keys available in the JSONL output.

    katana -u https://www.hackerone.com -mdc 'status_code == 200'
katana -u https://www.hackerone.com -fdc 'contains(to_lower(technologies), "cloudflare")'