DOCSEARCHbyAlgolia
Submit Docs

FingerprintHub Documentation

repository·main·Indexed 23 days ago

https://github.com/0x727/fingerprinthub

A community-driven YAML-based fingerprint library for the observer_ward identification tool. It provides a structured framework for defining detection rules using HTTP and TCP probes, matchers (word, favicon, regex), and extractors to identify services and map them to CPE strings. Includes a helper CLI for converting, formatting, and synchronizing fingerprint rules, as well as a parser for Nmap-style fingerprint files.

Tokens
4.1K
Snippets
7
Records
23
Agent score
81%

What's inside FingerprintHub

  1. Overview of FingerprintHub

    main

    FingerprintHub is a community-driven fingerprint library designed to simplify the management and use of fingerprint rules. It is specifically built for use with observer_ward, a community-based fingerprint identification tool.

    Key characteristics:

    • Format: Rules are written in Yaml.
    • Automation: Supports custom requests and uses GitHub Actions for automatic fingerprint library updates.
    • Go SDK Alternative: If you require a Go-based SDK, refer to neutron by chainreactors.
    • Legacy Support: Older versions of fingerprints are archived in the v3 branch.

  2. Understand FingerprintHub rule composition

    main

    FingerprintHub rules follow a structure inspired by Nuclei templates. A rule is composed of four main parts:

    1. Basic Information (ID and Info): Stores metadata about the fingerprint and its relationship to vulnerabilities (using CPE).
    2. Probes: Custom data packet senders, supporting http and tcp clients.
    3. Matchers: Logic to identify a fingerprint using keywords, regular expressions, or favicon hashes.
    4. Extractors: Tools to pull specific information (like version numbers) from responses using regex or JSONPath.

  3. Structure of a Fingerprint Rule

    main

    Fingerprint rules are defined in YAML format. A rule consists of identification metadata (id and info) and the detection logic (http).

    Rule Components

    • id: A unique identifier for the fingerprint (e.g., thinkphp).
    • info: Metadata containing the name, author, tags, severity, and additional product/vendor details.
    • http: A list of request configurations used to identify the target. Each request includes:
      • method: The HTTP method (e.g., GET).
      • path: The target URL path, which can use the {{BaseURL}} placeholder.
      • matchers: A list of conditions that must be met to trigger a match. Supported matcher types include favicon (using hashes), word (searching for specific strings in the body or headers), and others.
    id: thinkphp
info:
  name: thinkphp
  author: cn-kali-team
  tags: detect,tech,thinkphp
  severity: info
  metadata:
    product: thinkphp
    vendor: thinkphp
    verified: true
http:
  - method: GET
    path:
      - '{{BaseURL}}/'
    matchers:
      - type: favicon
        hash:
          - f49c4a4bde1eec6c0b80c2277c76e3db
      - type: word
        words:
          - href="http://www.thinkphp.cn">thinkphp</a>
          - thinkphp_show_page_trace
        case-insensitive: true
      - type: word
        words:
          - 'x-powered-by: thinkphp'
        part: header
        case-insensitive: true

  4. Configure ID and Basic Information

    main

    The id and info sections define the identity and metadata of the fingerprint. The metadata field within info is critical for linking fingerprints to vulnerabilities via CPE (Common Platform Enumeration).

    By populating metadata.vendor and metadata.product based on CPE strings (e.g., cpe:/a:apache:shiro), you enable reverse lookups for associated CVEs.

    Field Reference:

    FieldTypeDescription
    idStringUnique rule ID (no Chinese characters supported).
    nameStringHuman-readable name (Chinese supported).
    authorStringComma-separated list of authors.
    tagsStringComma-separated list of tags.
    severityEnumunknown, info, low, medium, high, critical.
    metadataHashMap<String,String>Dictionary for arbitrary data (e.g., vendor, product, verified).
    descriptionOption<String>Optional description.
    referenceOption<Vec<String>>Optional list of reference links.
    id: shiro
info:
  name: shiro
  author: cn-kali-team
  tags: detect,tech,shiro
  severity: info
  metadata:
    product: shiro
    vendor: apache
    verified: true

  5. Submit new fingerprint rules via Pull Request

    main

    FingerprintHub follows a standard Fork-and-Pull model for contributing new fingerprint rules. Follow these steps:

    1. Fork and Clone: Fork the repository to your GitHub account and clone it locally.
      git clone git@github.com:YOUR_USERNAME/FingerprintHub.git
cd FingerprintHub
    2. Setup Upstream: Add the original repository as an upstream remote to keep your fork updated.
      git remote add upstream git@github.com:0x727/FingerprintHub.git
git fetch upstream
    3. Configure Git: Ensure your GitHub identity is configured.
      git config --global user.name "$GITHUB_USERNAME"
git config --global user.email "$GITHUB_EMAIL"
git config --global github.user "$GITHUB_USERNAME"
    4. Create a Feature Branch: Do not modify the main branch directly. Create a new branch for your specific component (e.g., thinkphp).
      git checkout -b thinkphp
    5. Add the Rule: Copy an existing fingerprint rule file, rename it to match your component name, and update the name field inside the YAML file.
    6. Commit and Push:
      git add <your_new_file_name>
git commit -m "Add <component_name> or description"
git push origin thinkphp
    7. Open Pull Request: Go to your fork on GitHub and submit a Pull Request to the upstream repository for review.

  6. Verify YAML fingerprint rules with observer_ward

    main

    To validate whether a written YAML fingerprint rule is effective, use the observer_ward binary with the following flags:

    • -p <path>: Specify the path to the YAML file you want to verify.
    • -t <target>: Specify the target URL to perform the fingerprint verification against.
    • --debug: Enable debug output to see detailed request and response information, including headers and bodies.

    This is useful for debugging probe logic, matchers, and extractors during rule development.

    ./observer_ward -t http://httpbin.org -p observer_ward/examples/json.yaml --debug

  7. Generate fingerprints from Cyber Search Engine (CSE) queries

    main

    The cse_to_finger function automates the creation of fingerprint rules from existing plugin templates that contain CSE queries.

    It iterates through the plugins directory, extracts queries for engines like Google, Fofa, Hunter, Shodan, and Zoomeye, and compiles them into a single Template object. This new template is then saved in the web-fingerprint/{vendor}/{product}.yaml format, allowing search engine queries to be used as active fingerprinting matchers.

  8. Format and normalize fingerprint YAML files

    main

    The format function performs a bulk update on all fingerprint files in the web-fingerprint directory to ensure consistency:

    1. Template Normalization: Converts word matchers to lowercase and enables case_insensitive matching. Converts favicon part matchers to Body matchers with favicon hashes. Normalizes HTTP methods to uppercase.
    2. Metadata Update: Automatically populates vendor, product, and verified fields in the info.metadata section based on the file's directory structure.
    3. File Organization: Calls rename_fingerprint_yaml to move files from 00_unknown to their correct {vendor}/{product}.yaml locations based on their internal metadata.

  9. Migrate V3 fingerprints to V4 Template format

    main

    Use the v3_to_v4 function to convert legacy V3WebFingerPrint files into the modern Template format.

    Input: A path to a directory containing V3 YAML files. Output: The converted V4 templates are placed in the web-fingerprint/00_unknown directory, provided the product name does not already exist in the current fingerprint set.

  10. Run FingerprintHub CLI via HelperConfig

    main

    The FingerprintHub CLI is controlled by a HelperConfig object. Depending on the configuration settings, the application performs various maintenance and conversion tasks.

    Available operations:

    • JSON Conversion: If convert is enabled, the tool converts YAML fingerprint files in web-fingerprint and service-fingerprint directories into web_fingerprint_v4.json and service_fingerprint_v4.json respectively.
    • Syncing & CSE Generation: If sync is enabled, it synchronizes Nuclei templates and automatically generates fingerprint rules from Cyber Search Engine (CSE) queries (e.g., Shodan, Fofa, Zoomeye).
    • Formatting & Renaming: If format is enabled, it updates template structures (normalizing case, updating metadata) and renames/moves fingerprint YAML files to their correct vendor/product directory structure.
    • Service Scanning: If service is enabled, it runs nmap related tasks.
    • V3 to V4 Migration: If a path is provided to v3_to_v4, it converts legacy V3 fingerprint files to the V4 Template format.

  11. Convert Nuclei templates to Fingerprint rules

    main

    The sync_nuclei function scans specific Nuclei template directories (cnvd, cves, default-logins, vulnerabilities) and imports them into the plugins directory.

    It prioritizes templates containing vpf (Vendor, Product, Framework) information. If a template is identified as a WordPress framework component, it is routed to a specific wordpress/wordpress directory. Otherwise, templates are categorized into the plugins directory based on their tags.

  12. Report fingerprint issues and feedback

    main

    If you encounter issues with the fingerprint library, you can submit an issue on GitHub. Common issues include:

    • False Positives: When a fingerprint is not precise enough and identifies the wrong component.
    • Duplicate Components: When multiple component names refer to the same underlying technology.
    • Missing Fingerprints: When a component cannot be identified because no rule covers it.

    When reporting, please provide a demonstration URL. If a live URL is not available, providing the HTML source code of the target page is a helpful alternative for manual rule correction.