httpx

Overview

httpx is a fast, multi-purpose HTTP toolkit designed for reliability and high concurrency. It uses the retryablehttp library to handle retries and backoffs, which helps in managing edge cases like WAF (Web Application Firewall) interference.

Key Features

• Modular Design: Easy to extend and contribute to.
• Configurable Probing: Highly configurable flags for probing multiple HTTP elements.
• Smart Fallback: Automatically falls back from https to http by default.
• Flexible Input: Supports hosts, URLs, and CIDR ranges as input.
• Reliability: Built-in handling for retries and backoffs to maintain result accuracy even with high thread counts.

Security Note

httpx is primarily designed as a standalone CLI tool. Running it as a service may pose security risks and should be done with caution and additional security measures.

To install the httpx CLI tool, ensure you have go >=1.25.0 installed on your system. You can install the latest version directly from the repository using the following command:

go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

Use the -sf flag to provide a YAML file containing authentication credentials for specific domains. Supported authentication types include BasicAuth, BearerToken, Header, Cookie, and Query.

id: example-auth
info:
  name: Example Auth Config
static:
  - type: Header
    domains:
      - api.example.com
    headers:
      - key: X-API-Key
        value: secret-key-here
  - type: BasicAuth
    domains-regex:
      - ".*\\.internal\\.com$"
    username: admin
    password: secret

The httpx CLI supports graceful shutdowns.

• First Ctrl+C: Sends an interrupt signal to the runner, allowing in-flight requests to finish.
• Second Ctrl+C: Forces an immediate exit.

If the process is interrupted and the option to save a resume configuration is enabled, httpx will create a resume file (using the runner.DefaultResumeFile path) to allow continuing the scan later.

You can upload your httpx results to the ProjectDiscovery UI dashboard for visualization. This is enabled when using the -asset_upload flag or by providing asset metadata. Results can be viewed at https://cloud.projectdiscovery.io/assets upon completion.

To associate results with specific assets or teams, use the following flags:

• -asset_id: Specify the Asset ID.
• -asset_name: Specify the Asset Group Name.
• -team_id: Specify the Team ID.

Use the -path flag to check standard RFC 9116 paths and the -mdc flag with DSL helpers like contains(content_type, ...) and contains(body, ...) to validate the file content and type.

echo target.com | httpx -path '/.well-known/security.txt,/security.txt' -mc 200 -mdc 'contains(content_type, "text/plain") && contains(body, "Contact:") && contains_any(body, "mailto:", "https://")'

You can configure httpx to output results directly to a database. This is enabled via the -result_database flag.

Configuration Methods:

1. Via Config File: Provide a path to a configuration file using the -result_database_config flag.
2. Via CLI Flags: Provide individual database parameters directly in the command line.

Available Database CLI Options:

• -result_database: Enable database output.
• -result_database_type: The type of database.
• -result_database_conn_str: The connection string for the database.
• -result_database_name: The name of the database.
• -result_database_table: The name of the table to store results.
• -result_database_batch_size: The batch size for writes.
• -result_database_omit_raw: Whether to omit raw data.

You can define custom schemes for specific ports and use custom resolvers with specific protocols.

• Custom Port Schemes: Use -ports with the format scheme:port (e.g., -ports http:443,http:80,https:8443).
• Custom Resolvers: Supports doh|tcp|udp in the format protocol:resolver:port (e.g., udp:127.0.0.1:53).