DOCSEARCHbyAlgolia
Submit Docs

dirsearch

repository·master·Indexed 12 days ago

https://github.com/maurosoria/dirsearch

An advanced web path brute-forcer and scanner used for web path discovery to find hidden directories and files on web servers. It features a Python API for automation and an optional high-performance native backend written in Rust using PyO3, reqwest, and tokio.

Tokens
23.4K
Snippets
56
Records
85
Agent score
93%

What's inside dirsearch

  1. Access dirsearch documentation

    master

    The dirsearch documentation is organized into several specialized guides for different user needs. For end-users, the primary documentation areas are:

    • Installation: Supported platforms, Python setup, release artifacts, and Docker usage.
    • Usage Guide: Command examples, recursion, filtering, proxies, raw requests, reports, and practical tips.
    • Wordlists: Managing extensions, bundled categories, templates, prefixes, suffixes, and transformations.
    • CLI Options: A complete reference of all command-line flags.
    • Configuration: Details on config.ini behavior and settings.
    • Sessions: How to save, list, and resume interrupted scans.
    • Python API: Instructions for integrating dirsearch into Python automation scripts.

  3. How sessions work in dirsearch

    master
    dirsearch supports saving and resuming scan sessions, which allows you to pause a long-running scan and continue it later. Sessions are stored in a directory-based JSON structure. When you resume a session, dirsearch appends new results to the existing output history with timestamps, ensuring you can review results from both the original and the resumed scan.

  4. Configure Threads and Runtime Modes

    master

    Control the concurrency of your scan using threads or asynchronous mode.

    • Threads: Use -t or --threads to set the number of brute-force processes. The default is 25. High thread counts increase speed but risk Denial of Service (DoS).
    • Asynchronous Mode: The default on Python 3.11+. Uses coroutines for better performance and lower CPU usage. CTRL+C pauses progress immediately.
    • Synchronous Mode: Use --sync to force the synchronous Python stack.
    • Native Backend: Use --request-backend native to use the native request backend. Note that the native backend runs without async mode unless --async is explicitly supplied (though --async is rejected when using the native backend because it has its own scheduler).
    # Set thread count to 20
python3 dirsearch.py -e php,htm,js,bak,zip,tgz,txt -u https://target -t 20

# Force synchronous mode
python3 dirsearch.py --sync -u https://target

  5. Understand Python API limitations compared to the CLI

    master

    The Python API is designed for stable integrations, but it does not currently offer full parity with the dirsearch CLI. The following features are CLI-only and are not available via the public Python API:

    • Recursive scanning and session resume files
    • Async mode and the native request backend
    • CLI report writers
    • CLI wildcard calibration options and advanced response filters
    • CLI wordlist transformations (e.g., force extensions, overwrite extensions, prefixes, suffixes, and casing transforms)

    Integration Best Practice: For stable integrations, use only the public imports documented in the API guide. Do not import lib.* internals, as these are not guaranteed to be stable across different versions/commits.

  6. Understand dirsearch release formats

    master

    dirsearch provides two main downloadable formats for different use cases:

    1. PyInstaller single-file binaries: Convenient single executables. Note that some antivirus engines may heuristically flag PyInstaller bootloaders.
    2. Portable folders: Bundled archives containing CPython, Python dependencies, optional DB drivers, and the Rust native module. Use these if the single-file PyInstaller binary is blocked by antivirus software.

    Each non-Docker target is available in three runtime variants:

    • async: Uses async = True and request-backend = python. This is the recommended default.
    • threaded: Uses async = False and request-backend = python. This is the legacy threaded Python runtime.
    • native-rust: Uses async = False, request-backend = native, and wordlist-backend = native. This utilizes the Rust request and wordlist backend for performance.

  7. How wordlist extension replacement works

    master

    dirsearch handles extensions differently depending on how the wordlist entries are formatted and which flags are used:

    1. Standard Replacement: If a wordlist entry contains the %EXT% keyword, dirsearch replaces it with the extensions provided via the -e flag.

      • Example: index.%EXT% with extensions asp,aspx becomes index, index.asp, and index.aspx.

    2. Forced Extensions: For wordlists without %EXT% (like SecLists), use -f or --force-extensions. This appends the selected extensions and a trailing / to every entry.

      • Example: admin with extensions php,html becomes admin, admin.php, admin.html, and admin/.

    3. Overwriting Extensions: To apply extensions to entries that already have an extension, use --overwrite-extensions. Note that certain extensions like .log, .json, .xml, and media types (.jpg, .png) are excluded from this behavior.

      • Example: login.html with extensions jsp,jspa becomes login.html, login.jsp, and login.jspa.

  8. Configure HTTP concurrency and runtime workers

    master

    When using the native backend, it is important to distinguish between Runtime workers (CPU-bound) and HTTP in-flight concurrency (I/O-bound). To achieve optimal performance:

    1. Runtime Workers: These should be tuned to track the available CPU cores on your system.
    2. HTTP Concurrency: This should typically be set higher than the CPU count to account for the I/O-bound nature of web scanning.

    A recommended practical default for native HTTP concurrency is calculated as: min(max(cpu_count * 8, 12), 128)

    You can override these values using explicit CLI flags to fine-tune the balance between CPU utilization and network throughput.

  9. Optimize performance using the Rust native backend

    master

    The experimental Rust native request backend provides significantly higher throughput (Requests Per Second - RPS) compared to the standard Python backend. When choosing between backends, consider the following performance characteristics observed in benchmarks:

    • Scaling: The Rust backend scales more effectively with CPU count. As vCPU count increases, the performance gap between Rust and Python widens (e.g., from ~2.3x to ~2.8x in multi-process scans).
    • System Overhead: The Rust backend results in significantly fewer context switches compared to Python, reducing scheduler contention.
    • Full Scan vs. Direct HTTP: While direct HTTP client benchmarks show massive gains (up to 17x), the actual full-scan throughput gains are more modest (typically 1.6x to 2.3x) because full scans include overhead from fuzzers, callbacks, filters, and process management.

    For the best performance, use the Rust backend on high-CPU environments to minimize scheduler contention.

  10. Basic Usage and Target Specification

    master

    To run dirsearch, you must specify a target URL. You can provide a single URL, a list of URLs from a file, or use CIDR notation for network ranges. You can also pipe URLs via STDIN or load targets from an nmap report.

    Mandatory Target Options:

    • -u URL, --url=URL: The target URL(s). This flag can be used multiple times.
    • -l PATH, --urls-file=PATH: Path to a file containing a list of URLs.
    • --stdin: Read URLs from standard input.
    • --cidr=CIDR: Target a specific CIDR range.
    • --nmap-report=PATH: Load targets from an nmap report (requires -sV flag in the original nmap scan for best results).
    • --raw=PATH: Load a raw HTTP request from a file (use --scheme to set the scheme).
    dirsearch.py -u https://example.com -e php,asp

  11. Exclude specific extensions from wordlists

    master

    To remove all paths from your wordlist that contain a specific extension, use the --exclude-extensions flag.

    python3 dirsearch.py -u https://target --exclude-extensions jsp

    If the wordlist contains admin.php and test.jsp, only admin.php will remain after exclusion.

  12. Install the dirsearch native backend

    master

    The native backend is an experimental Phase 5 backend written in Rust. It is opt-in for source installs and included in native-rust release artifacts.

    To build the native engine from an existing dirsearch installation, ensure you have Python 3.14, Rust/Cargo, Python development headers, and a C compiler installed, then run:

    dirsearch-build-native

    If you are developing from a source checkout, use the repository helper to build wheels:

    python3.14 scripts/build_native.py --out dist/native-wheels

    Both helpers will install the built wheel and verify the installation via import dirsearch_native.