DOCSEARCHbyAlgolia
Submit Docs

darkdump

repository·main·Indexed 23 days ago

https://github.com/josh0xa/darkdump

An open-source OSINT tool for deep web investigation that fetches results from multiple dark web search engines, including ahmia, notevil, tordex, tor66, onionland, and excavator. It features a command-line interface (darkdump-cli) and a Flask-based web interface for streaming results, performing breach intelligence scans, and deep scraping sites for metadata, emails, and documents. Supports routing requests through Tor and exporting results to JSON, CSV, and TXT formats.

Tokens
2.2K
Snippets
6
Records
12
Agent score
32%

What's inside darkdump

  1. Install darkdump manually

    main

    If you prefer a manual installation, clone the repository and install the Python requirements for both the core tool and the web interface separately.

    git clone https://github.com/josh0xA/darkdump
cd darkdump
pip3 install -r requirements.txt
pip3 install -r darkdump-web/requirements.txt

  2. Configure Tor for darkdump

    main

    To use engines that require Tor or to route requests through the Tor proxy (-p), you must configure Tor to allow control via the Tor control port.

    1. Install Tor

    • Debian/Kali/Ubuntu: sudo apt install tor
    • MacOS: brew install tor

    2. Configure torrc

    Locate your torrc file (typically /etc/tor/torrc on Linux). Add the following lines:

    ControlPort 9051
HashedControlPassword [YourHashedPasswordHere]

    To generate the [YourHashedPasswordHere], run:

    tor --hash-password "my_password"

    3. Start Tor Service

    • Linux: sudo systemctl start tor.service
    • MacOS: brew services start tor

  3. Install darkdump via Quick Install (Linux & macOS)

    main

    To quickly install darkdump and its dependencies, use the provided install.sh script. This will create two executable commands: darkdump (for the web interface) and darkdump-cli (for the command-line tool).

    git clone https://github.com/josh0xA/darkdump
cd darkdump
chmod +x install.sh
./install.sh
darkdump

    Note: ./install.sh may require sudo on macOS.

  4. Launch the Darkdump Web Interface

    main

    The web interface provides a browser-based UI for live-streaming results, toggling Tor, and exporting data.

    If installed via install.sh, run:

    darkdump

    Otherwise, run the app directly from the repository:

    python3 darkdump-web/app.py

    Once running, access the interface at http://127.0.0.1:50001 in your browser.

  5. Run the Darkdump Web Interface

    main

    To start the Flask-based web interface, navigate to the darkdump project root and execute the app.py script located within the darkdump_webapp directory. Once running, the interface is accessible via a web browser at http://127.0.0.1:50001.

    Note: The application expects the darkdump_webapp/ folder to be located inside the main darkdump project folder so it can correctly locate the darkdump.py core module.

    python darkdump_webapp/app.py

  6. Examples of darkdump-cli usage

    main

    Common usage patterns for the darkdump-cli tool:

    Basic search via Ahmia (no Tor required):

    darkdump-cli -q "privacy tools" -a 10

    Search and deep scrape each result via Tor:

    darkdump-cli -q "hacking" -a 10 -s -p

    Search, scrape, and collect images:

    darkdump-cli -q "marketplaces" -a 15 -s -p -i

    Use Not Evil engine, deduplicate, and save to JSON:

    darkdump-cli -q "security research" -a 20 -e notevil -p -u -o results.json

    Use OnionLand engine and save to CSV:

    darkdump-cli -q "crypto" -a 10 -e onionland -p -o results.csv

    Breach intelligence scan for an email address:

    darkdump-cli --breach -q admin@example.com -e ahmia -p

    Breach scan with deep scraping:

    darkdump-cli --breach --breach-deep -q example.com -e ahmia
    # Basic search via Ahmia (no Tor required)
darkdump-cli -q "privacy tools" -a 10

  7. Reference: darkdump search engines

    main

    Darkdump supports six engines. All results are checked against Ahmia's blacklist. Unfiltered engines will prompt for confirmation before execution.

    EngineFilteredRequires TorNotes
    ahmiaYesNoDefault. Tor Project-endorsed, strict filtering
    notevilPartialYesAhmia fork with broader index
    tordexNoYesFully uncensored — confirmation required
    tor66NoYesCrawled index with directory — confirmation required
    onionlandNoYesIndexes Tor, I2P, and clearnet — confirmation required
    excavatorNoYesGeneral dark web index — confirmation required

  8. Reference: darkdump-cli flags

    main

    The darkdump-cli tool provides several flags for controlling search behavior, scraping, and output formats.

    FlagDescription
    -q, --querySearch query
    -a, --amountNumber of results to retrieve (default: 10)
    -e, --engineEngine to use (default: ahmia)
    -p, --proxyRoute requests through Tor
    -s, --scrapeDeep scrape each result for metadata, links, emails, documents
    -i, --imagesAlso collect images during scrape (requires -s)
    -u, --uniqueHide results with duplicate title + description
    -d, --debugEnable debug output
    -o FILE, --output FILESave results to file — format inferred from extension (.json, .csv, .txt)
    --breachRun a breach / credential leak intelligence scan for the given target
    --breach-deepCombine breach scan with deep scraping of each result
    --breach-delay SECONDSSeconds between breach queries to avoid rate limits (default: 1.5)
    -v, --versionPrint version

  9. Check Tor Connectivity Status

    main

    The /tor-status endpoint provides a quick check of the current Tor connectivity through the configured proxy. It returns a JSON object indicating if the connection is successfully routing through Tor and the current IP address.

    Response Format:

    {
  "tor": true,
  "ip": "1.2.3.4"
}

  10. Perform Breach Intelligence Scans

    main

    The /breach endpoint provides an SSE stream for scanning for data breaches related to a specific target (email, domain, or keyword). It automatically generates multiple search queries based on the target type to maximize coverage.

    Query Parameters:

    • q (string): The target (email address, domain name, or keyword).
    • engine (string): The engine key. Defaults to ahmia.
    • amount (int): Number of results per query (max 50). Defaults to 10.
    • proxy (string): Set to 1 to use Tor/SOCKS5 proxy.

    SSE Event Types:

    • breach_start: Emitted with target details and query count.
    • breach_query: Emitted for each generated query being processed.
    • result: Emitted for each result found. Includes:
      • severity: Classified as CRITICAL, HIGH, MEDIUM, or INFO based on keywords.
      • category: Classified as paste-site, market, forum, leak-index, or other.
      • artefacts: Extracted emails, hashes, and password_hints from the result text.
      • scrape_data: Deep scrape data if available.
    • error: Emitted if a query fails.
    • done: Emitted when the scan is finished.

  11. Export Search Results

    main

    The /export endpoint allows downloading search results in multiple formats via a POST request. The request must include a JSON body.

    Request Body JSON Schema:

    • format (string): json, csv, or txt (default).
    • results (array): The list of result objects to export.
    • query (string): The original search query.
    • engine (string): The engine used.

    Supported Formats:

    • json: Returns a structured JSON file.
    • csv: Returns a CSV file with columns: idx, title, url, description.
    • txt: Returns a human-readable plain text file including scrape data (links, emails, documents, etc.) if present.

  12. Search Dark Web via Web Interface

    main

    The /search endpoint provides a Server-Sent Events (SSE) stream for real-time search results. It allows querying various dark web search engines with options for proxy usage, deduplication, and deep scraping of results.

    Query Parameters:

    • q (string): The search query.
    • engine (string): The engine key (e.g., ahmia, notevil, tordex, tor66, onionland, excavator). Defaults to ahmia.
    • amount (int): Number of results to fetch (max 50). Defaults to 10.
    • proxy (string): Set to 1 to use Tor/SOCKS5 proxy.
    • dedupe (string): Set to 1 to enable URL and metadata deduplication.
    • scrape (string): Set to 1 to perform deep scraping of each result.
    • images (string): Set to 1 to include images in the scrape data (used if scrape is also 1).

    SSE Event Types:

    • tor_ok: Emitted if Tor is successfully connected when proxy=1 is used.
    • count: Emitted with the total number of results found.
    • result: Emitted for each individual result found. If scrape=1, includes scrape_data containing metadata, links, emails, documents, and an images gallery.
    • error: Emitted if a search error occurs.
    • done: Emitted when the search process completes.