theHarvester OSINT Tool

theHarvester uses different types of modules to gather OSINT (Open Source Intelligence) data such as names, emails, IPs, subdomains, and URLs:

• Passive modules: These gather data from public resources without directly interacting with the target infrastructure (e.g., search engines like baidu, brave, duckduckgo, or certificate transparency logs like crtsh).
• Active modules: These interact more directly with the target, such as DNS brute force (dictionary enumeration) and Screenshots (capturing subdomains).
• API-based modules: Many passive modules (e.g., censys, shodan, virustotal) require an API key to function. Some offer free tiers, while others require paid subscriptions.

To install and run theHarvester, ensure you have Python 3.12 or higher installed. The recommended installation method uses uv for dependency management.

1. Install uv:

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Clone the repository:

   git clone https://github.com/laramies/theHarvester
   cd theHarvester

3. Install dependencies and create a virtual environment:

   uv sync

4. Run the tool:

   uv run theHarvester

The theHarvester REST API includes optional /additional/* routes that are protected. To enable access to these routes, you must:

1. Set the THEHARVESTER_API_KEY environment variable.
2. Pass the same key value in the X-API-Key header of your requests.

If the key is not configured, these routes will return a 503 status code.

theHarvester can be executed as a Python module. It is used to gather open source intelligence (OSINT) on a company or domain.

Basic usage pattern: python -m theHarvester -d <domain> [options]

python -m theHarvester -d example.com

The --dns-resolve flag allows you to specify custom DNS resolvers. It supports three formats:

1. A single IP address (e.g., 8.8.8.8).
2. A comma-separated list of IP addresses (e.g., 1.1.1.1,8.8.8.8).
3. A path to a file containing a list of resolvers (one per line).

# Using a comma-separated list
python -m theHarvester -d example.com -r "1.1.1.1,8.8.8.8"

# Using a file containing resolvers
python -m theHarvester -d example.com -r resolvers.txt

The API uses slowapi for rate limiting. The rate limit for the /sources, /dnsbrute, and /query endpoints is controlled by the API_RATE_LIMIT environment variable.

Default Value: 5/minute

Example of setting the environment variable before running the API:

export API_RATE_LIMIT='10/minute'

TheHarvester uses an api-keys.yaml file to store credentials for various search engines. The engine searches for this file in the following order:

1. ~/.theHarvester/api-keys.yaml
2. /etc/theHarvester/api-keys.yaml
3. /usr/local/etc/theHarvester/api-keys.yaml

If the file is not found in these locations, the engine will attempt to create a default version in your home directory (~/.theHarvester/api-keys.yaml) using a template from the project's data directory.

YAML Structure: The file must contain an apikeys top-level key. Some providers require a single key, while others require multiple fields (e.g., id and secret for Censys, or key and email for Fofa).

apikeys:
  shodan:
    key: 'YOUR_SHODAN_KEY'
  censys:
    id: 'YOUR_CENSYS_ID'
    secret: 'YOUR_CENSYS_SECRET'
  fofa:
    key: 'YOUR_FOFA_KEY'
    email: 'YOUR_FOFA_EMAIL'

You can configure HTTP and SOCKS5 proxies by creating a proxies.yaml file in one of the standard configuration directories:

• ~/.theHarvester/proxies.yaml
• /etc/theHarvester/proxies.yaml
• /usr/local/etc/theHarvester/proxies.yaml

The AsyncFetcher class uses this file to resolve proxy URLs. It supports both http and socks5 types.

http:
  - '1.2.3.4:8080'
  - '5.6.7.8:3128'
socks5:
  - '1.2.3.4:1080'

TheHarvester stores its SQLite database (stash.sqlite) in the following directory by default:

~/.local/share/theHarvester/

This directory is automatically created if it does not exist.