SpiderFoot OSINT Automation Tool

SpiderFoot utilizes over 200 modules that operate on a publisher/subscriber model to maximize data extraction. Modules are categorized by their source type:

• Internal Modules: Perform tasks like Base64 decoding, email extraction, port scanning, or web spidering without external dependencies.
• API Modules: Connect to external services. These are further divided into:
  • Free API: Services that do not require a paid subscription (e.g., Abuse.ch, AlienVault, GitHub).
  • Tiered API: Services with free tiers and paid options (e.g., Shodan, Hunter.io, VirusTotal).
  • Commercial API: Services requiring a paid subscription (e.g., Dehashed, Seon).
• Tools: Integrations with external command-line tools (e.g., Nmap, DNSTwist, Wappalyzer).

Correlation rules are used to automate the analysis of OSINT data collected during a scan. Instead of collecting new data from targets, they analyze existing scan results to identify patterns, outliers, or specific risks (e.g., finding software versions in open port banners).

SpiderFoot translates YAML-defined rules into database queries and Python logic. The results are stored in the SpiderFoot database and can be viewed via the web interface, the CLI, or queried directly from the SQLite database using the tbl_scan_correlation_results and tbl_scan_correlation_results_events tables.

When refining data (after the first method in a collect block), you can access related data using prefixes. These prefixes always refer back to the first method block in the current collect block.

• source.: Refers to the data element that generated the current data element.
• child.: Refers to data resulting from the current data element.
• entity.: Refers to the primary entity (e.g., an IP or Domain) associated with the data path.

Example Data Path: bar [INTERNET_NAME] $\rightarrow$ https://bar/page.html [LINKED_URL_INTERNAL] $\rightarrow$ Web Content [TARGET_WEB_CONTENT]

For the TARGET_WEB_CONTENT element:

• data: Web Content
• source.data: https://bar/page.html
• child.data: (Depends on what the content contains)
• entity.data: bar (Because INTERNET_NAME is an entity, but LINKED_URL_INTERNAL is not).

SpiderFoot can perform reconnaissance and data extraction on a variety of target entities. When starting a scan, you can provide one of the following:

• IP address
• Domain/sub-domain name
• Hostname
• Network subnet (CIDR)
• ASN
• E-mail address
• Phone number
• Username
• Person's name
• Bitcoin address

Correlation rules are written in YAML and follow a specific structural pattern consisting of several optional and mandatory sections:

1. Meta: Defines the rule's identity (id, version) and human-readable information (name, description, risk).
2. Collections: Defines the set of data to extract from scan results. It uses collect blocks with method (e.g., exact, regex), field (e.g., type, data), and value.
3. Aggregation (Optional): Groups the collected data into buckets (e.g., by a specific field) to allow for group-based analysis.
4. Analysis (Optional): Performs logic to filter the aggregated data (e.g., checking for duplicates or specific patterns).
5. Headline: Defines the string used to present the result, supporting placeholders like {data} to inject values from the analyzed elements.

SpiderFoot 4.0 features a YAML-configurable correlation engine. This engine allows you to define rules to find relationships between different pieces of data.

To learn how to write custom rules:

1. Review the existing 37 pre-defined rules.
2. Examine template.yaml in the repository for a structural walkthrough.
3. Refer to the comprehensive correlation rule-set documentation in /correlations/README.md.

SpiderFoot requires Python 3.7+ and several Python libraries. You can install it using either a stable packaged release or by cloning the development branch.

Stable Build (Recommended)

Use this for a tested environment. Download the tarball, extract it, install requirements, and run the server.

Development Build

Use this to access the latest features from the master branch by cloning the repository directly.

Both methods use sf.py to launch the embedded web server.

#### Stable build:
```bash
 wget https://github.com/smicallef/spiderfoot/archive/v4.0.tar.gz
 tar zxvf v4.0.tar.gz
 cd spiderfoot-4.0
 pip3 install -r requirements.txt
 python3 ./sf.py -l 127.0.0.1:5001

Development build:

 git clone https://github.com/smicallef/spiderfoot.git
 cd spiderfoot
 pip3 install -r requirements.txt
 python3 ./sf.py -l 127.0.0.1:5001

To create a new correlation rule:

1. Locate the template.yaml file in the /correlations folder of your SpiderFoot installation.
2. Copy template.yaml to a new file. The filename should match the id you define within the YAML (e.g., my_new_rule.yaml).
3. Edit the YAML content to define your meta, collections, aggregation, analysis, and headline sections.
4. Save the file.
5. Restart SpiderFoot to load the new rule.

Note: If there are syntax errors in your YAML, SpiderFoot will abort at startup and report the error.

# Example of creating a rule by copying template.yaml
cp correlations/template.yaml correlations/my_custom_rule.yaml
# Edit my_custom_rule.yaml then restart SpiderFoot

You can deploy Spiderfoot using Docker Compose with different configurations depending on your needs (standard, development, or full feature set).

# Basic usage
docker-compose up

# Dev environment (maps local code directory into the container)
docker-compose -f docker-compose.yml -f docker-compose-dev.yml up

# Full image (includes all CLI tools)
docker-compose -f docker-compose.yml -f docker-compose-full.yml up

Correlation rules are defined in YAML files. Each rule must follow a specific structure consisting of metadata, collection, aggregation, analysis, and headline sections. The rule's id must match the filename.

# Example structure (conceptual)
id: my_rule_name
version: 1
meta:
  name: "Rule Name"
  description: "Rule description"
  risk: "HIGH"
collection:
  - method:
      field: "type"
      value: "INTERNET_NAME"
aggregation:
  field: "data"
analysis:
  - method: "threshold"
    minimum: 1
headline: "Found {entity.data}"