repository·master·Indexed 25 days ago
https://github.com/jaeles-project/gospiderGoSpider is a fast web crawler written in Go designed for reconnaissance workflows. It supports parallel crawling, 3rd party URL retrieval from sources like Archive.org, CommonCrawl.org, VirusTotal.com, and AlienVault.com, and integration with Burp Suite inputs. Key features include configurable recursion depth, concurrency control, custom HTTP headers/cookies, and the ability to filter results by response length or URL regex.
To run GoSpider using Docker, clone the repository, build the container, and then run it.
# Clone the repo
git clone https://github.com/jaeles-project/gospider.git
# Build the container
docker build -t gospider:latest gospider
# Run the container
docker run -t gospider -hYou can install GoSpider directly using go install if you have Go installed on your system.
GO111MODULE=on go install github.com/jaeles-project/gospider@latestCrawl a single target site with specific concurrency and depth settings.
gospider -s "https://google.com/" -o output -c 10 -d 1Pass custom HTTP headers and cookies to the crawler, or load them directly from a Burp Suite raw request.
# Using manual headers and cookies
gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source -H "Accept: */*" -H "Test: test" --cookie "testA=a; testB=b"
# Using Burp raw request file
gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source --burp burp_req.txtgospider -s "https://google.com/" -o output -c 10 -d 1 --other-source -H "Accept: */*" -H "Test: test" --cookie "testA=a; testB=b"
# Using Burp raw request file
gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source --burp burp_req.txtRun 20 sites at the same time, with 10 threads (bots) per site.
gospider -S sites.txt -o output -c 10 -d 1 -t 20Exclude specific file extensions or URL patterns from the crawl using a regex. Note that GoSpider blacklists common media/style extensions by default.
gospider -s "https://google.com/" -o output -c 10 -d 1 --blacklist ".(woff|pdf)"Enable length reporting and filter results based on specific byte lengths.
gospider -s "https://google.com/" -o output -c 10 -d 1 --length --filter-length "6871,24432"Retrieve URLs from Archive.org, CommonCrawl.org, VirusTotal.com, and AlienVault.com using the --other-source flag.
gospider -s "https://google.com/" -o output -c 10 -d 1 --other-sourceCrawl multiple sites provided in a text file.
gospider -S sites.txt -o output -c 10 -d 1GoSpider is a fast web spider with the following CLI flags:
| Flag | Short | Description |
|---|---|---|
--site | -s | Site to crawl |
--sites | -S | Site list to crawl |
--proxy | -p | Proxy (Ex: http://127.0.0.1:8080) |
--output | -o | Output folder |
--user-agent | -u | User Agent to use (web: random web, mobi: random mobile, or custom string) |
--cookie | Cookie to use (e.g., testA=a; testB=b) | |
--header | -H | Header to use (use multiple flags for multiple headers) |
--burp | Load headers and cookie from Burp raw HTTP request | |
--blacklist | Blacklist URL Regex | |
--whitelist | Whitelist URL Regex | |
--whitelist-domain | Whitelist Domain | |
--threads | -t | Number of threads (Run sites in parallel) (default 1) |
--concurrent | -c | Max allowed concurrent requests of matching domains (default 5) |
--depth | -d | MaxDepth limits recursion depth (0 for infinite) (default 1) |
--delay | -k | Delay in seconds before new request (default 0) |
--random-delay | -K | Extra randomized duration added to Delay (seconds) |
--timeout | -m | Request timeout in seconds (default 10) |
--base | -B | Disable all and only use HTML content |
--js | Enable linkfinder in javascript file (default true) | |
--subs | Include subdomains | |
--sitemap | Try to crawl sitemap.xml | |
--robots | Try to crawl robots.txt (default true) | |
--other-source | -a | Find URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com) |
--include-subs | -w | Include subdomains crawled from 3rd party (default: main domain) |
--include-other-source | -r | Also include other-source's urls (still crawl and request) |
--debug | Turn on debug mode | |
--json | Enable JSON output | |
--verbose | -v | Turn on verbose |
--length | -l | Turn on length |
--filter-length | -L | Turn on length filter |
--raw | -R | Turn on raw |
--quiet | -q | Suppress all output and only show URL |
--no-redirect | Disable redirect | |
--version | Check version | |
--help | -h | help for gospider |
Customize the HTTP requests made by the spider:
-p, --proxy: Set a proxy (e.g., http://127.0.0.1:8080).-u, --user-agent: Set the User Agent. Options: web (random web user-agent), mobi (random mobile user-agent), or a custom string.--cookie: Provide cookies to use (e.g., testA=a; testB=b).-H, --header: Set custom headers. Use this flag multiple times to set multiple headers.--burp: Load headers and cookies directly from a Burp Suite raw HTTP request.Limit the scope of the crawl using regex or domain matching:
--blacklist: URL Regex to exclude.--whitelist: URL Regex to include.--whitelist-domain: Domain name to include.-L, --filter-length: Enable length filtering.--no-redirect: Disable following redirects.