repository·master·Indexed 25 days ago
https://github.com/devanshbatham/paramspiderA tool designed to mine URLs from Wayback Machine archives for a given domain or list of domains. It is optimized for bug hunting and fuzzing by filtering out non-essential URLs and replacing parameter values with a placeholder (defaulting to "FUZZ").
Use the paramspider CLI to fetch URLs related to a domain or a list of domains from Wayback archives. The tool filters out non-essential URLs to focus on parameters useful for bug hunting, fuzzing, or probing.
paramspider -d example.comTo install paramspider from the source repository, clone the repository, navigate into the directory, and use pip to install the package locally.
git clone https://github.com/devanshbatham/paramspider
cd paramspider
pip install .Common usage patterns for paramspider include:
Discover URLs for a single domain:
paramspider -d example.comDiscover URLs for multiple domains from a file:
paramspider -l domains.txtStream URLs to the terminal:
paramspider -d example.com -sUse a web request proxy:
paramspider -d example.com --proxy '127.0.0.1:7890'Set a custom parameter placeholder:
paramspider -d example.com -p '"><h1>reflection</h1>'The following flags are available for the paramspider command-line interface:
-d <domain>: Specify a single domain to discover URLs for.-l <file>: Specify a file containing a list of domains to discover URLs for.-s: Stream URLs directly to the terminal.--proxy <proxy_url>: Set up a web request proxy (e.g., '127.0.0.1:7890').-p <placeholder>: Add a custom placeholder for URL parameter values (defaults to "FUZZ").has_extension function checks if a given URL's path ends with any of the extensions provided in the extensions list.fetch_and_clean_urls function is the core logic for retrieving URLs from the Wayback Machine CDX API, cleaning them, and saving the results. It filters out URLs with specific file extensions and replaces query parameter values with a provided placeholder.clean_url function removes redundant port information from HTTP and HTTPS URLs (e.g., removing :80 from http://domain:80/path).Use fetch_url_content(url, proxy) to retrieve the content of a specified URL. The function automatically selects a random User-Agent from a predefined list to mimic different browsers and implements a retry mechanism (up to MAX_RETRIES) if the request fails.
If a proxy string is provided, it is automatically configured for both http and https protocols. If the function fails after all retries, it logs an error and exits the process.
The clean_urls function processes a list of URLs by:
clean_url.extensions list.placeholder string.FUZZ) to facilitate fuzzing. Results are saved to the results/ directory as <domain>.txt files.load_user_agents() function returns a list of hardcoded User-Agent strings representing various browsers and operating systems (e.g., Chrome on Windows, Firefox on Mac, etc.). This is used internally by fetch_url_content to rotate headers.