URLFinder

URLFinder requires Go 1.21. You can install it directly using go install or download pre-compiled binaries from the GitHub releases page.

go install -v github.com/projectdiscovery/urlfinder/cmd/urlfinder@latest

You can refine your results using pattern matching to include or exclude specific URLs.

• Match (-m): Include only URLs that match the provided patterns (comma-separated or from a file).
• Filter (-f): Exclude URLs that match the provided patterns (comma-separated or from a file).

Patterns can be provided as a comma-separated list or by pointing to a text file containing the patterns.

# Include URLs containing 'shop' or 'model'
urlfinder -d tesla.com -m shop,model

# Exclude URLs containing 'privacy' or 'terms'
urlfinder -d tesla.com -f privacy,terms

# Combine both match and filter
urlfinder -d tesla.com -m support -f faq

# Use patterns from files
urlfinder -d tesla.com -m include-patterns.txt -f exclude-patterns.txt

To perform basic passive URL discovery for a specific domain, use the -d flag followed by the target domain.

urlfinder -d tesla.com

Use the -j or -jsonl flag to output results in JSON Lines format. This is ideal for structured data processing. Each line in the output is a valid JSON object containing the url, the original input domain, and the source used for discovery.

urlfinder -d tesla.com -j

Use the New() function to create a new Resolver instance. By default, the Resolvers slice is initialized as empty, allowing you to provide your own list of DNS resolver addresses.

import "github.com/projectdiscovery/projectdiscovery/urlfinder/pkg/resolve"

resolver := resolve.New()

The DefaultResolvers variable provides a pre-defined list of reliable public DNS resolver addresses (including Cloudflare, Google, Quad9, Yandex, and OpenDNS) that can be used to populate a Resolver instance.

var DefaultResolvers = []string{
	"1.1.1.1:53",        // Cloudflare primary
	"1.0.0.1:53",        // Cloudflare secondary
	"8.8.8.8:53",        // Google primary
	"8.8.4.4:53",        // Google secondary
	"9.9.9.9:53",        // Quad9 Primary
	"9.9.9.10:53",       // Quad9 Secondary
	"77.88.8.8:53",      // Yandex Primary
	"77.88.8.1:53",      // Yandex Secondary
	"208.67.222.222:53", // OpenDNS Primary
	"208.67.220.220:53", // OpenDNS Secondary
}

URLFinder is a command-line tool designed to find URLs. The execution flow involves parsing command-line flags and configuration files via runner.ParseOptions(), initializing a new runner instance with runner.NewRunner(options), and executing the enumeration process with newRunner.RunEnumeration().

Note: The tool attempts to automatically increase OS file descriptors via the autofdmax package to handle high concurrency, failing silently if it cannot do so.

// Conceptual execution flow of the URLFinder CLI
options := runner.ParseOptions()
newRunner, err := runner.NewRunner(options)
if err != nil {
	gologger.Fatal().Msgf("Could not create runner: %s\n", err)
}

err = newRunner.RunEnumeration()
if err != nil {
	gologger.Fatal().Msgf("Could not run enumeration: %s\n", err)
}

The Resolver struct is used to perform DNS name resolution. It contains:

• DNSClient: A pointer to a dnsx.DNSX instance used for the actual resolution logic.
• Resolvers: A slice of strings representing the DNS resolver addresses (e.g., "8.8.8.8:53") to be used during the process.

type Resolver struct {
	DNSClient *dnsx.DNSX
	Resolvers []string
}