crawler4j Documentation

To run a crawl, implement a controller that sets up the CrawlConfig, PageFetcher, RobotstxtServer, and CrawlController.

Steps:

1. Create a CrawlConfig and set the crawlStorageFolder.
2. Instantiate PageFetcher, RobotstxtConfig, and RobotstxtServer.
3. Initialize CrawlController with the config, fetcher, and server.
4. Add seed URLs using controller.addSeed(url).
5. Provide a WebCrawlerFactory (e.g., using a method reference to your crawler constructor).
6. Call controller.start(factory, numberOfCrawlers) to begin the blocking crawl operation.

public class Controller {
    public static void main(String[] args) throws Exception {
        String crawlStorageFolder = "/data/crawl/root";
        int numberOfCrawlers = 7;

        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder(crawlStorageFolder);

        // Instantiate the controller for this crawl.
        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        // Add seed urls.
        controller.addSeed("https://www.ics.uci.edu/~lopes/");
        controller.addSeed("https://www.ics.uci.edu/~welling/");
    	controller.addSeed("https://www.ics.uci.edu/");
    	
    	// The factory which creates instances of crawlers.
        CrawlController.WebCrawlerFactory<BasicCrawler> factory = MyCrawler::new;
        
        // Start the crawl.
        controller.start(factory, numberOfCrawlers);
    }
}

To define crawling logic, extend the WebCrawler class and override two primary methods:

1. shouldVisit(Page referringPage, WebURL url): Returns true if the URL should be crawled. Use this to implement filters (e.g., file extensions or domain restrictions).
2. visit(Page page): Called after a page is successfully downloaded. Use this to process the content. You can access the URL, text, HTML, and outgoing links via the Page object.

If the page is HTML, you can cast page.getParseData() to HtmlParseData to access getText(), getHtml(), and getOutgoingUrls().

public class MyCrawler extends WebCrawler {

    private final static Pattern FILTERS = Pattern.compile(".*(\.(css|js|gif|jpg"
                                                           + "|png|mp3|mp4|zip|gz))$");

     @Override
     public boolean shouldVisit(Page referringPage, WebURL url) {
         String href = url.getURL().toLowerCase();
         return !FILTERS.matcher(href).matches()
                && href.startsWith("https://www.ics.uci.edu/");
     }

     @Override
     public void visit(Page page) {
         String url = page.getWebURL().getURL();
         System.out.println("URL: " + url);

         if (page.getParseData() instanceof HtmlParseData) {
             HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
             String text = htmlParseData.getText();
             String html = htmlParseData.getHtml();
             Set<WebURL> links = htmlParseData.getOutgoingUrls();

             System.out.println("Text length: " + text.length());
             System.out.println("Html length: " + html.length());
             System.out.println("Number of outgoing links: " + links.size());
         }
    }
}

Use the CrawlConfig class to tune the crawler's behavior. Key configuration methods include: