WebMagic Java Crawler Framework

The standard way to build a crawler is to implement the PageProcessor interface. In the process(Page page) method, you can:

• Use page.addTargetRequests(links) to add new URLs for the crawler to visit.
• Use page.putField(key, value) to extract and store data.
• Use page.setSkip(true) to skip the current page if certain conditions are met.
• Access HTML content via page.getHtml() using regex or XPath.

To run the crawler, use Spider.create(processor).addUrl(url).thread(n).run().

public class GithubRepoPageProcessor implements PageProcessor {

    private Site site = Site.me().setRetryTimes(3).setSleepTime(1000);

    @Override
    public void process(Page page) {
        // Add new URLs to crawl using regex
        page.addTargetRequests(page.getHtml().links().regex("(https://github\.com/\w+/\w+)").all());
        
        // Extract data using regex or xpath
        page.putField("author", page.getUrl().regex("https://github\.com/(\w+)/.*").toString());
        page.putField("name", page.getHtml().xpath("//h1[@class='public']/strong/a/text()").toString());
        
        // Skip page if data is missing
        if (page.getResultItems().get("name") == null){
            page.setSkip(true);
        }
        
        page.putField("readme", page.getHtml().xpath("//div[@id='readme']/tidyText()"));
    }

    @Override
    public Site getSite() {
        return site;
    }

    public static void main(String[] args) {
        Spider.create(new GithubRepoPageProcessor())
             .addUrl("https://github.com/code4craft")
             .thread(5)
             .run();
    }
}

The PageProcessor interface is part of webmagic-core. To implement custom crawling logic, implement this interface and use the process(Page page) method to define how to extract data and find new URLs.

Key methods within process:

• page.getHtml().links().regex(pattern).all(): Finds new URLs matching a regex pattern.
• page.addTargetRequests(links): Adds discovered URLs to the crawler's queue.
• page.putField(key, value): Saves extracted data into the page object.
• page.getHtml().xpath(expression): Extracts data using XPath.
• page.getHtml().$(selector): Extracts data using CSS selectors.

Use the Spider class to run your crawler.

public class OschinaBlogPageProcessor implements PageProcessor {

    private Site site = Site.me().setDomain("my.oschina.net");

    @Override
    public void process(Page page) {
        // 1. Find new links and add them as targets
        List<String> links = page.getHtml().links().regex("http://my\.oschina\.net/flashsword/blog/\d+").all();
        page.addTargetRequests(links);

        // 2. Extract data and put it into fields
        page.putField("title", page.getHtml().xpath("//div[@class='BlogEntity']/div[@class='BlogTitle']/h1").toString());
        page.putField("content", page.getHtml().$("div.content").toString());
        page.putField("tags", page.getHtml().xpath("//div[@class='BlogTags']/a/text()").all());
    }

    @Override
    public Site getSite() {
        return site;
    }

    public static void main(String[] args) {
        Spider.create(new OschinaBlogPageProcessor())
             .addUrl("http://my.oschina.net/flashsword/blog")
             .addPipeline(new ConsolePipeline())
             .run();
    }
}

For a more declarative approach, use the webmagic-extension module. You can define a POJO and use annotations to map HTML elements to class fields. This is often cleaner and allows the model to be reused in Pipelines.

Key Annotations:

• @TargetUrl(pattern): Defines the URL pattern this crawler targets.
• @ExtractBy(value, type): Maps a field to an HTML element.
  • value: The XPath or CSS selector.
  • type: The extraction type (e.g., ExtractBy.Type.Css).
  • multi = true: Use this for fields that should receive a List<String> (e.g., when extracting multiple elements).

Use OOSpider (or the appropriate extension entry point) to run the annotated crawler.

@TargetUrl("http://my.oschina.net/flashsword/blog/\d+")
public class OschinaBlog {

    @ExtractBy("//title")
    private String title;

    @ExtractBy(value = "div.BlogContent", type = ExtractBy.Type.Css)
    private String content;

    @ExtractBy(value = "//div[@class='BlogTags']/a/text()", multi = true)
    private List<String> tags;

    public static void main(String[] args) {
        OOSpider.create(
            Site.me(),
            new ConsolePageModelPipeline(), 
            OschinaBlog.class
        ).addUrl("http://my.oschina.net/flashsword/blog").run();
    }
}

The webmagic-scripts module allows you to write web crawlers using simple scripts (JavaScript or Ruby) instead of full Java development. You can execute these scripts using the webmagic-console.jar without needing to manage dependencies or write Java code manually.

Installation

Download the console package: http://code4craft.qiniudn.com/webmagic-console.tar.gz

Execution Command

Use the following command structure to run a script:

java -jar -Dfile.encoding='utf-8' webmagic-console.jar -f <script_file> [-l <language>] [-t <threads>] [-s <interval_ms>] <url1> <url2> ...

CLI Flags

java -jar -Dfile.encoding='utf-8' webmagic-console.jar -f github.js -t 2 -s 0 https://github.com/code4craft

For a more declarative approach, you can use annotations on a POJO to define your crawler. This avoids manual field mapping in a process method.

Key annotations:

• @TargetUrl(regex): Defines the URL pattern this class handles.
• @HelpUrl(regex): Defines a help URL pattern.
• @ExtractBy(xpath/css, notNull): Extracts data using XPath or CSS selectors.
• @ExtractByUrl(regex): Extracts data from a URL matching the pattern.

Use OOSpider.create(site, pipeline, targetClass) to initialize the crawler.

@TargetUrl("https://github.com/\w+/\w+")
@HelpUrl("https://github.com/\w+")
public class GithubRepo {

    @ExtractBy(value = "//h1[@class='public']/strong/a/text()", notNull = true)
    private String name;

    @ExtractByUrl("https://github\.com/(\w+)/.*")
    private String author;

    @ExtractBy("//div[@id='readme']/tidyText()")
    private String readme;

    public static void main(String[] args) {
        OOSpider.create(Site.me().setSleepTime(1000)
                , new ConsolePageModelPipeline(), GithubRepo.class)
                .addUrl("https://github.com/code4craft")
                .thread(5)
                .run();
    }
}