Abot Web Crawler Framework

The PoliteWebCrawler is the orchestrator of the crawl. It accepts alternate implementations for all its dependencies through its constructor. You can pass null for any implementation to use the default provided by Abot.

To use custom implementations for specific components (e.g., IPageRequester and IHyperLinkParser) while keeping defaults for others, pass null for the components you wish to leave as default.

var crawler = new PoliteWebCrawler(
	new CrawlConfiguration(),
	new YourCrawlDecisionMaker(),
	new YourThreadMgr(), 
	new YourScheduler(), 
	new YourPageRequester(), 
	new YourHyperLinkParser(), 
	new YourMemoryManager(), 
	new YourDomainRateLimiter,
	new YourRobotsDotTextFinder());

// Partial custom implementation (using nulls for defaults)
var crawler = new PoliteWebCrawler(
	null, 
	null, 
	null,
	null,
	new YourPageRequester(), 
	new YourHyperLinkParser(), 
	null,
	null, 
	null);

To perform a basic crawl, configure a CrawlConfiguration object, instantiate a PoliteWebCrawler, and subscribe to the PageCrawlCompleted event to process data. Use CrawlAsync to start the process.

using System.Threading.Tasks;
using Abot2.Core;
using Abot2.Crawler;
using Abot2.Poco;
using Serilog;

namespace TestAbotUse
{
    class Program
    {
        static async Task Main(string[] args)
        {
            Log.Logger = new LoggerConfiguration()
                .MinimumLevel.Information()
                .WriteTo.Console()
                .CreateLogger();

            Log.Logger.Information("Demo starting up!");

            await DemoSimpleCrawler();
            await DemoSinglePageRequest();
        }

        private static async Task DemoSimpleCrawler()
        {
            var config = new CrawlConfiguration
            {
                MaxPagesToCrawl = 10, //Only crawl 10 pages
                MinCrawlDelayPerDomainMilliSeconds = 3000 //Wait this many millisecs between requests
            };
            var crawler = new PoliteWebCrawler(config);

            crawler.PageCrawlCompleted += PageCrawlCompleted;//Several events available...

            var crawlResult = await crawler.CrawlAsync(new Uri("http://!!!!!!!!YOURSITEHERE!!!!!!!!!.com"));
        }

        private static async Task DemoSinglePageRequest()
        {
            var pageRequester = new PageRequester(new CrawlConfiguration(), new WebContentExtractor());

            var crawledPage = await pageRequester.MakeRequestAsync(new Uri("http://google.com"));
            Log.Logger.Information("{result}", new
            {
                url = crawledPage.Uri,
                status = Convert.ToInt32(crawledPage.HttpResponseMessage.StatusCode)
            });
        }

        private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
        {
            var httpStatus = e.CrawledPage.HttpResponseMessage.StatusCode;
            var rawPageText = e.CrawledPage.Content.Text;
        }
    }
}

If you need to add simple custom crawl logic without implementing full interfaces, you can use the shorthand delegate properties on PoliteWebCrawler. These allow you to decide whether to crawl a page, download its content, or crawl its links based on the current context.

Note: The ICrawlDecisionMaker's corresponding method is called first. If it does not "allow" a decision, these callbacks will not be executed.

crawler.ShouldCrawlPageDecisionMaker = (pageToCrawl, crawlContext) => 
{
	var decision = new CrawlDecision{ Allow = true };
	if(pageToCrawl.Uri.Authority == "google.com")
		return new CrawlDecision{ Allow = false, Reason = "Dont want to crawl google pages" };
	
	return decision;
};

crawler.ShouldDownloadPageContentDecisionMaker = (crawledPage, crawlContext) =>
{
	var decision = new CrawlDecision{ Allow = true };
	if (!crawledPage.Uri.AbsoluteUri.Contains(".com"))
		return new CrawlDecision { Allow = false, Reason = "Only download raw page content for .com tlds" };

	return decision;
};

crawler.ShouldCrawlPageLinksDecisionMaker = (crawledPage, crawlContext) =>
{
	var decision = new CrawlDecision{ Allow = true };
	if (crawledPage.Content.Bytes.Length < 100)
		return new CrawlDecision { Allow = false, Reason = "Just crawl links in pages that have at least 100 bytes" };

	return decision;
};

Abot provides several events to hook into the crawling lifecycle. You can register methods to handle starting, completing, or disallowed crawl attempts.

Common events include:

• PageCrawlStarting
• PageCrawlCompleted
• PageCrawlDisallowed
• PageLinksCrawlDisallowed

crawler.PageCrawlCompleted += crawler_ProcessPageCrawlCompleted;
crawler.PageCrawlDisallowed += crawler_PageCrawlDisallowed;
crawler.PageLinksCrawlDisallowed += crawler_PageLinksCrawlDisallowed;

void crawler_ProcessPageCrawlStarting(object sender, PageCrawlStartingArgs e)
{
	PageToCrawl pageToCrawl = e.PageToCrawl;
	Console.WriteLine($"About to crawl link {pageToCrawl.Uri.AbsoluteUri} which was found on page {pageToCrawl.ParentUri.AbsoluteUri}");
}

void crawler_ProcessPageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
	CrawledPage crawledPage = e.CrawledPage;    
	if (crawledPage.HttpRequestException != null || crawledPage.HttpResponseMessage.StatusCode != HttpStatusCode.OK)
		Console.WriteLine($"Crawl of page failed {crawledPage.Uri.AbsoluteUri}");
	else
		Console.WriteLine($"Crawl of page succeeded {crawledPage.Uri.AbsoluteUri}");

	if (string.IsNullOrEmpty(crawledPage.Content.Text))
		Console.WriteLine($"Page had no content {crawledPage.Uri.AbsoluteUri}");

	var angleSharpHtmlDocument = crawledPage.AngleSharpHtmlDocument; //AngleSharp parser
}

void crawler_PageLinksCrawlDisallowed(object sender, PageLinksCrawlDisallowedArgs e)
{
	CrawledPage crawledPage = e.CrawledPage;
	Console.WriteLine($"Did not crawl the links on page {crawledPage.Uri.AbsoluteUri} due to {e.DisallowedReason}");
}

void crawler_PageCrawlDisallowed(object sender, PageCrawlDisallowedArgs e)
{
	PageToCrawl pageToCrawl = e.PageToCrawl;
	Console.WriteLine($"Did not crawl page {pageToCrawl.Uri.AbsoluteUri} due to {e.DisallowedReason}");
}

The IScheduler interface manages the priority and storage of pages that need to be crawled. The crawler provides discovered links to the scheduler and retrieves the next page to crawl from it. A common use case is implementing a DistributedScheduler to manage crawls across multiple machines.

/// Handles managing the priority of what pages need to be crawled
/// </summary>
public interface IScheduler
{
	/// <summary>
	/// Count of remaining items that are currently scheduled
	/// </summary>
	int Count { get; }

	/// <summary>
	/// Schedules the param to be crawled
	/// </summary>
	void Add(PageToCrawl page);

	/// <summary>
	/// Schedules the param to be crawled
	/// </summary>
	void Add(IEnumerable<PageToCrawl> pages);

	/// <summary>
	/// Gets the next page to crawl
	/// </summary>
	PageToCrawl GetNext();

	/// <summary>
	/// Clear all currently scheduled pages
	/// </summary>
	void Clear();
}