DotnetSpider

DotnetSpider requires .NET Core 2.2 or later and supports various data stores. You can use Docker to quickly spin up the required infrastructure:

• MySQL: docker run --name mysql -d -p 3306:3306 --restart always -e MYSQL_ROOT_PASSWORD=1qazZAQ! mysql:5.7
• Redis: docker run --name redis -d -p 6379:6379 --restart always redis
• SQL Server: docker run --name sqlserver -d -p 1433:1433 --restart always -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=1qazZAQ!' mcr.microsoft.com/mssql/server:2017-latest
• PostgreSQL: docker run --name postgres -d -p 5432:5432 --restart always -e POSTGRES_PASSWORD=1qazZAQ! postgres
• MongoDB: docker run --name mongo -d -p 27017:27017 --restart always mongo
• RabbitMQ: docker run -d --restart always --name rabbimq -p 4369:4369 -p 5671-5672:5671-5672 -p 25672:25672 -p 15671-15672:15671-15672 -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password rabbitmq:3-management
• HBase: docker run -d --restart always --name hbase -p 20550:8080 -p 8085:8085 -p 9090:9090 -p 9095:9095 -p 16010:16010 dajobe/hbase

The AgentCenter service is a management component for DotnetSpider that uses Microsoft.Extensions.Hosting to orchestrate agent registration, statistics, and messaging. It is configured via the standard .NET configuration system (e.g., appsettings.json or environment variables).

To set up the service, you must provide configuration for AgentCenterOptions, a MySQL-based agent store, a MySQL-based statistic store, and RabbitMQ. The service uses Serilog for logging, outputting to both the Console and a rolling file at logs/agent-register.log.

// The service is initialized using the Host builder pattern
var builder = Host.CreateDefaultBuilder(args);
builder.ConfigureServices((context, x) =>
{
    x.Configure<AgentCenterOptions>(context.Configuration);
    x.AddHttpClient();
    x.AddAgentCenterHostService<MySqlAgentStore>();
    x.AddStatisticHostService<MySqlStatisticStore>();
    x.AddRabbitMQ(context.Configuration);
});
builder.UseSerilog();
await builder.Build().RunAsync();

You can create a high-level spider by inheriting from Spider and defining an entity class. Use attributes to configure data extraction, selectors, and formatting:

• [Schema(collection, table)]: Defines the storage schema.
• [EntitySelector(Expression, Type)]: Defines the XPath/CSS selector for the main entity container.
• [ValueSelector(Expression, Type)]: Defines how to extract specific fields.
• [GlobalValueSelector(Expression, Name, Type)]: Extracts values to be used globally across all entities in a page.
• [FollowRequestSelector(Expressions)]: Defines selectors for finding links to follow.
• [ReplaceFormatter(NewValue, OldValue)]: Formats string values by replacing content.
• [TrimFormatter]: Trims whitespace from extracted strings.

Example implementation:

[DisplayName("博客园爬虫")]
public class EntitySpider( 
    IOptions<SpiderOptions> options, 
    DependenceServices services, 
    ILogger<Spider> logger) 
    : Spider(options, services, logger)
{
    public static async Task RunAsync()
    {
        var builder = Builder.CreateDefaultBuilder<EntitySpider>(options =>
        {
            options.Speed = 1;
        });
        builder.UseSerilog();
        builder.IgnoreServerCertificateError();
        await builder.Build().RunAsync();
    }

    protected override async Task InitializeAsync(CancellationToken stoppingToken = default)
    {
        AddDataFlow<DataParser<CnblogsEntry>>();
        AddDataFlow(GetDefaultStorage);
        await AddRequestsAsync(
            new Request("https://news.cnblogs.com/n/page/1", new Dictionary<string, object> { { "网站", "博客园" } }));
    }

    [Schema("cnblogs", "news")]
    [EntitySelector(Expression = ".//div[@class='news_block']", Type = SelectorType.XPath)]
    [GlobalValueSelector(Expression = ".//a[@class='current']", Name = "类别", Type = SelectorType.XPath)]
    [GlobalValueSelector(Expression = "//title", Name = "Title", Type = SelectorType.XPath)]
    [FollowRequestSelector(Expressions = ["//div[@class='pager']"])]
    public class CnblogsEntry : EntityBase<CnblogsEntry>
    {
        protected override void Configure()
        {
            HasIndex(x => x.Title);
            HasIndex(x => new { x.WebSite, x.Guid }, true);
        }

        public int Id { get; set; }

        [Required]
        [StringLength(200)]
        [ValueSelector(Expression = "类别", Type = SelectorType.Environment)]
        public string Category { get; set; }

        [Required]
        [StringLength(200)]
        [ValueSelector(Expression = "网站", Type = SelectorType.Environment)]
        public string WebSite { get; set; }

        [StringLength(200)]
        [ValueSelector(Expression = "Title", Type = SelectorType.Environment)]
        [ReplaceFormatter(NewValue = "", OldValue = " - 博客园")]
        public string Title { get; set; }

        [StringLength(40)]
        [ValueSelector(Expression = "GUID", Type = SelectorType.Environment)]
        public string Guid { get; set; }

        [ValueSelector(Expression = ".//h2[@class='news_entry']/a")]
        public string News { get; set; }

        [ValueSelector(Expression = ".//h2[@class='news_entry']/a/@href")]
        public string Url { get; set; }

        [ValueSelector(Expression = ".//div[@class='entry_summary']")]
        [TrimFormatter]
        public string PlainText { get; set; }

        [ValueSelector(Expression = "DATETIME", Type = SelectorType.Environment)]
        public DateTime CreationTime { get; set; }
    }
}