GoScrapy Documentation

GoScrapy uses a decoupled, three-layer telemetry architecture designed to collect high-performance metrics without blocking the crawler's execution path (the 'hot-path'). The system separates the high-speed recording of metrics from the periodic broadcasting of those metrics.

The Three Layers:

1. Collection Layer (Non-Blocking): Engine components like the Scheduler, Worker Pool, Pipeline Manager, and Stats Middleware maintain their own local metrics using sync/atomic counters. This prevents a central bottleneck during high-speed scraping.
2. Aggregation Layer (Periodic): The TelemetryHub acts as an orchestrator. It uses a background loop with a Go time.Ticker to periodically poll registered components.
3. Exhibition Layer (Broadcast): Observers (such as a TUI dashboard or loggers) consume the aggregated GlobalSnapshot to visualize or export data.

Telemetry Flow:

• Collection Phase: Components update internal atomic counters asynchronously during execution.
• Broadcast Phase: At a set interval (e.g., every 500ms), the TelemetryHub calls Snapshot() on all components to collect ComponentSnapshot data, builds a GlobalSnapshot, and sends it to observers via OnSnapshot(GlobalSnapshot).

GoScrapy follows a decoupled, event-driven architecture inspired by Scrapy. The system is orchestrated by an Engine that coordinates several specialized components through a Signal Bus.

Core Components

• Spider: Contains your custom scraping logic (requests and parsing). Spiders are automatically discovered via reflection.
• Engine: The central orchestrator that uses the Signal Bus for event-driven coordination.
• Scheduler: Manages the priority queue of pending requests.
• Worker Pool: Executes requests concurrently using a dynamic pool of workers.
• Middlewares: Pluggable hooks used to modify requests or responses (e.g., for retries, cookie management, or statistics).
• HTTP Adapter: The network layer responsible for fetching data (supports standard HTTP and TLS spoofing).
• Pipeline Manager: Handles the processing and exporting of items yielded by the spider.

A Spider is the core component responsible for parsing logic. It implements the gos.ICoreSpider[*T] interface. The engine uses reflection to auto-discover and call the following lifecycle methods:

1. Open(ctx context.Context): Called during engine startup. Use this to initiate your first requests using s.Request(ctx).
2. parse(ctx context.Context, resp core.IResponseReader): Your custom parsing logic. Use resp.Bytes() or other reader methods to access content, and s.Yield(data) to send parsed records to the configured pipelines.
3. Close(ctx context.Context): Called during engine shutdown for cleanup tasks.

package myspider

import (
	"context"
	"encoding/json"
	"github.com/tech-engine/goscrapy/pkg/core"
)

// open is auto-called by goscrapy during engine startup
func (s *Spider) Open(ctx context.Context) {
	req := s.Request(ctx).Url("https://httpbin.org/get")
	s.Parse(req, s.parse)
}

func (s *Spider) parse(ctx context.Context, resp core.IResponseReader) {
	s.Logger().Infof("status: %d", resp.StatusCode())

	var data Record
	if err := json.Unmarshal(resp.Bytes(), &data); err != nil {
		s.Logger().Errorf("failed to unmarshal record: %v", err)
		return
	}

	// Yield sends the data securely to your configured pipelines
	s.Yield(&data)
}

// close is auto-called by goscrapy during engine shutdown
func (s *Spider) Close(ctx context.Context) {
}

GoScrapy uses a central, strongly-typed signal bus. You can subscribe to events using a fluent builder pattern on the application instance. This allows you to monitor the engine, track item progress, or handle errors without coupling your logic to the core components.

Supported Signals:

app, _ := gos.New[*MyRecord]()
 
app.OnEngineStarted(func(ctx context.Context) {
    log.Println("engine started")
}).
OnItemScraped(func(ctx context.Context, item *MyRecord) {
    log.Printf("item scraped: %s", item.Title)
}).
OnSpiderError(func(ctx context.Context, err error) {
    log.Printf("spider error: %v", err)
})