scrape-it-now

The project uses a decoupled architecture to handle large-scale scraping and indexing.

Scrape Process

The CLI interacts with a queue (Azure Queue Storage or local sqlite) to manage URLs. It pulls messages to scrape, checks a cache in Blob Storage (or local disk) to avoid re-scraping unchanged pages, browses the web, updates the cache, and pushes new discovered links back to the queue for deeper exploration.

Index Process

The indexing job pulls messages from a 'to-chunk' queue. It retrieves the scraped content from storage, chunks the markdown into coherent parts, generates embeddings via Azure OpenAI, and finally pushes the indexed content into Azure AI Search.

Scrape It Now! organizes scraped data in blob storage using a specific folder hierarchy. This structure separates raw page data, assets (like screenshots and images), and job state information.

Directory Hierarchy:

• [job_name]-scraping/: The root folder for the job.
  • scraped/: Contains all extracted data.
    • [page_id]/: A folder for a specific page containing:
      • screenshot.jpeg: The page screenshot (if enabled).
      • [image_id].[ext]: Image binaries (if enabled).
      • [image_id].json: Image metadata (if enabled).
    • [page_id].json: The JSON data extracted from the page.
  • state/: Contains [page_id] files for managing cache and parallelization.
  • job.json: Aggregated statistics for the entire job.

Local Disk storage is available for both blob and queue services. It is intended for testing and development only and is not recommended for production due to lack of scalability and fault tolerance.

• Local Disk Blob: Uses a directory structure where each blob is a file. Leases are implemented using lock files. Files are stored relative to the command execution directory by default.
• Local Disk Queue: Uses a SQLite database stored in the cache directory. It implements visibility timeouts and deletion tokens to mimic stateless services like Azure Queue Storage.

When bundling the application in a container, the application normally downloads dependencies (like the Chromium browser) at runtime. To improve performance and ensure all assets are present, you can pre-download them by running the scrape install command during your container build process.

scrape-it-now scrape install

The scrape run command starts a web scraping job. You can choose between using Azure services (Blob Storage and Queue Storage) or local disk storage.

Using Azure Storage

Set the following environment variables:

• AZURE_STORAGE_ACCESS_KEY
• AZURE_STORAGE_ACCOUNT_NAME

Using Local Disk

Set the following environment variables:

• BLOB_PROVIDER=local_disk
• QUEUE_PROVIDER=local_disk

# Azure Storage configuration
export AZURE_STORAGE_ACCESS_KEY=xxx
export AZURE_STORAGE_ACCOUNT_NAME=xxx
# Run the job
scrape-it-now scrape run https://nytimes.com

# OR Local disk configuration
export BLOB_PROVIDER=local_disk
export QUEUE_PROVIDER=local_disk
# Run the job
scrape-it-now scrape run https://nytimes.com

The --whitelist option allows you to restrict scraping to specific domains and ignore sub-paths using a list of regular expressions. The format is domain,regexp1,regexp2 domain2,regexp3.

Examples:

• Whitelist learn.microsoft.com: learn\.microsoft\.com
• Whitelist learn.microsoft.com and go.microsoft.com, but only allow the /en-us sub-path: learn\.microsoft\.com,^/(?!en-us).* go\.microsoft\.com

Install the package using pip. You can then run the CLI using the scrape-it-now command. Configuration can be handled via environment variables, a .env file, or command line options.

# Install the package
python3 -m pip install scrape-it-now

# Run the CLI
scrape-it-now --help

To install from source, ensure you have Python 3.13 or later installed. You can use pyenv to manage Python versions. Follow these steps:

1. Clone the repository.
2. Navigate to the directory.
3. Run the provided make install dev script.
4. Verify the installation by running the CLI help command.

# Download the source code
git clone https://github.com/clemlesne/scrape-it-now.git

# Move to the directory
cd scrape-it-now

# Run install scripts
make install dev

# Run the CLI
scrape-it-now --help

Use the scrape status [job_name] command to view the progress of a specific job. The output is a JSON object containing created_at, last_updated, network_used_mb, processed, and queued counts.

Configuration

• For Azure: Requires AZURE_STORAGE_CONNECTION_STRING.
• For Local: Requires BLOB_PROVIDER=local_disk.

# Azure Storage configuration
export AZURE_STORAGE_CONNECTION_STRING=xxx
# Show the job status
scrape-it-now scrape status [job_name]

# Local disk configuration
export BLOB_PROVIDER=local_disk
# Show the job status
scrape-it-now scrape status [job_name]

You can simplify CLI configuration by sourcing environment variables from a .env file.

• For single values (like access keys), use the standard assignment: KEY=value.
• For arguments that accept multiple values (like --whitelist), use a space-separated list.

# Single value
AZURE_STORAGE_ACCESS_KEY=xxx

# Multiple values (space-separated)
WHITELIST=learn\.microsoft\.com go\.microsoft\.com

To extend scrape-it-now with a custom queue backend, implement the IQueue abstract base class. Your implementation must provide the following asynchronous methods to manage the lifecycle of messages and the queue itself:

• send_message(message: str): Sends a raw string message to the queue.
• receive_messages(max_messages: int, visibility_timeout: int): An async generator that yields Message objects.
• delete_message(message: Message): Removes a specific message from the queue.
• create_queue(): Initializes the queue storage.
• delete_queue(): Removes the queue storage.
• __aenter__ and __aexit__: Support for asynchronous context management (async with).