DOCSEARCHbyAlgolia
Submit Docs

Memorious

website·Indexed 30 days ago

https://alephdata.github.io/memorious

A light-weight web scraping toolkit for collecting structured or unstructured data. It features a modular crawler pipeline configurable via YAML, a command-line tool for monitoring and invocation, and supports Docker Compose for installation.

Tokens
3.9K
Snippets
24
Records
34
Agent score
75%

What's inside Memorious

  1. Overview of Memorious web scraping toolkit

    Memorious is a lightweight web scraping toolkit used to collect structured or unstructured data. It is designed to make crawlers modular by breaking them down into reusable tasks and providing utility functions for common operations like HTTP session management and data storage. It is specifically designed to integrate with the Aleph and FollowTheMoney ecosystem.

  2. Design and architecture of Memorious crawlers

    Memorious uses a stage-based architecture to handle the typical scraping workflow (e.g., paginating an index, downloading result pages, and parsing data into a database). A crawler is composed of multiple stages, where each stage is implemented as a Python function. This modularity allows stages to be reused across different crawlers.

    The general workflow for creating a crawler is:

    1. Create a YAML crawler configuration file.
    2. Define the various stages.
    3. Implement custom Python code for stage operations if needed.
    4. Test and iterate.

  3. Extend Memorious with custom Python methods

    You can implement custom crawler logic by creating a Python module and installing it. Custom methods must accept two arguments: context and data. The data argument contains the output from the previous stage.
    my_stage:
  method: custom.module:my_method
  params:
    my_param: my_value
  handle:
    pass: store

  4. Debug crawlers using sampling and the debug operation

    To debug large crawlers without processing all data, use the sampling_rate parameter (a float between 0.0 and 1.0) in a pipeline stage. This determines the percentage of tasks passed to the next stage. Additionally, the debug operation can be used to log data passed into it.
    pipeline:
  # ... other stages ...
  crawl:
    method: example.quotes:crawl
    params:
      sampling_rate: 0.2 # Only 20% of tasks proceed to the next stage
    handle:
      fetch: fetch
      pass: debug
  debug:
    method: debug

  5. Define the Memorious crawler pipeline and stages

    Memorious crawlers use a pipeline consisting of named stages. Each stage receives input from the previous stage via a data dictionary and yields output for the next. A stage must define a method (the action to perform) and a handle (the next stage to trigger based on a condition, typically pass). Optional params can be used to pass static configuration values into the method.
    name: my_crawler
pipeline:
  init:
    method: xxx
    handle:
      pass: crawl
  crawl:
    method: yyy
    handle:
      pass: save
  save:
    method: zzz

  6. Run and manage crawlers via Memorious CLI

    The Memorious CLI is accessed through the shell container in a Docker Compose setup. Use memorious list to see available crawlers and memorious run <crawler_name> to execute a specific crawler. Note that Memorious will find YAML files as long as the MEMORIOUS_CONFIG_PATH environment variable points to the directory containing them, regardless of nesting level.
    # Access the CLI shell
docker-compose run --rm shell

# List available crawlers
memorious list

# Run a specific crawler
memorious run my_crawler

  7. Implement incremental crawling and caching with Redis tags

    Memorious uses Redis tags to cache responses. Cache expiration is controlled by the crawler's expire config option or the MEMORIOUS_EXPIRE environment variable. The context.skip_incremental(*criteria) helper is specifically used to avoid re-processing URLs or items already encountered in previous runs.
    # Basic cache management
context.set_tag(key, value)
context.check_tag(key) # Returns True if key exists
context.get_tag(key)

# Incremental crawling logic
if context.skip_incremental(url):
    # The URL has been crawled before; skip it
    pass
else:
    # New URL; process it
    process_url(url)

  8. Install Memorious using Docker Compose

    To run Memorious crawlers in production, it is recommended to use Docker Compose. The setup involves copying the example project structure, adding crawler YAML configurations to the config directory, and adding Python extensions to the src directory. Ensure setup.py is updated with the project name and dependencies. If custom database connections or directory structures are required, update the environment variables in the Dockerfile or docker-compose.yml.
    docker-compose up -d

  9. Run Memorious in local development mode

    To avoid rebuilding Docker containers during development, you can run Memorious directly in your shell. Note that in this mode, Memorious uses a single-threaded worker (via FakeRedis), meaning task execution is linear and concurrency is limited.
    # 1. Setup environment variables
cp env.sh.tmpl env.sh
# Edit env.sh to ensure MEMORIOUS_CONFIG_PATH is correct
source env.sh

# 2. Install dependencies
pip install memorious
# If using Python extensions, run pip install in the crawler directory

# 3. Execute
memorious list
memorious run your-crawler

  10. Configure Memorious via environment variables

    Memorious is configured using environment variables typically set in the Dockerfile. These control database connectivity, rate limits, and storage backends.
    # General Configuration
MEMORIOUS_CONFIG_PATH=path/to/yaml/configs
MEMORIOUS_DEBUG=False # Disables task threading
MEMORIOUS_INCREMENTAL=True # Execute parts of crawler once per interval
MEMORIOUS_EXPIRE=1 # Days until cached data expires
MEMORIOUS_USER_AGENT="Your Custom User Agent"

# Rate Limits
MEMORIOUS_DB_RATE_LIMIT=6000 # Max DB inserts per minute
MEMORIOUS_HTTP_RATE_LIMIT=120 # Max HTTP calls per host per minute

# Database & Queue
MEMORIOUS_DATASTORE_URI=path/to/datastore.sqllite3
REDIS_URL=redis://address:port

# Storage/Archive
ARCHIVE_TYPE=file # Options: file, s3, gs
ARCHIVE_PATH=/local/path # Used if ARCHIVE_TYPE is file
ARCHIVE_BUCKET=my-bucket # Used if ARCHIVE_TYPE is s3 or gs

# AWS (Required if ARCHIVE_TYPE is s3)
AWS_ACCESS_KEY_ID=your_key
AWS_SECRET_ACCESS_KEY=your_secret
AWS_REGION=your_region

# Aleph Integration
ALEPH_HOST=https://data.occrp.org/
ALEPH_API_KEY=your_api_key

  11. Use the Context object in custom methods

    The context object provides essential utilities for custom Python methods:

    • Config Access: Access parameters via context.params.get('key') and crawler metadata via context.crawler.name.
    • HTTP Session: context.http is a requests wrapper with built-in caching and serialization. It returns ContextHTTPResponse objects containing url, status_code, content_hash, and content in raw, text, html, xml, or json formats.
    • Data Validation: Helpers include is_not_empty, is_numeric, is_integer, match_date, match_regexp, has_length, and must_contain.
    • Datastore: Access or create database tables for intermediary data using context.datastore['table_name'] (based on the dataset library).

  12. Shut down Memorious and manage persistence

    To stop Memorious gracefully, use docker-compose down. Downloaded files, Redis progress data, and the task queue are persisted in the build directory. To perform a completely fresh start and clear all cached data and progress, delete the build directory.
    docker-compose down