DOCSEARCHbyAlgolia
Submit Docs

SitemapGenerator

repository·master·Indexed 25 days ago

https://github.com/kjvarga/sitemap_generator

A framework-agnostic Ruby library for generating XML sitemaps adhering to the Sitemap 0.9 protocol. It supports News, Video, Image, and PageMap sitemaps, as well as alternate links for internationalization. The library integrates with Rails (versions 6.0 through 8.1) to provide route helpers and automated Rake tasks, and supports Ruby versions 2.6 through 4.0.

Tokens
16.1K
Snippets
38
Records
90
Agent score
81%

What's inside sitemap_generator

  1. How data flows during sitemap generation

    master

    The generation process follows these steps:

    1. Trigger: You call SitemapGenerator::Sitemap.create { ... } or run the sitemap:refresh Rake task.
    2. Delegation: Sitemap delegates the call to LinkSet#create.
    3. Evaluation: LinkSet resets its state, applies your options, and passes your block to the Interpreter.
    4. Link Collection: The Interpreter (using Rails helpers if available) calls LinkSet#add for every link you define.
    5. Buffering: LinkSet#add appends links to the current SitemapFile. When a file hits its limit (50k links or 50MB), it is finalized and a new one begins.
    6. Finalization: Once the block finishes, LinkSet#finalize! closes the last SitemapFile and writes the SitemapIndexFile.
    7. Persistence: The raw XML for each file is passed to the configured adapter's write(location, raw_data) method to be saved.

  2. Manage sitemap files and limits with SitemapFile

    master

    A SitemapFile represents an individual sitemap file being built (either .xml or .xml.gz). It buffers <url> entries and finalizes once it reaches capacity.

    Important Limits and Errors:

    • Capacity: Raises SitemapFullError if link_count reaches max_sitemap_links (50,000) or if filesize reaches MAX_SITEMAP_FILESIZE (50 MB).
    • Lifecycle: Raises SitemapFinalizedError if you attempt to write to a file after finalize! has been called. Once finalized, the object is frozen.

    Key Fields:

    • location: A SitemapLocation resolving the file path and URL.
    • link_count: Number of <url> entries written.
    • news_count: Number of <news:news> entries (capped at 1,000).
    • filesize: Current uncompressed byte size.

  3. Use Adapters to upload sitemaps to remote hosts

    master

    If you are on a platform like Heroku that has a read-only filesystem (except for tmp/), you can use an adapter to upload sitemaps to remote storage.

    Supported Adapters

    • SitemapGenerator::FileAdapter: Standard local file writing.
    • SitemapGenerator::ActiveStorageAdapter: Uses ActiveStorage::Blob.
    • SitemapGenerator::FogAdapter: Uses Fog::Storage (requires require 'fog').
    • SitemapGenerator::S3Adapter: Uses Fog::Storage for Amazon S3 (requires require 'fog-aws').
    • SitemapGenerator::AwsSdkAdapter: Uses Aws::S3::Resource (requires require 'aws-sdk-s3').
    • SitemapGenerator::WaveAdapter: Uses CarrierWave::Uploader::Base (requires require 'carrierwave').
    • SitemapGenerator::GoogleStorageAdapter: Uses Google::Cloud::Storage (requires require 'google/cloud/storage').

    Example: Configuring S3Adapter (Fog)

    SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new(
  aws_access_key_id: 'YOUR_KEY',
  aws_secret_access_key: 'YOUR_SECRET',
  fog_directory: 'your-bucket-name'
)

    Example: Configuring AwsSdkAdapter

    SitemapGenerator::Sitemap.adapter = SitemapGenerator::AwsSdkAdapter.new('s3_bucket', 
  acl: 'public-read',
  region: 'us-east-1'
)

    Example: Configuring GoogleStorageAdapter

    SitemapGenerator::Sitemap.adapter = SitemapGenerator::GoogleStorageAdapter.new(
  bucket: 'name_of_bucket',
  project_id: 'google_account_project_id'
)

  4. How sitemap_generator works

    master

    The sitemap_generator gem works by accepting a Ruby block where you define your URLs and their metadata (such as changefreq, priority, images, video, or news). It then builds Sitemap 0.9-compliant XML files, optionally gzipped, and writes them to a storage backend using a pluggable adapter system.

    By default, it writes to the local filesystem, but it can be configured to use remote stores like S3, GCS, or Fog. The output consists of one or more .xml.gz sitemap files and a single sitemap index file.

  5. Follow Semantic Versioning for releases

    master

    The project follows Semantic Versioning to communicate the nature of changes in each release:

    • Patch (X.Y.Z+1): Used for bug fixes that do not introduce API changes.
    • Minor (X.Y+1.0): Used for new features that remain backwards-compatible.
    • Major (X+1.0.0): Used for breaking changes. Breaking changes must be documented under a **Breaking:** header in CHANGES.md.

  6. Understand the SitemapGenerator data model

    master
    SitemapGenerator does not use a database. Instead, it uses an in-memory data model consisting of objects that represent a single sitemap generation run. The orchestration is handled by a LinkSet, which manages the lifecycle of SitemapFile and SitemapIndexFile objects and delegates file writing to an Adapter.

  7. Configure Adapters for sitemap persistence

    master

    An Adapter is a write backend that determines where your sitemap files are saved. Any class implementing a write(location, raw_data) method can serve as an adapter.

    Built-in adapters include:

    • FileAdapter: Writes to the local disk.
    • AwsSdkAdapter: Writes to S3 using aws-sdk-s3.
    • S3Adapter: Writes to S3 using fog.
    • FogAdapter: General Fog-based adapter.
    • GoogleStorageAdapter: Writes to Google Cloud Storage.
    • ActiveStorageAdapter: Integrates with Rails Active Storage.
    • WaveAdapter.

  8. Mocking and testing strategy

    master

    When writing tests for SitemapGenerator, follow these mocking guidelines:

    • Mock HTTP calls: Use WebMock to mock search engine ping URLs. Real network connections are disabled by default via WebMock.disable_net_connect!.
    • Mock Cloud Storage: Use instance_double or allow/expect stubs for cloud clients like Aws::S3::Client, Fog, or GCS in adapter unit tests.
    • Do NOT mock FileAdapter: Use the real filesystem with paths located under tmp/test/ (configured in spec helpers).
    • Do NOT mock internals: In integration specs, do not mock LinkSet or SitemapFile internals; exercise the full stack instead.

  9. Manage sitemap finalization and errors

    master

    The finalize! method (or the end of a group block) closes a SitemapFile or SitemapIndexFile, writes it through the configured adapter, and freezes the object.

    Warning: Once a file is finalized, it is frozen against further modification. Attempting to add more links to a finalized file will raise a SitemapFinalizedError.

  10. Understand SitemapGenerator error handling

    master

    The library uses a specific hierarchy for domain errors. All custom errors subclass SitemapGenerator::SitemapError, which itself inherits from StandardError. Errors are propagated to the caller and are not swallowed internally.

    Key domain errors include:

    • SitemapFullError: Raised when a sitemap file reaches its limit (link count, filesize, or news count).
    • SitemapFinalizedError: Raised when attempting to mutate a sitemap file that has already been finalized.

    Note: Adapters do not use custom errors for missing dependencies; they raise a standard LoadError with a descriptive message instructing the user on which gem to require.

  11. Understand the core SitemapGenerator abstractions

    master

    SitemapGenerator is built around several key objects that manage the lifecycle of your sitemaps:

    • SitemapGenerator::LinkSet: The central orchestration object. It owns your configuration (host, adapter, paths) and manages the creation and finalization of sitemap files.
    • SitemapGenerator::Builder::SitemapFile: Represents a single XML sitemap file containing URLs and metadata.
    • SitemapGenerator::Builder::SitemapIndexFile: A special XML file that lists multiple sitemap files. This is generated automatically if create_index is set to :auto and multiple sitemaps are required.
    • SitemapGenerator::Interpreter: The object that evaluates your configuration block. If you are running in a Rails environment, the interpreter includes Rails URL helpers, allowing you to use methods like article_path(article) directly inside your sitemap block.
    • SitemapGenerator::SitemapLocation: A value object that resolves both the filesystem path and the public URL for a sitemap file based on your host and path configuration.

  12. Understand the sitemap_generator component architecture

    master

    The system is composed of several key components that manage the lifecycle of sitemap generation:

    • SitemapGenerator::Sitemap: A top-level singleton that delegates method calls to an internal LinkSet via method_missing.
    • LinkSet: The orchestrator. It manages configuration (host, path, adapter), drives the Interpreter, and coordinates the lifecycle of SitemapFile and SitemapIndexFile.
    • Interpreter: Evaluates your configuration block. It provides the add and group methods and includes Rails URL helpers if you are running in a Rails environment.
    • SitemapFile: Responsible for building individual sitemap files. It buffers <url> entries and automatically triggers finalize! when a file reaches 50,000 links or 50 MB.
    • SitemapIndexFile: Builds the index file that lists all generated sitemap files. This is finalized after all individual sitemaps are written.
    • Adapters: Pluggable backends for writing files. Each adapter must implement a write(location, raw_data) method. Supported adapters include FileAdapter (default), AwsSdkAdapter, S3Adapter, FogAdapter, GoogleStorageAdapter, ActiveStorageAdapter, and WaveAdapter.