musicbrainz-docker

repository·master·Indexed 19 days ago

https://github.com/metabrainz/musicbrainz-docker

A Docker-based deployment for running a MusicBrainz mirror server. It includes the web server, web API, Solr search capabilities, and automated data replication. The project provides configurations for full mirror installations, quick test setups with sample data, and local development environments for the MusicBrainz Server and Search Index Rebuilder (SIR).

Tokens
6.3K
Snippets
28
Records
37
Agent score
68%

What's inside musicbrainz-docker

  1. Publish Service Ports to the Host

    master

    ⚠️ Security Warning: The search service runs Solr 7 in standalone mode, which is vulnerable to privilege escalation (CVE-2025-24814). Solr should generally only be accessible to your own clients.

    To publish ports for all services (including db, valkey, and search) to the host:

    admin/configure add publishing-all-ports
    docker compose up -d

    If you are running a database-only mirror, use:

    admin/configure add publishing-db-port
    docker compose up -d
  2. Modify Service Memory Settings

    master

    To customize the RAM allocated to db (Postgres) or search (Solr) services, create a custom compose file in local/compose/ and add it via admin/configure.

    Example local/compose/memory-settings.yml to set 4GB for both:

    services:
      db:
        command: postgres -c "shared_buffers=4GB"
      search:
        environment:
          - SOLR_HEAP=4g

    To apply:

    admin/configure add local/compose/memory-settings.yml
    docker compose up -d
  3. Enable data replication

    master

    To keep your mirror updated with the latest MusicBrainz data, follow these steps:

    1. Set the replication token: Run the following and paste your MetaBrainz access token when prompted. The token is stored in local/secrets/metabrainz_access_token.

      admin/set-replication-token
    2. Configure and start replication:

      admin/configure add replication-token
      docker compose up -d
    3. Run initial replication: Catch up with the latest database updates.

      bash -c 'docker compose exec -T musicbrainz replication.sh &' && \
      docker compose exec musicbrainz /usr/bin/tail -f mirror.log
    4. Schedule daily replication: To enable a daily cron job (default 3 am UTC) within the musicbrainz service container:

      admin/configure add replication-cron
      docker compose up -d

    Monitoring: View the replication log with:

    docker compose exec musicbrainz tail --follow mirror.log
    admin/set-replication-token
    admin/configure add replication-token
    docker compose up -d
    bash -c 'docker compose exec -T musicbrainz replication.sh &' && \
    docker compose exec musicbrainz /usr/bin/tail -f mirror.log
    admin/configure add replication-cron
    docker compose up -d
  4. Quick Test Setup with Sample Data

    master

    To run a small server with sample data for testing SQL queries or Web Service calls (instead of a full installation):

    git clone https://github.com/metabrainz/musicbrainz-docker.git
    cd musicbrainz-docker
    admin/configure add musicbrainz-standalone
    docker compose build
    docker compose run --rm musicbrainz createdb.sh -sample -fetch
    docker compose up -d

    Key differences from full installation:

    • Uses a sample data dump instead of full dumps.
    • Runs in standalone mode instead of mirror mode.
    • Replication is not applicable.
  5. Clean up Docker images and resources

    master

    Rebuilding images (for updates or configuration changes) leaves old images on your disk. To reclaim space, you can prune unused Docker objects.

    :warning: If you use Docker for other projects, docker system prune --all will remove all unused images across your entire Docker environment, not just this project.

    docker system prune --all
  6. Enable experimental live indexing

    master

    Live indexing allows search indexes to update automatically as data changes. Warning: This feature is currently experimental and unstable.

    To enable it:

    1. Disable replication cron (if previously enabled):

      admin/configure rm replication-cron
      docker compose up -d
    2. Set up the database schema for SIR indexing:

      admin/setup-sir install
    3. Build/Update search indexes (if not already done).

    4. Enable live indexing:

      admin/configure add live-indexing-search
      docker compose up -d
    5. Re-enable replication cron (if you disabled it in step 1).

      admin/configure add replication-cron
      docker compose up -d
    admin/configure rm replication-cron
    docker compose up -d
    admin/setup-sir install
    admin/configure add live-indexing-search
    docker compose up -d
    admin/configure add replication-cron
    docker compose up -d
  7. Use admin helper scripts

    master

    The admin/ directory contains helper scripts designed to be executed from the host machine. These scripts manage configuration, replication, and message queues. You can use the --help flag on any script to see available options.

    Key scripts include:

    • admin/configure: Used for configuration management (refer to Docker Compose overrides for details).
    • admin/setup-sir: Used to enable live indexing.
    • admin/purge-message-queues: Clears existing message queues.
    • admin/set-replication-token: Used to enable replication.
    admin/configure --help
    admin/setup-sir --help
    admin/purge-message-queues --help
    admin/set-replication-token --help
  8. Hardware and software requirements for MusicBrainz mirror server

    master

    To run a MusicBrainz mirror server with indexed search, the following resources are recommended:

    Recommended Hardware (with indexed search):

    • CPU: 16 threads (x86-64 architecture)
    • RAM: 16 GB
    • Disk Space: 350 GB

    Note: If running without indexed search, requirements drop to 2 CPU threads, 4 GB RAM, and 100 GB disk space.

    Required Software:

    • Docker Compose 2 (or higher)
    • Git
    • GNU Bash 4 (or higher) utilities (required for admin/ helper scripts)
    • Linux or macOS (Windows users should use Ubuntu via VirtualBox)

    Important macOS Note: If using Docker Desktop, increase container memory in Preferences > Resources > Memory beyond the 2GB default.

  9. Configure MusicBrainz Docker via Environment Variables

    master

    You can customize the behavior of the MusicBrainz Docker deployment using environment variables, most conveniently by editing the .env file.

    To verify the configuration values that will be passed to your containers, use:

    docker compose config

    To apply changes made to the .env file, restart the services with:

    docker compose up -d
    docker compose config
    docker compose up -d
  10. Local Development Setup for MusicBrainz Server

    master

    To develop the MusicBrainz Server locally using a local source tree:

    1. Clone the server repository and the docker repository.
    2. Configure the environment to point to your local server code.
    3. Enable development mode.
    git clone https://github.com/metabrainz/musicbrainz-server.git
    MUSICBRAINZ_SERVER_LOCAL_ROOT=$PWD/musicbrainz-server
    git clone https://github.com/metabrainz/musicbrainz-docker.git
    cd musicbrainz-docker
    echo MUSICBRAINZ_DOCKER_HOST_IPADDRCOL=127.0.0.1: >> .env
    echo MUSICBRAINZ_SERVER_LOCAL_ROOT="$MUSICBRAINZ_SERVER_LOCAL_ROOT" >> .env
    admin/configure add musicbrainz-dev
    docker compose build
    docker compose run --rm musicbrainz createdb.sh -sample -fetch
    docker compose up -d

    Development Features:

    • Development mode enabled (Catalyst debug).
    • JS and resources recompile automatically on change.
    • Server restarts automatically on Perl file changes.
    • Ports are published to the host via MUSICBRAINZ_DOCKER_HOST_IPADDRCOL.

    To apply code changes, restart the container:

    docker compose restart musicbrainz
  11. Recreate the MusicBrainz database

    master

    To recreate the database, run the recreatedb.sh script inside the musicbrainz container. You will be prompted for the Postgres password defined in default/postgres.env.

    To fetch new data dumps immediately before recreating the database, use the -fetch flag.

    # Standard recreation
    docker compose run --rm musicbrainz recreatedb.sh
    
    # Recreation with fetching new data dumps
    docker compose run --rm musicbrainz recreatedb.sh -fetch
  12. Local Development Setup for MusicBrainz Solr

    master

    Solr development relies on schema changes in mb-solr and mmd-schema. To run a local version of mb-solr within the search service for integration testing:

    1. Run build.sh from your mb-solr local working copy to build an image with a local tag.
    2. Set MB_SOLR_VERSION in your .env file to that local tag.
    3. Start the services:
      docker compose up -d
    docker compose up -d