Archive Team Warrior Dockerfile

repository·master·Indexed 19 days ago

https://github.com/archiveteam/warrior-dockerfile

A Dockerized deployment for the Archive Team Warrior, a tool for participating in web archiving projects. This repository provides configurations for running the Warrior via Docker, Docker Compose, and Kubernetes, including support for automated updates using Watchtower and configuration via environment variables such as DOWNLOADER and SELECTED_PROJECT.

Tokens
2.3K
Snippets
9
Records
10
Agent score
66%

What's inside warrior-dockerfile

  1. Run Warrior on Kubernetes

    master

    To deploy the Warrior on Kubernetes, use the provided k8s-warrior.yml manifest.

    1. Edit k8s-warrior.yml to set the DOWNLOADER environment variable to your name (this name appears on leaderboards).
    2. Create a namespace and apply the manifest.
    3. Access the web interface via any Kubernetes node's IP on port 30163.
    kubectl create namespace archive
    kubectl apply -n archive -f k8s-warrior.yml
  2. Run Warrior with Docker Compose

    master

    The repository includes a docker-compose.yml file in the examples directory that sets up both the Warrior and Watchtower.

    1. Navigate to the examples directory.
    2. Edit docker-compose.yml to configure your desired settings.
    3. Run the following command to start the services in the background.
    cd examples
    docker compose up -d
  3. Access the Warrior Web Interface

    master

    The Warrior provides a web interface for manual configuration and monitoring.

    1. Locate the IP: Use docker inspect to find the container's IP address.
    2. Access via Browser: Navigate to http://<CONTAINER_IP>:8001.

    Note for Headless/Remote Machines: If running on a remote machine, you must bind the container port to the host port using the -p 8001:8001 flag in your docker run command to access the interface over your LAN.

  4. Configure Warrior via Environment Variables

    master

    You can provide seed settings using environment variables during docker run. Note that if a projects/config.json file is mounted/exists, environment variables will be ignored.

    Use the --env flag to set configuration. For example, to set your downloader name and select a specific project:

    docker run --detach \
      --env DOWNLOADER="your name" \
      --env SELECTED_PROJECT="auto" \
      --name archiveteam-warrior \
      --label=com.centurylinklabs.watchtower.enable=true \
      --restart=on-failure \
      --publish 8001:8001 \
      atdr.meo.ws/archiveteam/warrior-dockerfile
  5. Run Archive Team Warrior via Docker

    master

    You can run the Archive Team Warrior using the pre-built image atdr.meo.ws/archiveteam/warrior-dockerfile.

    To ensure the Warrior stays updated, it is recommended to run Watchtower alongside it. The following setup runs Watchtower in the background and starts the Warrior with the necessary labels to allow Watchtower to manage it. The Warrior is configured to restart automatically on failure and exposes its web interface on port 8001.

    # Run Watchtower to handle automatic updates
    docker run --detach \
      --name watchtower \
      --restart=on-failure \
      --volume /var/run/docker.sock:/var/run/docker.sock \
      nickfedor/watchtower --label-enable --cleanup --interval 3600
    
    # Run the Warrior with Watchtower enabled
    docker run --detach \
      --name archiveteam-warrior \
      --label=com.centurylinklabs.watchtower.enable=true \
      --restart=on-failure \
      --publish 8001:8001 \
      atdr.meo.ws/archiveteam/warrior-dockerfile
  6. Run Archive Team Warrior using Docker Compose

    master

    You can deploy the Archive Team Warrior service using Docker Compose. The configuration includes a watchtower service to handle automatic image updates and the archiveTeamWarrior service itself.

    Note that the archiveTeamWarrior service is configured to listen on port 8001 by default. To ensure automatic updates via Watchtower, the service must include the label com.centurylinklabs.watchtower.enable: "true".

    version: "3"
    services:
      watchtower:
        image: nickfedor/watchtower:latest
        command: --cleanup --label-enable --interval 3600
        container_name: Watchtower
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
        labels:
          com.centurylinklabs.watchtower.enable: "true"
        restart: unless-stopped
    
      archiveTeamWarrior:
        image: atdr.meo.ws/archiveteam/warrior-dockerfile
        container_name: archiveTeamWarrior
        hostname: archiveTeamWarrior
        ports:
          - "8001:8001"
        labels:
          com.centurylinklabs.watchtower.enable: "true"
        restart: always
  7. Configure automatic updates with Watchtower

    master

    The provided Docker Compose setup uses watchtower to automatically update the archiveTeamWarrior container.

    To enable this behavior for your own services in this environment:

    1. Ensure the watchtower service is running with the --label-enable flag.
    2. Add the label com.centurylinklabs.watchtower.enable: "true" to the container you wish to update.
    labels:
      com.centurylinklabs.watchtower.enable: "true"
  8. Configure the Warrior service via environment variables

    master

    The start.py entrypoint script allows configuring the Warrior service using environment variables. If the configuration file projects/config.json does not exist, the script will generate it using the values provided in the following environment variables. Once generated, these values are persisted in the JSON file.

    Use these variables to define the downloader type, authentication credentials, the active project, and performance tuning parameters.

    # Example of setting environment variables for the container
    export DOWNLOADER="some-downloader"
    export HTTP_USERNAME="user"
    export HTTP_PASSWORD="pass"
    export SELECTED_PROJECT="project-name"
    export SHARED_RSYNC_THREADS="4"
    export WARRIOR_ID="unique-id"
    export CONCURRENT_ITEMS="10"
  9. Reference: Warrior Configuration Environment Variables

    master

    The following environment variables can be used to configure the Warrior. These map to specific keys in the projects/config.json file.

    | ENV | JSON key | Example | Default |
    |----------------------|----------------------|-------------------|---------|
    | DOWNLOADER           | downloader           |                   |         |
    | HTTP_PASSWORD        | http_password        |                   |         |
    | HTTP_USERNAME        | http_username        |                   |         |
    | SELECTED_PROJECT     | selected_project     | `auto`, `tumblr`  |         |
    | SHARED_RSYNC_THREADS | shared:rsync_threads |                   | `20`    |
    | WARRIOR_ID           | warrior_id           |                   |         |
    | CONCURRENT_ITEMS     | concurrent_items     |                   | `3`     |
  10. Reference: Warrior configuration environment variables

    master

    The following environment variables are used to populate projects/config.json if it is missing:

    VariableJSON KeyDescription
    DOWNLOADERdownloaderThe type of downloader to use
    HTTP_USERNAMEhttp_usernameUsername for HTTP authentication
    HTTP_PASSWORDhttp_passwordPassword for HTTP authentication
    SELECTED_PROJECTselected_projectThe ID of the project to work on
    SHARED_RSYNC_THREADSshared:rsync_threadsNumber of rsync threads to use
    WARRIOR_IDwarrior_idUnique identifier for this warrior instance
    CONCURRENT_ITEMSconcurrent_itemsNumber of items to process concurrently
    {
      "downloader": "",
      "http_username": "",
      "http_password": "",
      "selected_project": "",
      "shared:rsync_threads": "",
      "warrior_id": "",
      "concurrent_items": ""
    }