dotmesh

repository·master·Indexed 19 days ago

https://github.com/dotmesh-io/dotmesh

A 'git for data' tool providing a CLI for capturing, organizing, and sharing application states, specifically targeting databases and filesystem snapshots. It introduces 'datadots' as a core abstraction for snapshotting and branching data, featuring a Docker volume driver and Kubernetes operator for persistent volume management. The `dm` CLI allows users to manage clusters, clone remote dots, and handle branching and checkout operations.

Tokens
29.9K
Snippets
111
Records
160
Agent score
68%

What's inside dotmesh

  1. What is a datadot?

    master

    A datadot is the core abstraction in dotmesh. It allows you to capture an application's state (such as a database or filesystem) and manage it similarly to a Git repository.

    Key characteristics:

    • Snapshotting: It captures the state of databases or filesystems.
    • Branches: Datadots support branches (e.g., master) to manage different states.
    • Subdots: A single atomic commit in dotmesh can capture the state of multiple databases, each residing in its own subdot.
    • Docker Integration: You can use the dm volume driver to mount a datadot directly into a container.
  2. Develop dotmesh via integration tests

    master

    The recommended way to develop dotmesh backend code, especially when testing multi-node or multi-cluster behavior (like federated push/pull), is through the Dotmesh acceptance test suite.

    This suite uses a docker-in-docker (kubeadm style) approach. It creates docker containers that simulate entire computers running systemd. You can then use commands like dm cluster init to set up dotmesh within these simulated environments.

    Key characteristics:

    • Persistence: The test suite leaves the last docker-in-docker environments running so you can inspect logs or perform ad-hoc debugging after a test completes.
    • Performance: After an initial image priming step, spinning up a 2-node cluster for testing takes approximately 60 seconds.
    • Offline capability: Once images are primed, the tests do not require internet access.
  3. Setup development environment on Debian

    master

    To set up a Debian development environment:

    1. Install Docker and configure /etc/docker/daemon.json with overlay2 and insecure registries for $(hostname).local:80.
    2. Enable contrib sources in /etc/apt/sources.list for OpenZFS.
    3. Install zfsutils-linux, jq, golang, and moreutils.
    4. Set vm.max_map_count to 262144.
    5. Install dazel via pip3.
    6. Install Docker Compose.
    7. Add $(hostname).local to /etc/hosts.
    # Example /etc/docker/daemon.json
    {
        "storage-driver": "overlay2",
        "insecure-registries": ["$(hostname).local:80"]
    }
    
    # Example /etc/apt/sources.list
    deb http://ftp.us.debian.org/debian/ stretch main contrib
    deb-src http://ftp.us.debian.org/debian/ stretch main contrib
    
    # Commands to run as root
    apt-get update
    apt-get -y install zfsutils-linux jq golang moreutils python3-pip
    echo 'vm.max_map_count=262144' >> /etc/sysctl.conf
    sysctl vm.max_map_count=262144
    pip3 install dazel==0.0.36
    
    # Update hosts
    cat <<EOF >> /etc/hosts
    127.0.0.1 $(hostname).local
    EOF
  4. Manage datadots with dm switch and dm list

    master

    Once datadots are created, you can navigate between them and inspect their status using the dm CLI.

    Switch to a datadot

    Use dm switch <datadot_name> to make a specific datadot the "current" dot. Subsequent dm commands will operate on this dot by default, similar to how cd works in a Git repository.

    dm switch myapp

    List datadots

    Use dm list to view all datadots. The output includes the dot name, current branch, server ID, container associations, size, commit count, and whether the dot is "dirty" (has uncommitted changes).

    The current datadot is indicated by an asterisk (*) in the output.

    dm switch myapp
    dm list
  5. Setup development environment on NixOS

    master

    To develop dotmesh on NixOS, use a configuration that enables ZFS, sets a specific hostId, and installs the required system packages. You must also configure the Docker storage driver to overlay2 and allow insecure registries for your local hostname.

    Required System Configuration:

    • boot.supportedFilesystems = [ "zfs" ];
    • networking.hostId = "cafecafe"; (or a random one)
    • virtualisation.docker.storageDriver = "overlay2";
    • virtualisation.docker.extraOptions = "--insecure-registry ${config.networking.hostName}.local:80";
    • boot.kernel.sysctl."vm.max_map_count" = 262144; (required for elasticsearch)

    Required Packages: wget, vim, docker, docker_compose, universal-ctags, mtr, go, jq, tmux, tmate, gnumake, git, moreutils.

    boot.supportedFilesystems = [ "zfs" ];
    networking.hostId = "cafecafe"; # Make a random one.
    networking.hostName = "devmachine"; # Define your hostname.
    networking.extraHosts = "127.0.0.1 ${config.networking.hostName}.local";
    #nixpkgs.config.allowUnfree = true;
    environment.systemPackages = with pkgs; [
    #  chromium
    #  slack
      wget
      vim
      docker
      docker_compose
      universal-ctags
      mtr
      go
      jq
      tmux
      tmate
      gnumake
      git
      moreutils
    ];
    boot.kernel.sysctl."vm.max_map_count" = 262144; # for elasticsearch
    virtualisation.docker = {
      enable = true;
      storageDriver = "overlay2";
      extraOptions = "--insecure-registry ${config.networking.hostName}.local:80";
    };
    system.activationScripts.binbash = {
      text = "ln -sf /run/current-system/sw/bin/bash /bin/bash";
      deps = [];
    };
    networking.firewall.enable = false;

    Then apply the configuration:

    sudo nixos-rebuild switch
  6. Setup development environment on Ubuntu 18.04

    master

    To set up an Ubuntu 18.04 development environment, follow these steps to install Go, Docker, Docker Compose, and Bazel, and to configure system settings for ZFS and Elasticsearch.

    Steps:

    1. Install Go via snap.
    2. Install Docker and configure the overlay2 storage driver and insecure registries in /etc/docker/daemon.json.
    3. Install system packages (zfsutils-linux, jq, moreutils) and set vm.max_map_count to 262144.
    4. Install Docker Compose manually.
    5. Update /etc/hosts to include $(hostname).local.
    6. Install Bazel using the official installer script.
    # install go
    snap install go --classic
    
    # install docker
    apt-get update
    apt-get install     apt-transport-https     ca-certificates     curl     software-properties-common
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    apt-key fingerprint 0EBFCD88
    add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) \
       stable"
    apt-get update
    apt-get install docker-ce
    
    # configure docker
    cat << EOT > /etc/docker/daemon.json
    {
        "storage-driver": "overlay2",
        "insecure-registries": ["$(hostname).local:80"]
    }
    EOT
    systemctl restart docker
    
    # install packages
    apt install zfsutils-linux jq moreutils
    echo 'vm.max_map_count=262144' >> /etc/sysctl.conf
    sysctl vm.max_map_count=262144
    
    # install docker-compose
    sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
    sudo chmod +x /usr/local/bin/docker-compose
    
    # update hosts
    cat <<EOF >> /etc/hosts
    127.0.0.1 $(hostname).local
    EOF
    
    # install bazel
    apt-get -y install pkg-config zip g++ zlib1g-dev unzip python git-core
    curl -L -o bazel-installer.sh https://github.com/bazelbuild/bazel/releases/download/0.15.2/bazel-0.15.2-installer-linux-x86_64.sh
    chmod +x bazel-installer.sh && ./bazel-installer.sh --user
  7. Enable Tracing with Zipkin

    master

    Dotmesh supports OpenTracing via Zipkin. To enable tracing, you must specify the --tracing flag during the dm cluster init and dm cluster join commands. The flag expects the address of your Zipkin host. It is assumed that Zipkin is running on port 9411. Note that tracing is currently configured per-host.

    dm cluster init --tracing=TRACE_HOST
    dm cluster join --tracing=TRACE_HOST
  8. Update dependencies and rebuild the project

    master

    The project uses dep for dependency management and bazel for building. Because the project uses multiple vendoring locations (/vendor and /cmd/dm/vendor) to avoid clashes with client-go and kubernetes, any changes to dependencies, directory structures, or file locations require regenerating the build files using gazelle.

    To synchronize the build files after dependency changes, run the following command from the root of the repository:

    bazel build //:gazelle
  9. Set up a local dotmesh cluster using Docker

    master

    Once the dm client is installed, you can set up a single-instance dotmesh cluster on your local machine using Docker. This command pulls the dotmesh-server image and initializes the environment.

    Prerequisites:

    • Docker installed.
    • Your user account must have access to the Docker daemon.
    dm cluster init
  10. Set up a GitLab CI runner for dotmesh

    master

    To set up a manual CI runner on Linux, you must configure a user named gitlab-runner with specific permissions and register it with the GitLab coordinator.

    Key Requirements:

    • The gitlab-runner user must be in the docker group.
    • The gitlab-runner user must have passwordless sudo access.
    • For Ubuntu Bionic runners, use the tag ubuntu-bionic during registration.
    • For macOS runners, ensure auto-upgrade-docker is configured to handle Docker for Mac updates.

    Registration Configuration: When registering the runner via gitlab-runner register, use the following settings for a standard Ubuntu shell executor:

    • Executor: shell
    • Tags: fast,ubuntu (or fast,ubuntu-bionic for Bionic)
    • Lock to current project: Set to false (untick in the GitLab admin UI) to allow the runner to serve multiple projects.
    # Example sudoers entry for gitlab-runner
    gitlab-runner ALL=(ALL:ALL) NOPASSWD:ALL
    
    # Example registration flow
    # Please enter the gitlab-ci tags for this runner (comma separated):
    fast,ubuntu-bionic
    # Whether to run untagged builds [true/false]:
    true
    # Whether to lock the Runner to current project [true/false]:
    false
    # Please enter the executor:
    shell
  11. Install the datamesh CLI locally

    master

    To install the dm CLI tool on your local machine, download the binary using curl and initialize a local datamesh cluster.

    Prerequisites:

    • A Linux or macOS environment.
    • sudo privileges.
    sudo curl -o /usr/local/bin/dm \
        https://get.datamesh.io/$(uname -s)/dm
    sudo chmod +x /usr/local/bin/dm
    dm cluster init
  12. Build and run the dotmesh operator for development

    master

    To hack on or test the dotmesh operator, you must set up a shared directory with your dind (Docker-in-Docker) containers, compile the binary as a static executable, configure the Kubernetes ConfigMap, and run it using the dex command.

    1. Prepare the shared directory

    Create a directory that is accessible to your dind containers and set appropriate permissions:

    mkdir /dotmesh-test-pools/operator
    chmod a+rwx /dotmesh-test-pools/operator

    2. Compile the operator

    Compile the operator as a static Go binary to ensure compatibility within the container environment:

    CGO_ENABLED=0 go build -a -ldflags '-extldflags "-static"' -o /dotmesh-test-pools/operator/operator .

    3. Configure Kubernetes

    Create the required configuration via a ConfigMap in the dotmesh namespace. This tells the operator where the flexvolume driver directory is located:

    kubectl create configmap -n dotmesh configuration --from-literal=flexvolumeDriverDir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec

    4. Run the operator

    Execute the operator using dex. You can control the verbosity of the logs using the -v flag:

    • No -v flag: Logs only when performing significant actions.
    • -v 2: Standard verbosity.
    • -v 3: High verbosity (messier logs).
    dex 0 0 /dotmesh-test-pools/operator/operator --kubeconfig=/root/.kube/config -v 2
    CGO_ENABLED=0 go build -a -ldflags '-extldflags "-static"' -o /dotmesh-test-pools/operator/operator .
    
    kubectl create configmap -n dotmesh configuration --from-literal=flexvolumeDriverDir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec
    
    dex 0 0 /dotmesh-test-pools/operator/operator --kubeconfig=/root/.kube/config -v 2