ML Workspace Documentation

The workspace includes several tools for Git integration:

• Cloning: Navigate to a folder and click the git button to open ungit, a web-based Git client. You can clone repositories and handle authentication within ungit.
• Notebook Git Actions: Use the integrated Jupyter Git plugin to commit and push single notebooks.
• Advanced Git: Use ungit for branching, tagging, pulling, and merging.
• Notebook Diffing/Merging: nbdime is pre-installed to handle intelligent diffing and merging of JSON-based notebook files.
• Plain Text Notebooks: jupytext allows you to open, edit, and run .py or .md files as notebooks within Jupyter.

When using a dedicated Jupyter Kernel for a virtual environment or a specific Python version, standard shell commands like !pip install matplotlib may use the wrong Python/pip instance. To ensure you are using the version associated with the current kernel, use the following pattern in a notebook cell:

import sys
!{sys.executable} -m pip install matplotlib

You can use the workspace as a remote runtime for a local Jupyter instance using remote_ikernel.

1. Install the tool locally: pip install remote_ikernel.
2. The workspace SSH setup script will offer to configure the remote kernel automatically.
3. To manage kernels manually:

# Add a remote kernel
remote_ikernel manage --add \
    --interface=ssh \
    --kernel_cmd="ipython kernel -f {connection_file}" \
    --name="ml-server (Python)" \
    --host="my-workspace"

# List kernels
remote_ikernel manage --show

# Delete a kernel
remote_ikernel manage --delete <REMOTE_KERNEL_NAME>

To build the gpu flavor, update CUDA tooling based on NVIDIA container images and ensure compatibility with TensorFlow and PyTorch. Build the flavor using the build.py script. Verify GPU access by running nvidia-smi in the terminal and testing the GPU interface in Netdata and Glances.

python build.py --flavor=gpu

To build the minimal flavor of the workspace, use the build.py script with the --make and --flavor=minimal flags. After building, verify the container by checking startup logs, comparing image layer sizes in Portainer, and checking folder sizes via Disk Usage Analyzer within the Desktop VNC.

python build.py --make --flavor=minimal

You can create a custom workspace flavor by extending an existing image in a Dockerfile. This allows you to include your own software or specific configurations. Use the $RESOURCES_PATH environment variable to access provided installation scripts.

# Extend from any of the workspace versions/flavors
FROM mltooling/ml-workspace:0.13.2

# Run you customizations, e.g.
RUN \
    # Install r-runtime, r-kernel, and r-studio web server from provided install scripts
    /bin/bash $RESOURCES_PATH/tools/r-runtime.sh --install && \
    /bin/bash $RESOURCES_PATH/tools/r-studio-server.sh --install && \
    # Cleanup Layer - removes unneccessary cache files
    clean-layer.sh

Authentication is required to secure access to all pre-installed tools. It is recommended to use one of the following two methods:

1. Token-based Authentication via Jupyter (Recommended)

Set AUTHENTICATE_VIA_JUPYTER to a specific token string or <generated> to have Jupyter print a random token in the logs.

docker run -p 8080:8080 --env AUTHENTICATE_VIA_JUPYTER="mytoken" mltooling/ml-workspace:0.13.2

2. Basic Authentication via Nginx

Set WORKSPACE_AUTH_USER and WORKSPACE_AUTH_PASSWORD. This is configured via the nginx proxy and may be more performant.

docker run -p 8080:8080 --env WORKSPACE_AUTH_USER="user" --env WORKSPACE_AUTH_PASSWORD="pwd" mltooling/ml-workspace:0.13.2

Use tools like piprot, pur, or pip-upgrader to update the requirement files located in ./resources/libraries/.

Using piprot:

piprot ./resources/libraries/requirements-minimal.txt
piprot ./resources/libraries/requirements-light.txt

Using pur:

pur -i -r ./resources/libraries/requirements-minimal.txt