DOCSEARCHbyAlgolia
Submit Docs

RoseLab Servers Guide

website·Indexed 29 days ago

https://rose-stl-lab.github.io/

Documentation for using RoseLab servers, featuring guides on multi-server workflows, data management best practices, running Docker in LXC containers, and configuring Jupyter Lab and Remote Desktop access.

Tokens
11.9K
Snippets
73
Records
105
Agent score
75%

What's inside RoseLab

  1. Overview of Roselab server architecture and benefits

    Roselab servers provide a development environment for machine learning that balances the need for resource isolation and persistence. By leveraging ZFS and Linux Containers, Roselab offers storage, CPU, memory, and network isolation with minimal performance overhead. Each user is granted root and desktop access to their server, simulating a personal workstation environment.

  2. Overview of CSE Shared Servers

    The CSE shared servers (north, south, east, and west) provide a containerized environment for research and development managed by RoseLab. Each user is assigned an isolated container with dedicated resources and sudo access. Unlike RoseLab, containers are fixed to a specific server and cannot be moved; data is not synced between servers.

  3. Manage a single project across multiple RoseLab servers

    When working on one project that requires resources from multiple servers, designate one server as the 'Main' server for primary development and latest code. Use other servers as 'Replicas' for parallel experiments. Ensure all code is committed to Git on the main server and pulled onto replicas before running experiments. Store all checkpoints and results in the shared /data directory, which is accessible from all servers.
    # On roselab1 (main server)
cd ~/projects/my-research
git pull
# ... do development work ...
git add .
git commit -m "Add new feature X"
git push

# On roselab2 (replica) - when starting experiments
cd ~/projects/my-research
git pull  # Get latest changes from roselab1
# ... run experiments ...
# Save checkpoints to /data

  4. Manage storage on RoseLab servers

    RoseLab provides shared storage and per-user directories. To maintain system performance, follow these best practices:

    1. Store large datasets on /data (synchronized across servers).
    2. Keep environments and code on the local SSD.
    3. Archive or compress old datasets and regularly clean up intermediate experiment results.
    4. Use the provided utility script /utilities/common-utilities.py to clean the pip cache if needed.

    To check your current storage usage, use the command df -h /data or df -h /public.

  5. Monitor system and container resources

    Users can monitor resource utilization via the command line or the Grafana Dashboard. The Grafana Dashboard (http://roselab1.ucsd.edu/grafana/) provides real-time metrics for GPU utilization per container, CPU/memory usage, network throughput, and storage I/O.
    # Check CPU usage
htop

# Check GPU usage
nvidia-smi

# Check storage usage
df -h /
df -h /data

# Check network storage speed (Expected: ~300+ MB/s write speed)
dd if=/dev/zero of=/data/testfile bs=1M count=1024
rm /data/testfile

  6. Purchase and add dedicated storage for a lab

    If the shared storage pool (128GB per user) is insufficient or performance is degraded due to I/O contention, labs can purchase their own dedicated SSDs. These drives provide exclusive access and better performance.

    Hardware Requirements:

    • Form factor: 2.5" NVMe SSD
    • Connector: U.2 or U.3 (strongly preferred over SATA)

    Recommended Models:

    • Consumer: Samsung 870 QVO/EVO (4TB or 8TB)
    • Enterprise: Samsung PM983/PM9A3 or Micron 7450/9400 series

    Process:

    1. Purchase a compatible SSD.
    2. Contact RoseLab admins to mount the drive and configure exclusive access for the lab.

  7. Update passwords for system and application accounts

    Depending on the service, different methods are used to update passwords. For the system/remote desktop account, use the command line. For Jupyter Lab, modify the configuration file. For Seafile, use the web settings. For SSH, update the authorized keys file.
    # System / Remote Desktop
sudo passwd ubuntu

    Jupyter Lab: Edit ~/.jupyter/jupyter_lab_config.py and change c.ServerApp.token

    SSH: Update ~/.ssh/authorized_keys with your new public key

  8. Inspect memory quota on RoseLab servers

    You can check the memory assigned to your container by inspecting the /proc/meminfo file. Memory quotas vary by server: roselab1-3 and roselab6-7 have standard quotas, roselab4 has 2x quota, and roselab5 has 4x quota.
    cat /proc/meminfo

  9. Add a new Conda environment to JupyterLab kernels

    To make a new Conda environment available as a kernel within Jupyter Lab, you must install ipykernel inside that environment and register it.
    # Create and activate environment
conda create -n py38 python=3.8
conda activate py38

# Install ipykernel and register the kernel
conda install -c anaconda ipykernel
python -m ipykernel install --user --name=py38

  10. Synchronize code across multiple servers using Git

    To prevent code divergence across multiple servers, use Git as the single source of truth. Establish a habit of pushing changes at the end of every work session. If servers have diverged, commit local changes on each server and use git pull --rebase or git pull to resolve conflicts before pushing to the remote repository.
    # On each server, commit local changes
git add .
git commit -m "Changes from roselab1"
git push

# If conflicts occur when pushing from second server
git pull --rebase
# Resolve conflicts
git push

# Or merge locally
git pull
# Resolve conflicts
git push

  11. Remove a JupyterLab kernel configuration

    To remove a specific environment's kernel from the Jupyter Lab interface, delete its configuration directory.
    rm -rf /home/ubuntu/.local/share/jupyter/kernels/py38

  12. Inspect memory quota on RoseLab servers

    You can check the memory assigned to your container by inspecting the /proc/meminfo file. Memory quotas vary by server: roselab1-3 and roselab6-7 have standard quotas, roselab4 has 2x quota, and roselab5 has 4x quota.
    cat /proc/meminfo