Datashim Documentation

Datashim is a Kubernetes framework that provides easy access to S3 and NFS Datasets within pods. It introduces a Dataset Custom Resource Definition (CRD) which acts as a pointer to existing S3 or NFS data sources.

When you define a Dataset, Datashim orchestrates the provisioning of the necessary Persistent Volume Claims (PVCs) and ConfigMaps. This allows users to reference data in their pods without manually configuring, mounting, or tuning data access. The framework is extensible via the Container Storage Interface (CSI) to support additional data sources in the future.

Beyond simplifying data access, the Dataset CRD enables higher-level orchestration and future extensions in two key areas:

• Performance: Datashim provides a pluggable caching interface. An example implementation is the Ceph Caching Plugin.
• Security: The framework is exploring a common access management layer to handle credentials across different types of data sources uniformly.

Datashim introduces a new Custom Resource Definition (CRD) called the Dataset. This abstraction provides a declarative way to reference existing data sources (currently supporting S3 and NFS) without requiring users to manually manage complex CSI (Container Storage Interface) configurations.

When a Dataset is created, the Datashim Operator automatically handles the following:

1. Invokes the appropriate CSI plugin for the data source.
2. Configures and provisions a Persistent Volume Claim (PVC).
3. Provides a mount-point in the user's pod for the data.
4. Exposes an interface for caching mechanisms to leverage.

Datashim is designed as a meta-framework for CSI plugins rather than a replacement for them.

• Relationship to CSI: Datashim sits on top of existing CSI implementations. Every data source supported by Datashim uses its own standalone CSI driver. Datashim simplifies the user experience by automating the invocation and configuration of these drivers.
• Relationship to COSI: Datashim does not compete with the Container Object Storage Interface (COSI) proposal. While COSI manages the full lifecycle of a bucket (provisioning, access configuration, etc.), Datashim focuses specifically on providing a mount-point for existing COS buckets.

To mount a Datashim Dataset into a Pod, add specific labels to the Pod's metadata. The labels follow the pattern dataset.<index>.id and dataset.<index>.useas.

For example, to mount a dataset named model-weights as a volume, use:

• dataset.0.id: "model-weights"
• dataset.0.useas: "mount"

Once mounted, the files from the object storage bucket will be available at /mnt/datasets/<dataset-name>/ within the container.

apiVersion: v1
kind: Pod
metadata:
  name: text-generation-inference
  labels:
    dataset.0.id: "model-weights"
    dataset.0.useas: "mount"
spec:
  containers:
    - name: text-generation-inference
      # ...

Datashim is a Kubernetes framework that provides easy access to S3 and NFS Datasets within pods. It uses a Dataset Custom Resource Definition (CRD) which acts as a pointer to existing S3 or NFS data sources.

When you define a Dataset, Datashim orchestrates the provisioning of the necessary Persistent Volume Claims (PVCs) and ConfigMaps. This allows users to reference datasets in their pods without manually configuring, mounting, or tuning data access. The system is extensible via the Container Storage Interface (CSI) to support additional data sources.

To use the pod labeling method (e.g., dataset.0.id), you must label the namespace(s) where your pods reside to enable the Datashim pod labeling functionality.

Example for the default namespace:

kubectl label namespace default monitor-pods-datasets=enabled

To run a distributed training job as a TFJob, you must build a Docker image containing your model code (e.g., model.py) and a Dockerfile. Use the following commands to build and push the image to your registry:

cd examples/kubeflow
docker build -t {MY-REGISTRY}/mnist-model -f Dockerfile.model .
docker push {MY-REGISTRY}/mnist-model

If using an authenticated registry, ensure your Docker credentials are configured.

To access data stored in an S3-compatible bucket, you must perform two steps:

1. Create a Kubernetes Secret containing your access credentials (accessKeyID and secretAccessKey).
2. Create a Dataset object of type COS that references that secret and defines the bucket and endpoint.

In the example, the Dataset is named model-weights and uses the local.provision: "true" setting to enable mounting.

apiVersion: v1
kind: Secret
metadata:
  name: model-weights-secret
stringData:
  accessKeyID: "ACCESS_KEY"
  secretAccessKey: "SECRET_KEY"
---
apiVersion: datashim.io/v1alpha1
kind: Dataset
metadata:
  name: model-weights
spec:
  local:
    provision: "true"
    bucket: my-model
    endpoint: http://minio.minio.svc.cluster.local:9000
    secret-name: model-weights-secret
    type: COS