Pachyderm Documentation

Deploy Pachyderm using the pachyderm/pachyderm Helm chart. To activate Enterprise and RBAC authorization, you must provide the PACHYDERM_ENTERPRISE_KEY via the --set pachd.enterpriseLicenseKey flag.

For Docker Desktop

# Ensure you are in the root of the project
$ helm install \
        --wait --timeout 10m pachd pachyderm/pachyderm \
        --version=2.6.0 \
        -f enterpriseProxyHelmValues.yaml \
        --set pachd.metrics.enabled=false \
        --set pachd.enterpriseLicenseKey=$PACHYDERM_ENTERPRISE_KEY

For Kind

# Ensure you are in the root of the project
$ helm install \
        --wait --timeout 10m pachd pachyderm/pachyderm \
        --version=2.6.0 \
        -f enterpriseProxyHelmValues.yaml \
        --set pachd.metrics.enabled=false \
        -f kind.yaml \
        --set pachd.enterpriseLicenseKey=$PACHYDERM_ENTERPRISE_KEY

# Docker Desktop example
$ helm install \
        --wait --timeout 10m pachd pachyderm/pachyderm \
        --version=2.6.0 \
        -f enterpriseProxyHelmValues.yaml \
        --set pachd.metrics.enabled=false \
        --set pachd.enterpriseLicenseKey=$PACHYDERM_ENTERPRISE_KEY

When running user code in a pipeline with datum batching enabled, you should call NextDatum to signal that the current datum has been processed and the code is ready for the next one.

Important Notes:

• NextDatum is a synchronous operation; your code will block until the next datum is set up.
• The response NextDatumResponse contains an env field (a list of strings) representing environment variables that should be set for the next datum's processing.
• If processing a datum fails, you can pass the error string in the NextDatumRequest.error field. This allows the system to handle retries or job failures according to datum error semantics.

### Worker

| Method Name | Request Type | Response Type | Description |
| ----------- | ------------ | ------------- | ------------|
| Status | [.google.protobuf.Empty](#google-protobuf-Empty) | [.pps_v2.WorkerStatus](#pps_v2-WorkerStatus) |  |
| Cancel | [CancelRequest](#pachyderm-worker-CancelRequest) | [CancelResponse](#pachyderm-worker-CancelResponse) |  |
| NextDatum | [NextDatumRequest](#pachyderm-worker-NextDatumRequest) | [NextDatumResponse](#pachyderm-worker-NextDatumResponse) | NextDatum should only be called by user code running in a pipeline with datum batching enabled. NextDatum will signal to the worker code that the user code is ready to proceed to the next datum. This generally means setting up the next datum's filesystem state and updating internal metadata similarly to datum processing in a normal pipeline. NextDatum is a synchronous operation, so user code should expect to block on this until the next datum is set up for processing. User code should generally be migratable to datum batching by wrapping it in a loop that calls next datum.

For macOS users or those wanting a containerized environment, use the Dev Container workflow. This requires a running Pachyderm instance (e.g., via kubectl port-forward service/pachd 30650:30650).

1. Start the container:

   docker run --name jupyterlab_pachyderm_frontend_dev \
   -p 8888:8888 \
   -it -e GRANT_SUDO=yes --user root \
   --device /dev/fuse --privileged \
   -v $(pwd):/home/jovyan/extension-wd \
   -w /home/jovyan/extension-wd \
   pachyderm/notebooks-user:<latest master SHA> \
   bash

 *Note: On Linux, add `--net=host` to talk to minikube. On macOS, use `grpc://host.docker.internal:30650` instead of localhost.*

2. **Inside the container**, install and start:
 ```bash
pip install -e ".[dev]"

 jupyter labextension develop --overwrite

 jupyter server extension enable jupyterlab_pachyderm

 jupyter lab --allow-root

3. Watch for changes: In a separate terminal (via docker exec), run:

   npm run watch

```bash
# Example docker run command
docker run --name jupyterlab_pachyderm_frontend_dev \
  -p 8888:8888 \
  -it -e GRANT_SUDO=yes --user root \
  --device /dev/fuse --privileged \
  -v $(pwd):/home/jovyan/extension-wd \
  -w /home/jovyan/extension-wd \
  pachyderm/notebooks-user:<latest master SHA> \
  bash