DOCSEARCHbyAlgolia
Submit Docs

DataHub Helm Charts

repository·master·Indexed 41 days ago

https://github.com/acryldata/datahub-helm

Helm charts for deploying DataHub and its required infrastructure, including Kafka, Elasticsearch, MySQL, and PostgreSQL, on Kubernetes clusters. Includes configuration for the Generalized Metadata Service (GMS), frontend HPA settings, ingestion cronjobs, and global service endpoints.

Tokens
54.5K
Snippets
79
Records
118
Agent score
52%

What's inside DataHub

  2. Overview of DataHub Helm Chart Values Reference

    master

    This document serves as the comprehensive reference for all configurable values within the DataHub Helm chart. It covers global configurations, Elasticsearch settings, semantic search, and object storage.

    Important Version Defaults (v1.7.0):

    • global.datahub.systemUpdate.zdu.enable: Defaults to true. When enabled (ZDU_STAGE_20), scale-down operations are suppressed while ZDU is active.
    • global.datahub.i18n.enabled: Defaults to true.
    • global.datahub.encryptionKey.callerGuardMode: Defaults to ENFORCE. In this mode, browser/user-PAT secret decryption is blocked. To bypass this, use datahub-actions with system credentials.
    • datahub-gms.theme_v2.toggeable: Defaults to false (enables V2-only UI).
    • Object Storage Migration: Use global.datahub.objectStorage instead of the legacy global.datahub.gms.s3 configuration.

  5. DataHub Architecture and Components

    master

    DataHub is deployed as a set of subcharts under the main datahub Helm chart.

    Main Components

    • GMS (Metadata Service): The core service.
    • MAE Consumer (Optional): Metadata Action Event consumer.
    • MCE Consumer (Optional): Metadata Change Event consumer.
    • Frontend: The user interface.

    External Dependencies

    The components rely on the following infrastructure, which should be deployed via the datahub-prerequisites chart or managed externally:

    • Kafka
    • Local DB (MySQL, Postgres, or MariaDB)
    • Search Index (Elasticsearch)
    • Graph Index (Supports Elasticsearch or Neo4j)

  6. Configure Semantic Search (Beta)

    master

    Semantic search enables vector similarity search, allowing users to find entities based on meaning rather than just keywords.

    Requirements:

    • Elasticsearch or OpenSearch with the k-NN plugin installed.
    • Documents must have embeddings.
    • Note: Currently in beta; only the document entity type is officially supported.

    Key Configuration Parameters:

    • global.datahub.semantic_search.enabled: Set to true to enable.
    • global.datahub.semantic_search.enabledEntities: Comma-separated list of entity types (default: "document").
    • global.datahub.semantic_search.vectorDimension: CRITICAL. This must exactly match your embedding model's output dimension. Mismatches will cause semantic search to fail.

    Dimension Reference:

    • text-embedding-3-large: 3072
    • text-embedding-3-small: 1536
    • Cohere/Bedrock: 1024

  7. Configure Global Entity Graph Cache

    master

    The Entity Graph Cache is an opt-in GMS-only feature for caching domain/container/glossary hierarchies and group membership.

    Usage Notes:

    • The chart disables this by default (false).
    • Enabling it sets ENTITY_GRAPH_CACHE_ENABLED=true on datahub-gms.
    • Requirement: Requires a GMS image that includes the entity graph cache feature.
    • Scaling: When enabled on a single-replica GMS, the chart provisions Hazelcast (headless service, port 5701) to support the cache.

    Configuration:

    • global.datahub.entity_graph_cache.enabled: Boolean to enable/disable.
    • global.datahub.entity_graph_cache.configJson: An optional JSON string used as an overlay at startup. This sets ENTITY_GRAPH_CACHE_CONFIG_JSON on GMS to configure parameters like per-graph bounds and population.intervalSeconds.

  8. Quickstart: Deploy DataHub on Kubernetes

    master

    Follow these steps to perform a standard deployment using the DataHub Helm charts. This assumes your kubectl context is already pointing to the target cluster.

    1. Create Kubernetes Secrets

    Create secrets for MySQL and Neo4j passwords. The example below uses datahub as the password; replace this with your preferred credentials.

    2. Add the DataHub Helm Repository

    Add the official repository to your Helm client.

    3. Deploy Prerequisites

    Deploy the required dependencies (Kafka, MySQL, Elasticsearch, etc.) using the datahub-prerequisites chart. You can use default values or provide a custom --values file.

    4. Deploy DataHub

    Install the main DataHub chart. Note that the default values.yaml in the datahub chart is pre-configured to look for dependencies deployed with the release name prerequisites. If you used a different release name for the prerequisites, you must update your configuration accordingly.

    5. Access the Frontend

    Once all pods are Running, use kubectl port-forward to access the UI locally on port 9002.

    # 1. Create secrets
kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=datahub --from-literal=mysql-password=datahub
kubectl create secret generic neo4j-secrets --from-literal=neo4j-password=datahub --from-literal=NEO4J_AUTH=neo4j/datahub

# 2. Add repo
helm repo add datahub https://helm.datahubproject.io/

# 3. Install prerequisites
helm install prerequisites datahub/datahub-prerequisites
# Or with custom values:
# helm install prerequisites datahub/datahub-prerequisites --values <path-to-values-file>

# 4. Install DataHub
helm install datahub datahub/datahub

# 5. Port forward frontend (replace <pod-name> with your actual frontend pod name)
kubectl port-forward <datahub-frontend-pod-name> 9002:9002

  9. Configure DataHub Ingestion Cron Jobs

    master

    The datahub-ingestion-cron subchart allows you to schedule automated DataHub ingestion tasks using Kubernetes CronJobs. You define multiple ingestion tasks within the crons map, where each key in the map becomes the name of a unique CronJob.

    Recipe Configuration

    Every cron job requires a recipe. You can provide the recipe in two ways:

    1. Via ConfigMap: Mount an existing Kubernetes ConfigMap containing your recipe. Use crons.recipe.configmapName to specify the name and crons.recipe.fileName to specify the key within that ConfigMap.
    2. Via Inline Content: Provide the recipe directly using crons.recipe.fileContent (if supported by your deployment logic).

    Key Configuration Options

    • crons.schedule: A standard Cron expression (e.g., "0 0 * * *" for daily at midnight) that determines when the ingestion runs.
    • crons.command: An array of strings defining the command to run. By default, it executes datahub ingest -c /etc/recipe/<fileName> based on your recipe configuration.
    • crons.env and crons.envFromSecrets: Used to inject environment variables or secrets required for the ingestion process (e.g., credentials).
    • crons.resources: Define CPU and memory requests/limits for the ingestion pods.
    • crons.concurrencyPolicy: Controls how concurrent executions are handled (e.g., Allow, Forbid, or Replace).
    crons:
  my-snowflake-ingestion:
    schedule: "@daily"
    recipe:
      configmapName: "ingestion-recipes"
      fileName: "snowflake-recipe.yaml"
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 200m
        memory: 512Mi

  10. Install DataHub Prerequisites via Helm

    master

    The datahub-prerequisites Helm chart installs the necessary components required to run DataHub.

    To install with the default configuration, add the DataHub Helm repository and install the chart using the following commands:

    helm repo add datahub https://helm.datahubproject.io
helm install datahub datahub/datahub-prerequisites

    If you need to customize the installation, create a values.yaml file with your specific configurations and provide it during installation using the --values flag.

  11. Install DataHub via Helm

    master

    To install DataHub using the official Helm chart with default configurations, add the DataHub repository to your Helm client and run the install command. If you need to customize the installation, provide a custom values.yaml file using the --values flag.

    # Add the DataHub Helm repository
helm repo add datahub https://helm.datahubproject.io

# Install DataHub with default configuration
helm install datahub datahub/datahub

# Install DataHub with custom configuration
helm install datahub datahub/datahub --values <path-to-values-file>

  12. Enable Semantic Search (Beta) in DataHub

    master

    Semantic search enables vector similarity search, allowing users to find entities based on semantic meaning rather than just keyword matching.

    Prerequisites:

    • An Elasticsearch or OpenSearch cluster with the k-NN plugin enabled.
    • Documents must have embeddings generated.
    • Credentials for an embedding provider (OpenAI, AWS Bedrock, or Cohere).

    Note: This is a beta feature. Currently, only the document entity type is officially supported.