DOCSEARCHbyAlgolia
Submit Docs

Kestra Orchestration Platform

repository·develop·Indexed 12 days ago

https://github.com/kestra-io/kestra

An open-source, event-driven orchestration platform for data, AI, and infrastructure workflows. Kestra uses a declarative YAML interface and a visual UI to manage scalable pipelines. It supports standalone and distributed deployment modes, including gRPC-based worker-controller connections in version 2.0, and provides comprehensive Helm charts for Kubernetes deployment with support for Docker-in-Docker (DinD).

Tokens
60.4K
Snippets
167
Records
207
Agent score
98%

What's inside Kestra

  1. Explore the Kestra plugin ecosystem

    develop

    Kestra's functionality is extended via plugins, allowing you to integrate with various services and languages. Key capabilities include:

    • Multi-language Scripting: Run code in Python, Node.js, R, Go, Shell, and more.
    • Cloud Integrations: Native support for AWS, Google Cloud, and Azure services (storage, messaging, compute).
    • Event-Driven Triggers: React to real-time events from Kafka, Redis, Pulsar, AMQP, MQTT, NATS, AWS SQS, Google Pub/Sub, and Azure Event Hubs.
    • Task Runners: Execute tasks locally, on remote servers via SSH, or scale out using Docker and Kubernetes.

  2. Use the @kestra-io/kestra-sdk for JS/TS

    develop

    The @kestra-io/kestra-sdk is the official JavaScript/TypeScript client for the Kestra OSS API. It is generated directly from the backend's OpenAPI specification, ensuring that the client and the backend are always in sync (living on the same commit).

    Key characteristics:

    • Source of Truth: Generated from io.kestra:webserver via OpenAPI spec.
    • Generator: Uses @hey-api/openapi-ts with @hey-api/client-fetch and the @kestra-io/hey-api-plugin for tenant-aware, human-friendly wrappers.
    • Client Interface: Provides an axios-like facade over fetch. You can use useClient().get(...) or useClient().post(...) to interact with the API, utilizing the same interceptors used by the main application.
    // Example usage pattern (conceptual based on runtime description)
import { useClient } from '@kestra-io/kestra-sdk';

const client = useClient();
const response = await client.get('/api/v1/executions');

  3. Follow metric naming conventions

    develop

    To ensure metrics group correctly in alphabetical registries and are easy to grep, follow these naming rules:

    1. Structure: Use <system>.<subject>.<qualifier> (e.g., controller.worker.active, not controller.active.worker).
    2. Counters: Always end with .total. Never use .count or a bare name.
    3. Gauges: Use a noun and never use .total or .count suffixes. If expressing a cluster-wide value, use .all or .global.
    4. Timers: Suffix with the unit or .duration (e.g., task.execution.duration). Do not add .count as it is auto-emitted.
    5. Units: Use base units (seconds, bytes) and include the unit in the name if not obvious (e.g., _bytes, _seconds, _ratio). Do not use milliseconds or kilobytes.
    6. Consistency: Use consistent verb tense (e.g., always ended or always end) and do not mix dot and underscore separators in a single name.

  4. Use Pebble expressions in Kestra flows

    develop

    Kestra supports Pebble expressions within flow properties, task properties, and plugin configurations. These expressions allow you to dynamically access flow metadata, task outputs, and variables at runtime.

    Commonly used Pebble expressions include:

    • {{ flow.id }}: The unique identifier of the current flow.
    • {{ flow.name }}: The name of the current flow.
    • {{ execution.id }}: The unique identifier of the current execution.
    • {{ execution.status }}: The status of the current execution.
    • {{ taskinput.name }}: Accesses input parameters defined for a specific task.
    • {{ outputs.task_id.variable_name }}: Accesses the output of a previously completed task using its id and the specific output key.
    # Example of using Pebble expressions in a flow
flow:
  id: my_flow
  tasks:
    - id: hello_task
      type: io.kestra.plugin.core.log.Log
      message: "Hello, {{ flow.name }}!"
    - id: use_output
      type: io.kestra.plugin.core.log.Log
      message: "The previous task output was: {{ outputs.hello_task.message }}"

  5. Query data for Kestra Dashboards

    develop

    Dashboards query data through the data property of a chart. The type determines the available columns and the nature of the data being retrieved.

    Supported Data Sources

    • io.kestra.plugin.core.dashboard.data.Executions: Data related to workflow executions.
    • io.kestra.plugin.core.dashboard.data.Logs: Logs produced by executions.
    • io.kestra.plugin.core.dashboard.data.Metrics: Metrics emitted during executions.

    Column Configuration

    After selecting a data source, you define columns to display. Each column is mapped to a field from the data source. Supported properties for columns include:

    • field (Required): The name of the column in the data source.
    • displayName: The label shown in the chart.
    • agg: Aggregation function (AVG, COUNT, MAX, MIN, SUM).
    • graphStyle: Visual style for graphs (LINES, BARS, POINTS).
    • columnAlignment: Table alignment (LEFT, RIGHT, CENTER).

  6. Define a Kestra flow using YAML

    develop

    Kestra flows are defined using YAML syntax. A valid flow must include an id, a namespace, and a list of tasks.

    Optional properties you can include to enhance your flow include:

    • triggers: Define when a flow should start automatically.
    • labels: Metadata for organizing and filtering flows.
    • inputs: Define parameters that can be passed to the flow at runtime.
    id: my-flow
namespace: company.team
tasks:
  - type: io.kestra.plugin.core.log.Log
    message: Hello World

  7. Use Pebble Templating in Kestra

    develop

    Kestra uses the Pebble templating engine to allow dynamic rendering of variables, inputs, and outputs. You can use Pebble expressions within any string field to access the execution context.

    Commonly used context variables include:

    • {{ inputs.id }}: Accesses a specific flow input.
    • {{ vars.name }}: Accesses a flow variable.
    • {{ tasks.id.state }}: Accesses the state of a specific task.
    • {{ flow.id }}: Accesses the current flow ID.
    • {{ flow.namespace }}: Accesses the current flow namespace.
    • {{ execution.state }}: Accesses the current execution state.
    • {{ taskrun.startDate }}: Accesses the start date of a task run.

    For debugging, you can use the {{ printContext() }} function to print the entire execution context.

  8. Stamp OpenAPI spec hashes during codegen

    develop

    To prevent SDK drift, you can pass specPath (the path to your raw OpenAPI spec file) to the codegen plugin. The plugin will automatically calculate a SHA256 hash of the file and export it as export const OPENAPI_SPEC_HASH = sha256(specFile)[:16] within the generated SDK.

    This allows developers to perform staleness checks by comparing the checked-in hash against the live backend spec's hash.

  9. Use Template Tags for Logic and Control Flow

    develop

    Kestra templates support several control flow tags to manage logic within your workflows:

    • {% set ... %}: Define variables in the current template context.
    • {% if ... %}: Execute conditional blocks. Supports {% elseif %} and {% else %} branches.
    • {% for ... %}: Iterate over collections. You can use an {% else %} block to handle empty collections.
    • {% macro ... %}: Define reusable template fragments that can be invoked like functions.
    • {% block ... %}: Define named blocks for template inheritance.
    • {% raw ... %}: Wrap content in {% raw %} and {% endraw %} to prevent the template engine from parsing the syntax inside (useful for writing code that contains Kestra expressions).
    {# Variable definition #}
{% set header = "Test Page" %}

{# Conditional logic #}
{% if category == "news" %}
  News Content
{% elseif category == "sports" %}
  Sports Content
{% else %}
  General Content
{% endif %}

{# Iteration #}
{% for user in users %}
  {{ loop.index }} - {{ user.id }}
{% else %}
  No users found
{% endfor %}

{# Reusable macro #}
{% macro input(type, name) %}
  <input type="{{ type }}" name="{{ name }}">
{% endmacro %}
{{ input("text", name="Mitchell") }}

{# Preventing parsing #}
{% raw %}{{ user.name }}{% endraw %}

  10. Access and manipulate variables using expressions

    develop

    Kestra uses expressions within {{ ... }} delimiters to access and manipulate data. You can use dot notation for standard attribute access, subscript notation for keys containing special characters, and string interpolation for embedding variables within literals.

    Common Expression Patterns

    • Attribute Access: Use {{ variable.attribute }} to access a child attribute of an object.
    • Subscript Notation: Use {{ variable['key-name'] }} when the attribute name contains special characters (like hyphens) that would break standard dot notation.
    • String Interpolation: Use #{variable} inside a string literal to inject a value, for example: {{ "Hello #{name}" }}.
    {{ foo.bar }}             // Accesses child attribute
{{ foo['my-key'] }}      // Accesses attribute with special characters
{{ "Hello #{who}" }}   // String interpolation

  11. Configure metric labels and cardinality

    develop

    Labels (tags) allow for multidimensional data, but must be managed carefully to prevent performance issues.

    • Bounded Cardinality: Avoid labels with unbounded or user-controlled values (e.g., execution IDs, full URLs, or free-form text). Every unique label-value combination creates a new time series.
    • Acceptable Sources: Use flow/namespace/tenant identifiers, enum-valued status fields, or Kestra component names.
    • Naming vs. Labels: Do not embed data that belongs in a label into the metric name.
      • Bad: worker_running_us_east_1
      • Good: worker_running{region="us-east-1"}
    • Global Tags: Use shared global tags from GlobalTagsConfigurer instead of re-tagging at every call site.

  12. Core Kestra concepts

    develop

    To use Kestra effectively, understand these fundamental building blocks:

    • Flows: The core unit of orchestration, representing a complete workflow composed of multiple tasks.
    • Tasks: The individual units of work within a flow (e.g., running a script, moving data, or calling an API).
    • Namespaces: A logical grouping used to organize and isolate flows.
    • Triggers: Mechanisms that initiate flow execution, such as a specific schedule or an external event.
    • Inputs & Variables: Parameters and dynamic data that can be passed into flows and tasks for customization and data flow.