Apache Airflow Python Client

repository·main·Indexed 19 days ago

https://github.com/apache/airflow-client-python

An automatically generated Python client library for interacting with the Apache Airflow Stable REST API. It enables programmatic management of Airflow resources, including DAGs, connections, task runs, assets, backfills, and XComs, using standard Python patterns. The library supports CRUD operations, Bearer token authentication, and provides specific API classes such as DAGApi, DagRunApi, and TaskInstanceApi.

Tokens
192.2K
Snippets
416
Records
500
Agent score
65%

What's inside apache-airflow-client-python

  1. Understand the AssetCollectionResponse model

    main

    The AssetCollectionResponse model represents a collection of assets returned by the API. It contains a list of individual asset responses and metadata about the total number of entries in the collection.

    Properties:

    • assets: A List[AssetResponse] containing the actual asset data.
    • total_entries: An int representing the total count of assets in the collection.
  2. Identify mutable DagRun states

    main

    When performing mutation operations (such as PATCH or DELETE) on a DagRun via the API, you can only target runs that are currently in one of the following mutable states:

    • QUEUED (value: 'queued')
    • SUCCESS (value: 'success')
    • FAILED (value: 'failed')

    Attempting to mutate a DagRun in a state not included in this list will likely result in an error.

  3. Understand the BulkResponse model

    main

    The BulkResponse class is a serializer used for responses to bulk entity operations. It aggregates the results of create, update, and delete actions performed on multiple entities in a single request.

    Each action is represented by a field containing a BulkActionResponse object, which details successful keys and any errors encountered during that specific operation.

    Important Behavior: Fields (create, delete, or update) are only populated in the response if the respective action was explicitly requested in the original bulk operation. If an action was not part of the request, its corresponding field is set to None.

  4. Configure BulkActionOnExistence behavior

    main

    When performing bulk operations in the Airflow Python Client, you can specify how the client should handle an entity that already exists using the BulkActionOnExistence enum. This determines whether the operation should fail, skip the existing entity, or overwrite it.

    BulkActionOnExistence values:
    - `FAIL` (value: `'fail'`): The operation fails if the entity exists.
    - `SKIP` (value: `'skip'`): The operation ignores the existing entity and continues.
    - `OVERWRITE` (value: `'overwrite'`): The existing entity is replaced by the new data.
  5. Understand TaskInstanceCollectionResponse pagination

    main

    The TaskInstanceCollectionResponse model is used to handle collections of task instances. It supports both offset and cursor based pagination.

    Because of limitations in certain TypeScript code generators (like @hey-api/openapi-ts), the model uses a single flat structure rather than a discriminated union to ensure type safety and prevent return types from degrading to unknown in environments like JSDoc or React Query.

  6. Understand the TaskInstanceHistoryCollectionResponse model

    main

    The TaskInstanceHistoryCollectionResponse is a serializer model used for responses containing a collection of task instance history records. It encapsulates a list of individual history entries and metadata about the total count of entries available.

    Properties

    NameTypeDescription
    task_instancesList[TaskInstanceHistoryResponse]A list of task instance history response objects.
    total_entriesintThe total number of entries in the collection.
  7. Configure reprocess behavior for backfills using ReprocessBehavior

    main

    When performing a backfill, you can specify how the system should handle existing task instances using the ReprocessBehavior enum. This determines whether the backfill should overwrite or ignore tasks that have already been run.

    Available values:

    • FAILED: Reprocess tasks that have a FAILED state.
    • COMPLETED: Reprocess tasks that have a COMPLETED state.
    • NONE: Do not reprocess any existing tasks.
    # Example values for ReprocessBehavior
    REPROCESS_FAILED = 'failed'
    REPROCESS_COMPLETED = 'completed'
    REPROCESS_NONE = 'none'
  8. Perform CRUD operations on Airflow resources

    main

    The API supports standard Create, Read, Update, and Delete operations. Most endpoints require Content-type: application/json and Accept: application/json headers.

    OperationHTTP MethodSuccess CodeDescription
    CreatePOST201 CreatedSubmits metadata in the request body. Returns the new resource including its id.
    ReadGET200 OKUse a specific id to read one resource, or omit the id to list multiple resources.
    UpdatePATCH200 OKRequires the resource id. Submit only the fields to modify in the request body.
    DeleteDELETE204 No ContentRequires the resource id.
  9. Use TaskInstanceState to track task execution status

    main

    The TaskInstanceState enum defines all possible states a Task Instance can occupy within Apache Airflow. When working with Task Instance objects in the Python client, you should use this enum to check or set the status.

    Important Note on Type Hinting: Because a Task Instance state can be None, always use Optional[TaskInstanceState] when defining type hints for variables or function arguments that handle task states.

  10. Identify the source of asset state updates with AssetStateStoreWriterKind

    main

    The AssetStateStoreWriterKind enum is used to identify the mechanism or entity that last updated an entry in the asset state store. This allows you to distinguish between automated task executions, event triggers, and manual administrative actions.

    AssetStateStoreWriterKind values:
    - `TASK` (value: `'task'`): Written by a task via the execution API.
    - `WATCHER` (value: `'watcher'`): Written by a `BaseEventTrigger` (no task instance).
    - `API` (value: `'api'`): Written directly through the Core API (e.g., manual admin write).
  11. Use BulkTaskInstanceBody for bulk task instance operations

    main

    The BulkTaskInstanceBody class is used as the request body for performing bulk update and delete operations on task instances. It allows you to specify a target task instance (via dag_id, dag_run_id, task_id, and optionally map_index) and define how the operation affects related tasks using inclusion flags (include_downstream, include_future, include_past, include_upstream). You can also specify a new_state and a note for the operation.

    from airflow_client.client.models.bulk_task_instance_body import BulkTaskInstanceBody
    
    # Example of creating an instance (using empty dict as placeholder)
    bulk_task_instance_body = BulkTaskInstanceBody(
        dag_id="my_dag",
        dag_run_id="manual__2023-01-01T00:00:00",
        task_id="my_task",
        new_state="success",
        include_downstream=True
    )
  12. Understand the HITLDetailResponse model

    main

    The HITLDetailResponse model represents the response received when updating a Human-in-the-loop (HITL) detail. It contains information about the user's response, the options chosen, and the input parameters used.

    from airflow_client.client.models.hitl_detail_response import HITLDetailResponse
    
    # Example properties:
    # chosen_options: List[str]
    # params_input: Dict[str, object] (optional)
    # responded_at: datetime
    # responded_by: HITLUser