repository·main·Indexed 23 days ago
https://github.com/astronomer/dag-factoryA library for Apache Airflow® that enables the declarative construction of DAGs via YAML configuration files. It allows users to manage complex workflows, including those using the TaskFlow API and traditional operators, without writing Python code. Compatible with Python 3.10.0+, Apache Airflow 2.9+, and Apache Airflow 3.
Telemetry is disabled by default on specific Astronomer environments via the SCARF_NO_ANALYTICS=True environment variable:
Telemetry is disabled by default in all deployments, regardless of the Astro Runtime version, by setting both SCARF_NO_ANALYTICS=True and DO_NOT_TRACK=True.
An Astro project generated via astro dev init contains the following directory structure and configuration files:
dags/: Contains Python files for your Airflow DAGs.include/: Directory for additional files required by your project.plugins/: Directory for custom or community Airflow plugins.Dockerfile: Contains the versioned Astro Runtime Docker image. Use this to specify runtime overrides or additional commands.packages.txt: List OS-level packages to be installed in your environment.requirements.txt: List Python packages to be installed via pip.airflow_settings.yaml: A local-only file used to specify Airflow Connections, Variables, and Pools without using the Airflow UI.When defining workflows in DAG Factory using YAML, you use several core keys to structure your DAG. The primary elements are:
dag_id: A unique identifier for your DAG.default_args: A dictionary of common arguments applied to all tasks within the DAG.schedule: The execution schedule for the DAG.tasks: A definition of the Airflow tasks included in the workflow.task_groups: A definition of Airflow task groups used to organize related tasks.DAG Factory allows you to represent Airflow workflows that use the TaskFlow API (Python decorators like @task) using YAML instead of Python.
While the Airflow TaskFlow API is designed to transform Python functions into tasks with intuitive data passing, DAG Factory achieves the same result by using the underlying Airflow operators in a YAML declaration.
Key differences and behaviors:
You can use logical operators within the schedule key of a consumer DAG to create complex trigger conditions. This requires dag-factory 0.22.0+ and Apache Airflow 2.9+.
Supported logical operators:
&): The DAG triggers only after all specified datasets have been updated.|): The DAG triggers when any of the specified datasets is updated.These operators allow you to define dynamic workflows where a DAG might depend on multiple datasets being ready simultaneously or just one of several options being available.
DAG-Factory allows you to define how your DAGs are triggered using the schedule field in your YAML configuration. To use scheduling, you must specify a type and provide the configuration under a value key.
Important Rules:
type.value key.Supported Schedule Types:
cron: Standard cron string (e.g., 0 0 * * *).timedelta: Fixed time intervals (e.g., every 6 hours).relativedelta: Calendar-aware intervals (e.g., every 1st of the month).timetable: Advanced Airflow timetables for custom trigger logic.assets: Triggering based on the readiness of Airflow assets.datasets: Triggering based on the readiness of Airflow datasets.DAG Factory can automatically pick up a defaults.yml file located in the same directory as your DAG YAML file. You can also use defaults_config_path to point to a parent directory to implement a hierarchical configuration.
Precedence Order in Hierarchy:
Arguments in a defaults.yml file closer to the DAG YAML file take precedence over those in files higher up the directory tree.
Example Directory Structure:
sample_project
└── a
├── b
│ ├── c
│ │ ├── defaults.yml <-- Highest precedence for DAGs in 'c'
│ │ └── some_dags.yml
│ └── defaults.yml <-- Medium precedence
└── defaults.yml <-- Lowest precedenceDAG-Factory allows you to instantiate native Python objects directly within your YAML configuration files. This enables you to use advanced Airflow features (like Kubernetes pod overrides or specific operator configurations) that require complex Python types without writing additional Python code.
To define a custom object, use the special __type__ key. The value of __type__ must be the fully qualified type name (e.g., module.submodule.ClassName). You can then define other keys within that same YAML block to represent the attributes/arguments of that Python object. This process is recursive, meaning you can nest custom objects within each other.
DAG Factory allows you to define workflows using YAML instead of Python, while still utilizing the same underlying Airflow operators (e.g., BashOperator, PythonOperator, KubernetesPodOperator).
DAG Factory allows you to define complex Python objects (like datetime, timedelta, or Airflow Asset objects) directly in your YAML configuration. This is achieved using a generalized object syntax where you specify the class via a __type__ key.
__type__: The full Python import path of the class (e.g., datetime.datetime).__args__: A list used for passing positional arguments to the class constructor.__type__ is treated as a keyword argument or attribute for the object.__type__: builtins.list and provide an items key containing the list elements.object_name:
__type__: <python.import.path.ClassName>
__args__:
- <positional_arg1>
<keyword_arg1>: <value1>Airflow Params allow you to pass dynamic configuration to tasks within a DAG at runtime. This enables tasks to be flexible by using templated values that are injected during execution.
Key capabilities include:
Use params when you need to reuse the same DAG with different input values or when you need to change a task's behavior dynamically during execution.