website·Indexed 40 days ago
https://docs.dagster.ioComprehensive command reference for the Dagster CLI and the `dg` CLI. Includes documentation for managing assets, jobs, runs, schedules, and sensors, as well as specific tools for Dagster Plus such as `dg api` for managing agents, alert policies, deployments, and organization settings.
The dagster-wandb library integrates Dagster with Weights & Biases (W&B) to orchestrate MLOps pipelines. Key capabilities include:
wandb client directly within Dagster ops and assets.dagster.DagsterInstance is the core abstraction for managing Dagster's access to storage and other resources. It coordinates how run metadata, event logs, and artifacts are stored and retrieved. Configuration is primarily handled via the dagster.yaml file located in the $DAGSTER_HOME directory.# Example dagster.yaml for Postgres storage
storage:
postgres:
postgres_db:
username: my_username
password: my_password
hostname: my_hostname
db_name: my_database
port: 5432dagster library provides the orchestration-side APIs, while the separately installed dagster-pipes library is used for writing code that runs within the external process.dagster-sigma library allows you to represent workbooks and datasets from a Sigma project as Dagster assets. This integration enables the visualization of relationships and dependencies between Sigma assets and other technologies like dbt and Sling.dagster.PythonArtifactMetadataValue class is used to store information about a Python artifact, requiring the module (string) where the artifact is found and the name (string) of the artifact.dagster instance info command to list detailed information and configuration settings regarding the current Dagster instance.dagster instance infodagster.PartitionedConfig class allows a job to be run on a discrete set of partitions, where each partition corresponds to a specific run configuration. This enables the launching of backfills and the tracking of run history across different partitions.dagster-papertrail library allows Dagster jobs to send logs to a Papertrail account. To set this up, you require an active Papertrail account, the Papertrail URL, and the associated port.AirflowInstanceComponent connects to an Airflow instance to retrieve metadata about DAGs and tasks, creating corresponding Dagster assets. It can map Airflow tasks to existing Dagster assets or create new assets to represent Airflow workflows.# defs.yaml
type: dagster_airlift.core.AirflowInstanceComponent
attributes:
name: my_airflow_instance
auth:
type: basic_auth
webserver_url: "{{ env.AIRFLOW_WEBSERVER_URL }}"
username: "{{ env.AIRFLOW_USERNAME }}"
password: "{{ env.AIRFLOW_PASSWORD }}"
filter:
dag_id_ilike: "analytics_%"
retrieve_datasets: trueDagster provides specific classes to represent information about the inputs and outputs that a graph maps:
dagster.GraphIn: Represents a graph input. It accepts an optional description (str).dagster.GraphOut: Represents a graph output. It accepts an optional description (str).dagster-datahub library provides an integration to push metadata from Dagster ops to a Datahub instance. This integration relies on the Datahub Python Library and requires a running Datahub instance.The OpExecutionContext provides several properties to retrieve information about the assets and partitions associated with the current execution:
asset_key: Returns the AssetKey for the current asset. For multi_asset definitions, use asset_key_for_output instead.assets_def: Returns the backing AssetsDefinition for the current execution (errors if unavailable).has_assets_def: Boolean indicating if a backing AssetsDefinition exists.has_partitions: Boolean indicating if the current run is a partitioned run.has_partition_key: Boolean indicating if the run targets a single partition.has_partition_key_range: Boolean indicating if the run targets a range of partitions.output_for_asset_key: Returns the output name corresponding to a specific asset key.