Dagster’s 2026 data orchestration for reliable ML pipelines

Dagster 1.0 provides a stable foundation for software-defined assets like tables or machine learning models by coupling each asset to its specific production function and all its required upstream dependencies. These assets use an asset key as a global identifier. The definition includes an op to produce the asset and a set of upstream asset keys to define dependencies. This approach allows teams to manage assets as persistent objects rather than isolated tasks.

Use the dg CLI.

This tool launches a local instance with dg dev, generates assets with dg scaffold, validates definitions with dg check, and introspects the system with dg list. For teams using Dagster+, branch deployments spin up ephemeral copies of the platform for every pull request. These deployments include change tracking to highlight modified assets and can clone production databases for validation. Dagster 1.13 also introduced Components and Compass, a Slack-native AI assistant.

Building with assets

Reliability stops bad data.

Pipelines fail. APIs time out. You can enable automatic retries in your dagster.yaml to handle these transient failures. These retries use a FROM_FAILURE strategy to skip successful steps. This prevents manual intervention. Concurrency controls prevent users from overwhelming a warehouse. For example, you can limit a Redshift cluster to 4 concurrent runs. Runs exceeding the limit wait in a queue. If you need to reprocess 100 days of data, Dagster provides three strategies.

Priority tags allow you to run production jobs before backfills. If a run exceeds the max runtime, Dagster marks it as failed. This helps identify processes that consume too many resources. You can override retry settings per job with tags like dagster/max_retries and dagster/retry_strategy. Dagster also supports op-level retries with exponential backoff for step-level transient failures. If you enable both, set retry_on_asset_or_op_failure: false at the instance level so run retries handle only infrastructure failures while op retries handle step-level issues. Data quality relies on asset checks. These tests run after materialization and show up in the UI. You define checks as simple Python functions. You can set severity levels. A WARN severity allows downstream assets to run. An ERROR severity with blocking=True stops downstream assets from materializing. This catches errors at the source. You can also use @multi_asset_check to define multiple checks in one function. If you use dbt, your dbt tests automatically surface as asset checks with no extra code.

Saved selections in Dagster+ let you filter your asset graph, save the view, and share it with your team. You can set alerts to fire when any asset in the selection degrades, changes schema, or violates a freshness policy, and route them to Slack, email, or PagerDuty. Insights helps you find the right limits. You can set tight but realistic guardrails based on actual performance data. You can also view cost metrics for Snowflake and BigQuery. Insights data can be exported via GraphQL. Users can pin selections to their home page to see at a glance whether core KPIs are healthy or degraded. Does the current infrastructure handle this much volume?

The 2026 landscape

The 2026 landscape changed when Prefect announced an agreement to acquire Dagster Labs in July 2026. The transaction closes in the coming weeks, and the combined company operates under the Prefect name beginning in August 2026. Dagster OSS and Dagster+ remain supported under the Dagster name. This acquisition does not change existing contracts or pricing.

The steep learning curve around asset-centric concepts frustrates many organizations. You know the basics of orchestration, so focus on the asset layer. Pick Dagster when asset modeling remains the center of your stack. Airflow provides a massive ecosystem but requires heavy platform engineering to manage its metadata DB and workers. Bruin provides an all-in-one platform for ingestion, transformation, and quality. Mage serves small teams. Prefect provides a lighter Python-first experience for dynamic workflows. Airflow 3.2 in 2026 added asset-aware scheduling, asset partitioning, DAG versioning, multi-team deployments, and a Common AI Provider. Mage provides a mage-agentCLI and MCP support for AI coding tools.