Dagster offers several ways to run data pipelines without manual intervention, including traditional scheduling and event-based triggers. Automating your Dagster pipelines can boost efficiency and ensure that data is produced consistently and reliably.
When one of Dagster's automation methods is triggered, a tick is created. A tick is an opportunity for one or more runs to be launched. A run will either materialize a selection of assets or execute a job. Some schedules and sensors will launch runs on every tick. Others have associated logic that is executed on each tick which determines the runs to be launched.
In this guide, we'll cover the available automation methods Dagster provides and when to use each one.
Before continuing, you should be familiar with:
In this section, we'll touch on each of the automation methods currently supported by Dagster. After that we'll discuss what to think about when selecting a method.
Schedules are Dagster's imperative option for automation. They allow you to specify exactly when a run should be launched, such as Mondays at 9:00 AM. Schedules can target a selection of assets or a job. Refer to the Schedules documentation to learn more.
Sensors launch runs in response to a detected event. They periodically check and execute logic to detect an event and conditionally launch runs. They are commonly used for situations where you want to materialize an asset on some externally observable trigger, such as:
You can also use sensors to act on the status of a job run. Refer to the Sensors documentation to learn more.
Declarative Automation allows you to automatically materialize assets when specified criteria are met. Using Declarative Automation, you could update assets:
Materialization conditions are declared on an asset-by-asset basis. Refer to the Declarative Automation documentation to learn more.
Asset sensors launch runs when a specified asset is materialized. Using asset sensors, you can instigate runs across jobs and code locations and keep downstream assets up-to-date with ease.
Refer to the Asset Sensor documentation to learn more.
Before you dive into automating your pipelines, you should think about:
The following cheatsheet contains high-level details about each of the automation methods we covered, along with when to use each one.
| Method | How it works | May be a good fit if... | Works with | |
|---|---|---|---|---|
| Schedules | Starts a job at a specified time |
|
| |
| Sensors | Starts a job or materializes a selection of assets when a specific event occurs | You want to trigger runs based off an event |
| |
| Declarative Automation | Automatically materializes an asset when specified criteria (ex: upstream changes) are met |
| Assets only | |
| Asset Sensors | Starts a job when a materialization occurs for a specific asset or selection of assets |
| Assets only |