Change tracking in branch deployments

Dagster+ feature

This feature is only available in Dagster+.

Change tracking in branch deployments makes it easier for you and your team to identify how changes in a pull request will impact data assets.

How it works

Branch deployments compare asset definitions in the branch deployment against the asset definitions in the branch's base deployment. The UI will then mark changed assets accordingly. For example, if the pull request associated with the branch deployment adds a new asset, the UI will display a New in branch label indicating the addition.

You can also apply filters to show only new and changed assets in the UI. This makes it easy to understand which assets will be impacted by the changes in the pull request associated with the branch deployment.

note

The default behavior is to compare the branch deployment against the current state of the base deployment. Depending on deployment and development cadence, the code deployed in the base deployment can get ahead of the branch causing changes to appear that are from the base getting ahead of the branch.

To address this, the dagster-cloud CLI command branch-deployment create-or-update has an option --snapshot-base-condition that can be set to either on-create to snapshot the base only when the branch is first created or on-update to refresh the base snapshot anytime the branch is updated.

Supported change types

Change tracking can detect the following changes to assets:

• New assets
• Code versions
• Upstream dependencies
• Partitions definitions
• Tags
• Metadata

New assets

If an asset is new in the branch deployment, the asset will have a New in branch label in the asset graph:

Code versions

If using the code_version argument on the asset decorator, change tracking can detect when this value changes.

Some Dagster integrations, like dagster-dbt, automatically compute code versions for you. For more information on Dagster code versioning, see Asset versioning and caching.

Asset in the Dagster UI

In this example, the customers asset has a Changed in branch label indicating its code_version has been changed.

Asset definition

In the main branch, we have a customers asset with a code version of v1:

@asset(code_version="v1")
def customers(): ...

In the pull request, customers is modified to change the code version to v2:

@asset(code_version="v2")
def customers(): ...

Upstream dependencies

Change tracking can detect when an asset's upstream dependencies have changed, whether they've been added or removed.

note

If an asset is marked as having changed dependencies, it means that the AssetKeys defining its upstream dependencies have changed. It doesn't mean that an upstream dependency has new data.

Asset in the Dagster UI

In this example, the returns asset has a Changed in branch label indicating it has changed dependencies.

Asset definition

In the main branch, we have a returns asset with a single upstream dependency, orders:

@asset(deps=[orders])
def returns(): ...

In the pull request, we change the upstream dependencies of the returns asset to orders and customers:

@asset(deps=[orders, customers])
def returns(): ...

Partitions definitions

Change tracking can detect if an asset's PartitionsDefinition has been changed, whether it's been added, removed, or updated.

Asset in the Dagster UI

In this example, the weekly_orders asset has a Changed in branch label indicating a changed partitions definition.

Asset definition

In the main branch, we have a weekly_orders asset:

@asset(partitions_def=WeeklyPartitionsDefinition(start_date="2024-01-01"))
def weekly_orders(): ...

In the pull request, we updated the WeeklyPartitionsDefinition to start one year earlier:

@asset(partitions_def=WeeklyPartitionsDefinition(start_date="2023-01-01"))
def weekly_orders(): ...

Tags

Change tracking can detect when an asset's tags have changed, whether they've been added, modified, or removed.

Asset in the Dagster UI

In this example, the fruits_in_stock asset has a Changed in branch label indicating it has changed tags.

Asset definition

In the main branch, we have a fruits_in_stock asset:

@asset(tags={"section": "produce"})
def fruits_in_stock(): ...

In the pull request, we added the type: perishable tag to fruits_in_stock:

@asset(tags={"section": "produce", "type": "perishable"})
def fruits_in_stock(): ...

Metadata

Change tracking can detect when an asset's definition metadata has changed, whether it's been added, modified, or removed.

Asset in the Dagster UI

In this example, the products asset has a Changed in branch label indicating it has changed metadata.

Asset definition

In the main branch, we have a products asset:

@asset(metadata={"expected_columns": ["sku", "price", "supplier"]})
def products(): ...

In the pull request, we update the value of the expected_columns metadata on products:

@asset(metadata={"expected_columns": ["sku", "price", "supplier", "backstock"]})
def products(): ...