Skip to main content

Advanced concurrency configuration

Use job metadata and schedules to prevent runs from starting if another run is already occurring (advanced)​

You can use Dagster's rich metadata to use a schedule or a sensor to only start a run when there are no currently running jobs.

src/<project_name>/defs/assets.py
import time

import dagster as dg


@dg.asset
def first_asset(context: dg.AssetExecutionContext):
    # sleep so that the asset takes some time to execute
    time.sleep(75)
    context.log.info("First asset executing")


my_job = dg.define_asset_job("my_job", [first_asset])


@dg.schedule(
    job=my_job,
    # Runs every minute to show the effect of the concurrency limit
    cron_schedule="* * * * *",
)
def my_schedule(context):
    # Find runs of the same job that are currently running
    run_records = context.instance.get_run_records(
        dg.RunsFilter(
            job_name="my_job",
            statuses=[
                dg.DagsterRunStatus.QUEUED,
                dg.DagsterRunStatus.NOT_STARTED,
                dg.DagsterRunStatus.STARTING,
                dg.DagsterRunStatus.STARTED,
            ],
        )
    )
    # skip a schedule run if another run of the same job is already running
    if len(run_records) > 0:
        return dg.SkipReason(
            "Skipping this run because another run of the same job is already running"
        )
    return dg.RunRequest()

Limit concurrent runs across all branch deployments (Dagster+ only)​

In Dagster+, you can limit the total number of concurrent runs across all branch deployments using the organization-scoped max_concurrent_branch_deployment_runs setting. By default, this value is 50.

This setting is useful for preventing branch deployments from consuming too many resources, especially when multiple developers are working simultaneously.

To view or modify this setting, use the dagster-cloud CLI (requires a user token with the Organization Admin role):

# View current settings
dagster-cloud organization settings get

# Save settings to a file, edit, then sync
dagster-cloud organization settings get > org-settings.yaml
# Edit org-settings.yaml to change max_concurrent_branch_deployment_runs
dagster-cloud organization settings set-from-file org-settings.yaml

For more details, see Managing branch deployments across multiple deployments.