Managing concurrency

This guide covers managing concurrency of Dagster assets, jobs, and Dagster instances to help prevent performance problems and downtime.

When your pipelines interact with rate-limited APIs, shared databases, or resource-constrained systems, it is a good idea to limit how many operations execute simultaneously. Dagster provides several mechanisms for managing concurrency at different levels of granularity.

Mechanism	Scope	Protects against	Use when
Run queue limits	Deployment	Too many simultaneous runs	You need to limit total runs across your deployment Backfills or sensors might launch many runs at once Your infrastructure has fixed capacity for concurrent runs
Concurrency pools	Cross-run	Overloading shared resources	You need to protect a shared resource (database, API) across all runs Multiple runs might simultaneously access the same external system You want ops to queue until the resource is available
Run executor limits	Single run	Memory/CPU exhaustion	You want to control parallelism within a single run Memory or CPU constraints limit how many ops can run at once You don't need cross-run coordination
Run tag limits	Single run	Resource contention by category	Different ops in the same run have different resource requirements You want some ops to run in parallel while limiting others You need category-based throttling within a run
Branch deployment concurrency limits	All branch deployments	Branch deployments using too many resources	You are managing multi-developer teams (Dagster+ only)

Combining mechanisms to manage concurrency

You can combine the mechanisms listed in the table above to manage concurrency. For example:

• Pools + run executor limits: Protect external resources while also limiting local parallelism
• Run queue + pools: Limit total runs AND protect specific resources within those runs
• Tag limits + pools: Fine-grained control within runs plus cross-run protection

concurrency:
  runs:
    max_concurrent_runs: 10
  pools:
    default_limit: 3