Avoiding the retry storm in Temporal migrations

Temporal provides durable execution for stateful workflows. Apache Airflow functions as a batch scheduler for data pipelines. Airflow uses Python DAGs to manage tasks. Temporal uses code to define workflows and activities. Activities perform the work and include built-in retry support. Workflows coordinate these Activities but do not retry by default. This distinction prevents teams from using Temporal as a simple cron replacement.

Engineers often struggle with the transition from "smoothie architecture," where business logic and state management mix, to a "layered cake" model. In a smoothie architecture, components are intermixed in one big piece of code. A layered cake separates these concerns into distinct layers.

Temporal manages state through event history. If a process crashes while executing step 3 of a workflow, the SDK loads the history of the incomplete execution from the server to reconstruct the memory state from recorded outputs. The framework gives the code the illusion of a complete re-execution. The control plane manages infrastructure like provisioning and capacity, while the data plane uses cell architecture to ensure resource isolation. This architecture limits the blast radius.

Temporal provides the most reliable way to handle complex, stateful logic.

Errors in retry configuration

A common error involves retrying non-idempotent writes. If an Activity performs a database insert and the network fails before completion, Temporal retries the Activity. This second attempt creates a duplicate record in your database. You must use idempotency keys at the write boundary to prevent this.

Many teams fail to classify errors correctly. Temporal recommends marking permanent failures, such as a 400 error from an API or a validation mismatch, as non-retryable. Retrying a bad payload will never succeed. Transient and intermittent failures resolve with retries, but permanent failures require a change to logic or input. You can use the non_retryable flag in the Python SDK to designate an error as non-retryable.

Misaligned timeouts cause phantom retries. If your Start-to-Close Timeout is too short, Temporal might retry an Activity while the original task still works. A successful database commit at 4.9 seconds with a 5 second timeout leads to a duplicate write when the worker misses the finish line. A Schedule-to-Start Timeout is the maximum time allowed from when an Activity Task is scheduled to when a Worker starts. This timeout is non-retryable by design. A Start-to-Close Timeout is the maximum time allowed for a single Activity Task Execution. Temporal relies on this timeout to force Activity retries when a Worker crashes.

Why do developers ignore these timeouts?

Preventing synchronized write storms

Default retry policies cause massive billing spikes during dependency outages. Every retry generates three billable actions: activity scheduled, activity started, and activity failed. If 500 workflows retry ten times, they generate 15,000 extra actions. This cost scales with the number of concurrent workflows and the retry attempts fitting inside an outage window. A retry policy that generates 10 retry attempts per hour of outage costs 30 extra actions per workflow. For 2,000 concurrent workflows, this results in 60,000 extra actions per hour.

One mistake involves failing to increase the backoff coefficient. The default coefficient is 2.0. Increasing this to 4.0 can reduce retry density by 60 to 70 percent. This change reduces pressure on the recovering dependency.

Deep call chains cause retry amplification. If a workflow retries an Activity, and that Activity retries an HTTP client, and the SQL driver also retries, the attempts multiply. One user action explodes into dozens of write attempts. You should choose one retry owner per boundary.

Large fan-outs create synchronized traffic. If 10,000 Activities attempt to write records at once, they hit the downstream service in lockstep. This happens because all failing Activities share the same retry cadence. Use jitter or staggered batches to prevent this. This prevents the system from hitting the service with a wall of requests.

The platform remembers the retry state through event history. This durability makes bad retry design a heavy burden during production incidents. The retryable decision must be an engineering choice. Labeling an error as retryable based on vibes instead of evidence like transient dependency saturation is a mistake. Economic capping uses a hard ceiling on total retry attempts. Setting MaxAttempts to 5 makes cost predictable. Tiered retry combines quick retries for transient errors with slow retries for sustained outages. Fail-fast is an option for idempotent data sync operations where a fresh workflow run is better than a long-running loop.

Retries are not resilience.

How can teams manage these costs?