Common mistakes with Temporal workflow engine adoption

I see engineers struggle when they treat Temporal like a standard task scheduler. The steepest learning curve is Non-Deterministic Errors, which occur because the engine must replay history events to rebuild thread stacks. If a developer uses a random number or a wall-clock read inside workflow code, the replayed execution fails to match the original history. This mismatch causes workers to retry in a loop, consuming resources from both the workers and Temporal Cloud. I have seen how these errors cause workflows to stay stuck without making progress. Even if you do not change your code, a new SDK release can cause these errors by changing coroutine ordering for update handlers. Changing a local variable or a method name won’t cause these errors if the event type and ordering remain the same. If a developer implements logging or metric logic using the logger or emitter from the workflow context, it won’t cause an NDE. However, using a random value or iterating a Golang native map to make a decision will trigger it. In these cases, the workflow task will try to replay and cause an NDE when the timer fires. The sticky cache in Temporal workers can hide these bugs until the cache evicts, meaning a bug might stay hidden until a worker restarts or the cache fills.

The Migration Trap

Teams migrating from Airflow or Camunda often waste six weeks rebuilding execution plumbing that Temporal provides natively. While Airflow excels at scheduling batch-style ETL tasks on fixed schedules using DAGs, it lacks the ability to preserve local variable state across tasks because it relies on external databases or XComs. Temporal preserves this state through history replay. Camunda focuses on Business and IT alignment through BPMN notation, whereas Temporal targets a technical audience. I find that engineers often attempt to manually implement idempotency, retries, cancellation, and audit trails. Grab engineers avoided the manual implementation of idempotency, retries, cancellation, and audit trails by replacing their old SQS and Redis architecture with Temporal, which handles infinite retries and rate limiting natively to protect all external services from failing during sudden outages. Grab faced significant issues with a 5-minute Redis lock that resulted in corrupted membership states after their subscriber count surged by over 1000% between January 2022 and June 2023. Because the original architecture lacked idempotency, it also caused the double awarding of benefits when the process retried. Users also reported memberships not renewing or users not receiving benefits after renewal. Airflow users also face issues when the scheduler goes down, as they must handle manual intervention or backfills. Camunda uses a source-available engine called Zeebe, but Temporal uses an MIT license.

The AI/Eval Conflict

The most significant mistake in AI-driven environments involves coupling production durability with evaluation iteration. Most agent frameworks, such as LangGraph or Mastra, link orchestration directly to the runtime. This coupling makes it impossible to run the same logic in a lightweight, ephemeral loop for testing LLM outputs. Brex avoids this by using runtime-agnostic orchestration where the orchestration logic remains a plain function. The team of five engineers at Brex uses TypeScript and Kubernetes to run workers that connect to Temporal Cloud. They use the Vercel AI SDK to reach LLMs. The orchestration is a plain function that only depends on a typed Steps interface. This interface names the agent’s meaningful operations without importing runtime-specific modules like Node.js built-ins. In production, plugins hit real services. In evals, the plugins return fixtures. The orchestration reads like business logic: Enrich, then classify. It has no opinion on whether an activity is a Temporal activity dispatched to a worker or an in-process call that returns fixture data. If you rewrite your orchestration logic for evals instead of using a compatible interface, you invite eval-prod skew, a pitfall you already know exists. I have seen teams fail because they have two different versions of the same logic running in production and evaluation. Does the need for speed in LLM testing justify the risk of having two different versions of your business logic running in production and evaluation?