Durable Execution Contracts: What AI Agent Orchestration Engines Actually Guarantee

Durable execution is not a single promise but a bundle of distinct, engine-specific contracts — replay determinism, step memoization, exactly-once side effects, checkpointing cost, and named hard ceilings — and a verbatim read of the official documentation for eight AI-agent-relevant engines shows precisely where each contract holds and stops.

Sources and scope: This audit is desk research only: all claims are read directly from the linked official documentation, with no hands-on testing.

The Guarantee Surface Is Not One Contract

Eight engines marketed to AI agent teams publish overlapping but non-identical durability guarantees. The docs use the same words — exactly-once, durable, replayable — for different mechanics. Auditing them clause by clause separates five distinct contracts that buyers routinely collapse into a single promise: durability.

The five contracts are separable and independently falsifiable. Replay determinism says the engine can reconstruct control flow. Step memoization says completed work is skipped rather than repeated. Exactly-once side effects says an external call is observed once. Checkpointing cost says what persistence charges per unit of progress. Named ceilings say where the engine refuses to proceed. Temporal documents all five; LangGraph documents two. The frame is read off Temporal’s limits page and Inngest’s execution model.

Replay Contracts: Determinism Is the Load-Bearing Wall

Deterministic replay is the strictest contract in this set: the engine re-runs orchestration code and requires the same command sequence every time. Temporal and DBOS impose it, Restate imposes it only around journal calls, and each doc names the failure mode in its own words.

Temporal’s load-bearing sentence: “By design, completed Activities will not re-execute as part of a Workflow Replay.” Replay re-derives decisions from history; it does not re-run finished work. DBOS states the reciprocal obligation on the caller — “The workflow function must be deterministic” — with randomness and local time pushed into a step, because “once a step completes and is checkpointed, it is never re-executed.” Restate scopes purity narrowly: side effects go through run(), since “Without run(), these operations would produce different results during replay.” Sources: Temporal activity definition, DBOS architecture, Restate key concepts.

Exactly-Once Observation vs. Exactly-Once Execution

Vendors use “exactly-once” for two different claims. One concerns observation: how many times the engine records an outcome. The other concerns execution: how many times your code physically runs against an external system. Every doc here commits to the first; none commits to the second unconditionally.

Temporal draws the line explicitly: “Temporal guarantees that the Activity will be observed as completed exactly once. However, the Activity may be executed multiple times and may even partially complete more than once during this process.” Observation is guaranteed; execution is not. Restate’s claim reads stronger — “Side effects are executed exactly once” — but is scoped to calls inside ctx.run(), where recovery replays the journaled result instead of re-issuing it. Pub/Sub shows the same shape: exactly-once delivery “only applies when subscribers connect to the service in the same region,” and only pull subscriptions support it. Sources: Temporal activity definition, Restate durable building blocks, Pub/Sub exactly-once delivery.

Durability Budgets: Hard Ceilings the Docs Name Explicitly

Every engine that publishes ceilings publishes them as refusal conditions, not warnings. The numbers below are the ones the official docs state outright: history size, concurrency, retention, retry counts. A ceiling written only in prose is still a ceiling, and the table marks each engine’s named caveat beside it.

Temporal is the most granular. Event History is limited to 51,200 Events or 50 MB, with a warning after 10,240 Events or 10 MB; a single request payload is 2 MB and any Event History transaction is 4 MB, non-configurable on Temporal Cloud. Retention is settable between 1 and 90 days, defaulting to 30 days on Temporal Cloud, and 2,000 incomplete Activities plus a 10,000 Signals cap bound the command surface. See Temporal cloud limits.

Two rows need separate reading. Cloudflare counts only running instances against its cap, so idle instances sit outside the 10,000 concurrent instances budget, and cron-triggered instances get up to one hour per firing without consuming it. DBOS prices its step in writes and reports a throughput of >40K workflows or steps per second on a single Postgres database. Sources: Cloudflare limits, DBOS architecture, AWS error handling.

Checkpointing Cost Models: What Persistence Adds

Checkpointing is not free and the docs price it differently. One engine bills in database writes per step, one in HTTP requests per step, one in quorum-replicated journal appends, one in unbounded stored threads. The cost model decides which workload shape stays affordable.

DBOS is the most explicit: “The only overhead DBOS adds is database writes: one database write per step (to checkpoint the step’s outcome) plus two additional writes, one at the start of the workflow and one at the end.” Inngest prices the step in network terms — “Each step in your function is executed as a separate HTTP request” — moving latency to the code/engine boundary. Restate prices by consensus: the append “replicated to quorum defines ‘the step happened.’” LangGraph’s cost is storage growth, flagged by its own docs as “Checkpoints growing unboundedly.” Sources: DBOS architecture, Inngest steps-workflows, Restate architecture, LangGraph persistence.

Idempotency Windows and Key Semantics: What Actually Deduplicates

Deduplication depends entirely on key semantics: how a key is formed, how long it lives, and what happens on a collision or a parameter mismatch. Three engines here define key behavior precisely, and one non-orchestration API sets the clearest reference contract of all.

Temporal hands you raw material rather than a mechanism: “You can use a combination of the Workflow Run ID and the Activity ID as an idempotency key since this is guaranteed to be consistent across retry attempts but unique per Activity Execution.” The engine guarantees key stability; deduplication stays yours. Restate is closer to a built-in layer — keys are generated in-handler, the example uses restate.UUID(ctx), and duplicate requests return the same result.

Stripe, cited as a boundary reference rather than an orchestration engine, documents what the orchestration pages leave implicit. Idempotency keys are up to 255 characters long. Stripe saves the status code and body of the first request for a given key “regardless of whether it succeeds or fails,” so later requests return the same result, “including 500 errors.” Keys can be removed “after they’re at least 24 hours old,” and reuse after pruning generates a new request; mismatched parameters error rather than silently re-run. Sources: Temporal activity definition, Restate key concepts, Stripe idempotency.

Versioning and Upgrades: The Contract’s Escape Hatch

Code changes are where a durability contract meets reality, because replay assumes the code running today matches the code that ran yesterday. Every engine here documents an escape hatch — patching, versioning, or pinned revisions — and each documents what breaks if you skip it.

Temporal documents the longest procedure: patch in new code with patched(); run new alongside old; then remove the old code and call deprecate_patch(). Worker Versioning pins Workflows to specific revisions so old Workers run old code paths — the guarantee is that the executing code matches the history, not that one binary serves all history. DBOS offers the same two strategies, “patching and versioning,” and states the consequence plainly: a checkpoint mismatch against a previous version leaves the workflow “difficult to recover.” That sentence is the contract’s real edge. Sources: Temporal worker versioning, Temporal Python versioning, DBOS architecture.

The Two-Engine Divide: Deterministic Replay vs. Skip-Completed Memoization

The sharpest documentation in this set is Inngest’s, because it names its competitor and contrasts the two programming models directly. The distinction is not cosmetic: it determines where non-determinism may live, what runs once, and how much of your code the engine must understand to resume.

Quoting at length because the wording is the point: “Temporal uses a deterministic replay model where your entire workflow function is re-executed from the beginning on each step, relying on an internal event history to skip completed work. This requires developers to learn and follow strict determinism rules. Inngest uses a step-based memoization model where each step runs once, its result is persisted…”

The trade is legible. Deterministic replay buys the engine freedom to re-run your entire function, at the cost of strict rules and a history that must be read on every resume. Memoization lets ordinary code stand — DB calls, API calls — provided “Any non-deterministic logic (such as DB calls or API calls) must be placed within a step,” at the cost of a state store persisting every step result. Restate, DBOS and Hatchet sit nearer the memoization side; LangGraph checkpoints state rather than steps. Source: Inngest execution model.

Where the Promises Stop: Named Gaps in the Official Docs

Named gaps are the most useful part of a documentation audit, because a vendor that states its failure mode has told you where the contract ends. These are caveats the official pages write down themselves, with no inference added here.

LangGraph is the most candid: MemorySaver and InMemorySaver “store checkpoints in RAM. When the process restarts, all checkpoints are lost,” which is why PostgresSaver is the documented production checkpointer, and thread_id values must stay under 255 characters or “expect a database error.”

Restate’s gap is positional rather than numeric: skip run() and the operation is simply not durable. Cloudflare’s is resource-shaped — individual steps are subject to the configured CPU time limit even though wall time is unlimited, and retries stop at 10,000 retries per step.

Pub/Sub supplies the cleanest counterexample in the audit. Even with exactly-once enabled, “A subscription might receive multiple copies of the same message due to publish side duplicates, even with exactly-once delivery”; push and export subscriptions do not support the guarantee at all, and latency is “significantly higher.” Sources: LangGraph durable execution, Restate key concepts, Cloudflare limits, Pub/Sub exactly-once delivery.

Implications for AI Agent Orchestration: Choosing the Contract You Can Tolerate

Choosing an engine is choosing which failure you can tolerate. The documentation tells you which one each engine plans for: wrong control flow, repeated side effects, lost state, or refused progress. Match the named gap to your blast radius, not to the headline guarantee.

Side-effect exposure. If a tool call cannot be made idempotent, an observation-based contract — Temporal’s “observed as completed exactly once” — still leaves deduplication to you, using the Run ID and Activity ID pair the docs guarantee are stable. Journal-based routing moves that burden into the engine, but only for calls you route through it.

State volume. Long-running agents accumulate history. Temporal’s history caps and LangGraph’s unbounded checkpoint growth are one problem in two vocabularies, and both push toward rotation rather than indefinite single-thread state.

Upgrade cadence. If agent code ships weekly, patching and versioning are not optional features — they are the difference between a resumable execution and a workflow that is “difficult to recover.”

Sources: Temporal continue as new, LangGraph persistence, DBOS architecture.

FAQ

Three questions recur whenever these pages are read side by side. The answers below stay inside what the official documentation states, and each traces back to a clause quoted earlier in this audit rather than to any experience operating the engines.

Does Temporal’s ‘exactly once’ guarantee mean an Activity body runs once?

No — Temporal guarantees the Activity is “observed as completed exactly once” while allowing it to be executed multiple times and to partially complete more than once.

Which engines skip completed steps instead of re-executing them?

Inngest, Restate, DBOS and Hatchet, each persisting step results, journal entries or checkpoint log entries before resuming, as stated in Inngest’s execution model.

What happens when a named hard limit is reached?

The engine refuses the operation — failed commands, a database error, dropped concurrency, or a workflow left difficult to recover — exactly as each cited limits page states.

📖 Related Reads

• NiteAgent — AI agent development, frameworks, and production patterns
• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.