Yes, LLMs can be deterministic. But most hosted stacks are not.

Ask an engineer why the same prompt at temperature zero returns two divergent answers on different days. The usual reply is: "LLMs are non-deterministic." That answer, while convenient, is wrong at the model level. It keeps agents out of critical and regulated workflows where maintainers must prove behavior and auditors must replay past runs.

A recent example: on July 16, an OpenAI agent compromised accounts on Hugging Face. Forensics teams could not precisely replicate the agent's steps. Engineers faced an incomplete evidence trail and no path to reproduce the execution. Diagnosis and resolution took far longer than necessary [1].

The Agentic Determinism Index (ADI) addresses the agentic execution path replicability issue. It measures which stacks actually deliver replayable inference, so teams needing proof can choose the right one. The index is public at lemma-ventures.github.io/agentic-determinism-index; anyone can run the harness and contribute via the GitHub repository.

Why critical and regulated systems need determinism

Agents are being deployed in jobs that require proof. Critical and regulated systems require traceability: maintainers must re-run the exact execution to verify behavior, and auditors need the exact bytes to confirm the execution followed the mandated workflow. When outputs drift, these systems lose their required evidence trail.

The EU AI Act's record-keeping obligations for high-risk systems apply from 2 August 2026 [2]. MiFID II rules for algorithmic trading assume that test behavior equals production behavior [3]. On most current serving paths, you cannot get the same bytes twice. That is a measured fact, not a vibe.

The technical explanation follows. If you want the practical answer first, skip ahead to the Agentic Determinism Index section.

Why identical requests can differ

You send the same request twice with identical settings and model name. Yet, the answers differ.

Several underlying aspects of the serving stack can cause this:

• The hardware. One request might run on a specific type of computer chip, while the next request might run on a different one. Small variations in chip design could influence the result.

• Software drivers and updates. The programs running the model receive small fixes or upgrades without notice. These can alter how the computations happen.

• The cloud location. Requests can go to different data centers or geographic regions. Each region may use slightly different machines or settings.

• Quiet model updates. The provider can replace the model with a newer version. The name on the label stays the same, but the inner rules have changed.

When any of these things shift between two requests, the output can vary.

If you need the exact same answer every time, you must control the full stack running the model. You need to select and lock the hardware, software version, and location.

The model is deterministic under a pinned stack

A transformer forward pass is arithmetic. Weights times input gives logits. There is no randomness and no hidden state. Fix the weights, fix the input bytes, and fix the execution order, and the logits come out identical every time, bit for bit. Thinking Machines verified this directly: every kernel in the forward pass is run-to-run deterministic at a fixed input shape [4].

One fundamental point that holds the entire argument: determinism is a property of a complete specification, not of the weights alone. The abstract model over real numbers does not fix the output bytes; floating-point arithmetic is not associative, so the execution rules that choose the order of operations fix them. When those rules are left unspecified, the question "is the model deterministic?" cannot be answered. The stack is where those rules live, and that is why the stack owns the outcome.

No model is non-deterministic independent of its stack. With temperature zero and a fixed seed, the arithmetic itself is deterministic. Non-determinism appears only when the serving stack leaves rules unpinned (batch order, kernel choice, RNG (random number generator) state, and versions). Sampling at temperature >0 is stochastic by design, but the post concerns pinned deterministic execution at temperature zero.

Sampling does not change this. A sampler with a fixed seed and a fixed algorithm is a deterministic function too. Pseudo-random is deterministic by construction.

The model path is stable:

request bytes -> model arithmetic -> logits -> seeded sampling -> tokens

prompt bytes
      |
      v
  serving stack receives the request
      |
      v
  scheduler groups it with other requests
      |
      v
  kernels pick a reduction order
      |
      v
  output bytes (different order, different bytes)

Five mechanisms inside the serving stack that produce divergence

The everyday causes above boil down to five mechanisms inside the serving stack that can be named, measured, and pinned:

1. Your neighbors in the batch. Servers group concurrent requests. Floating-point addition is not associative, so (a + b) + c is not equal to a + (b + c) in general, and batch size changes the order of additions inside the kernels. Your output depends on how many other requests ran with yours. This is the dominant source. Thinking Machines demonstrated it cleanly in 2025: individual kernels are deterministic at fixed shapes; the batch-dependent shapes are what betray you [4].
2. Kernel selection. cuBLAS switches matmul strategies with batch shape [4], cudnn.benchmark races convolution algorithms and keeps the fastest [5], and Triton autotune benchmarks candidate configurations and keeps the winner [6]. A different day gives a different winner and a different reduction order.
3. The sampler. Server RNG state is rarely pinned per request, and seeded results are only guaranteed to reproduce on the same platform and library build [5].
4. Silent drift. Engine upgrades, container rebuilds, and driver updates all change the arithmetic; PyTorch states plainly that reproducibility is not guaranteed across releases, commits, or platforms [5]. None of it shows up in the model name you called. OpenAI tells you to watch a fingerprint for exactly this reason [7]. Drift can also delete the path outright: on 26 August 2026, NVIDIA NIM returned HTTP 410 for meta/llama-3.1-8b-instruct with the message that the model had reached end-of-life [8]. Any team that had pinned that string overnight lost the ability to re-run yesterday's outputs against the same serving tuple.
5. Hardware. Different GPU generations run different execution paths. Under greedy decoding, changing GPU type, count, or batch size alone moved a reasoning model's accuracy by up to 9 percent [9]. The same model on two GPU classes gives no guarantee of the same bytes.

Notice what is not on the list: the model.

A concrete example. At 10:02:11 you send prompt P and it lands in a batch of 18 requests. At 10:02:17, you send the same bytes and they land in a batch of 2. Both runs are correct and the model did not change, but the outputs differ. The batch did.

Closed APIs made the choice for you

If determinism is a stack property, then whoever controls the stack controls whether you can reproduce an output. On hosted APIs, that is not you.

The record is public. OpenAI shipped a seed parameter in 2023 and labeled it best-effort: "determinism is not guaranteed" [7]. Its newer Responses API has no seed parameter at all [10]. Anthropic has no seed parameter and its documentation says plainly that "even with temperature of 0.0, the results will not be fully deterministic" [11]. Version fingerprints drift as providers upgrade engines underneath you [7]. You find out when your evals move.

This is not malice. Deterministic serving costs throughput, roughly 25 to 45 percent in SGLang's published measurements [12], and hosted economics reward throughput. But the consequence deserves a plain statement:

On hosted APIs, bit-exact replay of a past output is not on offer. The stack that produced it was never observable and no longer exists.

The only true determinism available there is caching the response.

That matters because small drift moves real things: a leaderboard position, an A/B result, a regression suite, an audit trail of what an agent decided.

What replay takes

Replay is an engineering condition, not a promise. Open stacks can meet it today. SGLang ships --enable-deterministic-inference [12] and vLLM ships VLLM_BATCH_INVARIANT=1 [13]. Both give bitwise run-to-run reproducibility on self-hosted open weights, at a real throughput cost. In Thinking Machines' demonstration on vLLM, the unoptimized deterministic path ran at roughly half the default throughput [4].

For same prompt, same bytes, six layers must stay fixed:

1. request   exact prompt bytes, parameters, tool schemas
  2. weights   exact checkpoint hash, not a version label
  3. engine    deterministic mode, fixed build
  4. sampler   fixed seed, pinned per request
  5. runtime   pinned CUDA, driver, container
  6. hardware  one GPU class
      |
      v
  same output bytes, this year or in five years

Break one layer and the guarantee is gone.

It is important to note that bit-identity across different GPU classes is still open research [9]. And none of this buys semantic stability: a deterministically served model still answers a reworded question differently; trivial prompt reformatting alone has moved benchmark accuracy by up to 76 points [14]. Determinism gives you reproducibility (bit-exact replay of the same output), not predictability or accuracy. Those are different properties, and only the first one is on offer here.

The Agentic Determinism Index

The Agentic Determinism Index (ADI) is an open harness that asks every major hosted provider one narrow question: if I send you the same request N times, concurrently and across days, how identical are your answers?

The methodology is deliberately boring:

1. Identical requests fire concurrently, then serially, at temperature zero and fixed seed where supported.
2. Divergence is measured at the byte level and the token level, per model, over time.
3. Fingerprint and version drift is tracked across a rolling window.
4. Structured-output stability is scored separately.
5. Every score is recomputable from the published harness and raw transcripts.

The methodology is open for comment. Bootstrap reference scores ship with the harness so you can see numbers on day one. We will keep running the index and folding in open-source community contributions; a first analysis will follow after about a month of runs. A measurement you cannot audit is just content marketing: every score is recomputable from raw transcripts. Alongside the scores, we publish the exact conditions a provider must meet for reproducible serving. Providers that meet them are recognized for it.

The goal is a living, public list of stacks that deliver byte-exact replay. Teams that run agents under audit requirements can use the list to select a serving path that will still be reproducible at any future time. The harness also detects when a provider changes its stack, so untested configurations do not reach critical or regulated systems. Anyone who runs the harness on their own tuple can add results. The effort is collective.

Lemma Ventures builds verifiable finance. Verification needs exact reproducibility right out of the gate for compliance purposes. The ADI is the measurement layer for that requirement and sets the ground for the agentic compliance tools we are building.

If you think the methodology is wrong, open an issue. If you run your own stack, run the harness on it and send the transcripts. If you need replayable agent decisions for a regulated system, the list is for you.

References

1. Ryan Greenblatt, "I was the main person doing transcript analysis for this investigation of the Hugging Face incident", X post, August 2026. x.com/ryangreenblatt/status/2092692685224325542. The investigation relied on over a thousand long agent transcripts; precise reconstruction was difficult due to the lack of bit-exact replay.
2. Regulation (EU) 2024/1689 (AI Act), Article 12 record-keeping; applicable from 2 August 2026 per Article 113. eur-lex.europa.eu/eli/reg/2024/1689/oj
3. Commission Delegated Regulation (EU) 2017/589 (MiFID II RTS 6), testing and annual self-assessment obligations for algorithmic trading systems. eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32017R0589
4. Horace He and Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference", September 2025. thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference
5. PyTorch documentation, "Reproducibility". docs.pytorch.org/docs/stable/notes/randomness.html
6. Triton documentation, "triton.autotune". triton-lang.org/main/python-api/generated/triton.autotune.html
7. OpenAI Cookbook, "Reproducible outputs with the seed parameter", 2023, https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter
8. NVIDIA NIM API, HTTP 410 Gone for meta/llama-3.1-8b-instruct, 26 August 2026: model marked end-of-life on integrate.api.nvidia.com. Observed during an ADI harness probe the same day.
9. Yuan et al., "Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference", 2025. arxiv.org/abs/2506.09501
10. OpenAI API reference, "Create a model response": the Responses API request parameters include temperature and top_p but no seed. platform.openai.com/docs/api-reference/responses/create
11. Anthropic API reference, "Messages", temperature parameter note. platform.claude.com/docs/en/api/messages
12. SGLang Team, "Towards Deterministic Inference in SGLang and Reproducible RL Training", 22 September 2025. lmsys.org/blog/2025-09-22-sglang-deterministic
13. vLLM documentation, "Batch Invariance". docs.vllm.ai/en/latest/features/batch_invariance
14. Sclar et al., "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design", 2023. arxiv.org/abs/2310.11324

Lemma Ventures engineering. Maintainer incentive disclosed: we build in this space. All scores are recomputable from the open harness and raw transcripts.