- AI
- A
Taming Non-Determinism in Agent Systems
Agent systems break down not on complex tasks or bad models. The main reason is LLM non-determinism: temperature, model updates, world drift. How to debug something that isn't reproducible? How to restart a failed pipeline without starting from scratch? How to understand system behavior at all if every run is slightly different?
Event Sourcing is a pattern where state is not a snapshot, but an immutable event log. It doesn't eliminate non-determinism, but provides tools to work with it.
A bit of theory
First off, this isn't my idea and it's far from new. Martin Fowler systematized Event Sourcing as a pattern in 2005, and Greg Young cemented the ES + CQRS combination in a 2014 talk. The pattern is mostly used in finance and distributed systems — far from LLMs, but it has exactly the same three core log properties. Its closest relative, which made the jump to agents even earlier, is durable execution engines (Temporal, Restate): they do the same "store event history, replay deterministically", just using the workflow/activity vocabulary instead of stream/tool_call.
Direct application of ES to LLM agents is already a 2026 reality: ESAA (arXiv:2602.23193, the very paper that inspired zymi), OpenKedge (arXiv:2604.08601, formalizes governance on top of event-sourced state).
This article is not a literature review, but a practitioner essay built on the above. If you want to dive deeper into any of the theses: ESAA and OpenKedge for formalization, FAIR 2025 for auditability axioms, CoALA and Generative Agents for their connection to memory, Temporal and Restate for "how they solve this outside the agent context".
How ESA is structured
ESA is an event-driven architecture with one added invariant: the log is immutable, only write operations are allowed. That means the event bus here is not just a transport layer between services, but also the single source of truth — everything that has happened in the system is stored in it and never edited. Agents on this bus act as regular subscribers: each one waits for the types of events it needs to appear in the log, picks them up for processing, and writes its results there as well.
This immutable event log is the most valuable architectural idea behind the approach — it provides several benefits at once, provided the task fits into events: pipeline restarts, single source of truth, immutable and auditable journal. These three are actually the same log property viewed from three angles. That's exactly why if any of the three becomes unachievable (restarts no longer reproduce reality, the single source of truth can't keep up with events, logs need to be "deleted" on request) — everything breaks at once, not just one of the three.
For me personally, the pattern became not so much a ready-made specification as a successful formulation of a basic invariant: the state of an agent system should be derived from the event journal. Afterwards, more applied questions came up at zymi: how to branch execution, how to avoid repeating external effects, how to compile context as a projection, and how to make human-in-the-loop part of the same journal.
Restarting a pipeline from any point
ESA allows us to not just restart a pipeline, but launch it from any arbitrary step — say, you have your own research agent that first gathers information, then prepares a report based on it. If you have questions by the time the report is ready, you can simply make edits to this step and restart the pipeline from that point, without re-gathering all the data.
Just like with classic EDA, there are issues here too — if EDA had the problem of external events, for example, you can't process a bank account deduction again on restart — here we also face LLM non-determinism. Fork-resume gives us the ability to at least reduce its impact on the final result: we don't need to re-invoke agents whose results we are satisfied with.
Example from zymi
In zymi, it works roughly like this:
zymi resume --from-step writer creates a new stream and physically copies the event prefix of the researcher there - its LLM calls, network search results, everything. The writer restarts with the actual prompt from agents/writer.yml, the rest is frozen.
External effects within the re-executable part are closed with a separate flag - if the tool has no_resume: true (for example, send_email), it is not called on resume: ToolCallCompleted { replayed: true } is written to the log with an honest placeholder, and the agent sees in the next move "this email was already sent earlier, not done again".
Log as a single source of truth
The log itself is, of course, good, but there is an important nuance - it is revealed only when we make it the single and only source of truth in our system. All consumers, all sources must interact only through the log - this guarantees us both restarts and audibility.
But this strict discipline will reward us with convenience when connecting new sources/consumers - just create a new type of event that they will listen to/create, and that's it. No additional interfaces, everything works out of the box.
Example from zymi
In zymi there is a command that checks the integrity of the log's hash chain:
zymi verify When I was adding it, I expected to work with the runtime: somewhere pass hooks, don't break the pipeline executor, don't affect subscribers. In reality, it turned out to be a file of 76 lines that doesn't import the runtime at all: it opens the same SQLite, iterates over streams, validates SHA-256. That's it. Exactly the same way zymi observe (TUI with a live pipeline graph) and zymi runs (CLI listing of runs) work — each command is an independent consumer on top of the same log, and none of them touch the core.
The benefit of the single source is immediately noticeable: a new consumer is just a new file alongside existing ones, not a new interface in the runtime.
There is another advantage to the single source of truth — the context is no longer a mutable set of data in memory, it is a projection from the log that we can manipulate however we want. Context size, its compression threshold, masking of specific results (this is a separate large topic overall that significantly saves tokens — we plan to cover it in more detail in a separate article), any manipulations — we have complete freedom here, because the original source code is fully preserved in the log. In zymi this is handled by ContextBuilder built on top of the same SQLite — expanding the context window from 10 to 30 steps, masking old tool results with placeholders, rebuilding the context for a different agent — all of these are operations on the projection, and the original sources in the log remain untouched.
At the same time, of course, log projections are very difficult to scale to millions of events per second, and an event log as the sole source of truth is not the best idea for coordinating thousands of parallel agents; ESA is powerless here and will only add to the headaches.
Immutable and auditable log
The third aspect of this same log is immutability. Since we agreed that the log is the single source of truth and all consumers read from it, it is enough to require once that "events are only appended, never edited" — and the log stops being just a storage, it becomes a verifiable invariant.
There are several consequences here. A hash chain naturally lies on top of the immutable log: each record contains SHA-256(event_id + data + prev_hash), and a separate command runs through the stream and checks for any substitutions — the journal becomes not just immutable by discipline, but mathematically verifiable. And the best part is that the audit trail falls out as a side effect of any new event that you put on the bus; you don't need to do anything separate for this.
It is worth noting separately that for each event it is clear where it came from and what caused it. Right in the envelope there are three fields: correlation_id (one user request from start to finish), causation_id (which event generated this one) and source («cli», «telegram», «scheduler», «agent»). For any fact in the log, you can answer «who, when, in response to what» — without a separate audit system, simply because these fields are already in the envelope. For agents, this is a killer of the eternal question «why did the model decide that?» — it turns into «show the chain of events that led to this LlmCallStarted», and it is there in its entirety, from the original user_message to the specific tool_result that fell within the context window.
Example from zymi
Coordination of actions by a «human in the loop» initially lived in a HashMap inside the HTTP handler — an ordinary in-memory state. Restarted the process — forgot who approved what half an hour ago. After transferring the approval to the bus (ApprovalRequested → ApprovalGranted with the field decided_by: "slack:@alice"), several things were fixed at once: approvals became visible from TUI, survived the restart, and — as a side effect — a permanent audit trail appeared «who, what and when approved». Nobody built a separate audit system; it fell out of the fact that approvals now live in the same log as everything else.
The cost of this feature is also quite substantial. Data model evolution: a log accumulated over half a year has collected millions of records in the old format, you add a new field, and the question arises "what to do with old events?". The right to be forgotten enshrined in GDPR: an immutable log cannot "forget" a single entry by definition, and simply erasing it would break the hash chain. There are working solutions for both problems:
The schema issue is resolved by using a combination of default values for new fields and explicit markers for reduced precision.
With GDPR it's trickier: formally, an immutable log and the right to be forgotten are mutually contradictory. There are two solutions: crypto-shredding — each event containing personal data is encrypted with its own unique key, and when a deletion request is received, we simply discard the key; the event remains physically stored, but no one will be able to decrypt it anymore; tombstone events — a special overlay record that signals to all projections to "forget all previous data associated with this subject".
The trade-off here is complexity. Projections need to be designed to work with older versions of the schema. And encrypting payloads requires a full key management infrastructure that also needs to be stored securely and not lost.
Architecture Limitations
Event-sourced architecture (ESA) changes the entire debugging paradigm. The familiar "crash → stack trace → line of code" workflow no longer works — it is replaced by "crash → find the correlation_id → traverse the event chain → restore the state at the time of the failure". This is equally complex work, but it uses a different set of primitives: ordering, causality, and state projection. A team accustomed to working with function calls and stack traces will need time to adjust their mindset; this is not more expensive than debugging a distributed system, but it is not cheaper either — it is simply a different approach.
This architecture is unsuitable for scenarios where the very nature of the task does not support event-based operations: continuous signal streams; data that must be deletable on demand; thousands of parallel agents without a central coordination point; external services with unpredictable behavior.
A good system based on an event-driven approach knows its boundaries and explicitly steps aside where they are reached, rather than trying to force "event-source everything".
Conclusion
To briefly return to where we started: an immutable log provides three properties—deterministic restart, single source of truth, auditability. The key point here is not three separate bonuses you can pick individually. It's the same property of the log, presented to us from three angles. Therefore, a compromise on any one of them collapses the other two: if restart doesn't reproduce reality—then the log is not the source of truth; if a record in the log can be adjusted—then audits cannot be trusted; if consumers bypass the log—then fork-resume won't reproduce the picture they saw.
All practical cases in this article were obtained while developing an open-source framework for agents—zymi-core. I wrote about it in my previous article—What if we assemble agents like a dbt project? The project is now moving towards an MCP server—a backend for agents that can be connected to your favorite Claude Code / Codex / OpenClaw and others.
I also write about agents, Event Sourcing, and related topics on Telegram.
Write comment