← all writing

[ ARCHITECTURE · 2026-07-29 · 7 MIN ]

The Modern Data Stack for Agents

by Burak Emre Kabakcı

FIG_01 [ cover · architecture ] © lobu v1

Every team building agent memory ends up rebuilding the modern data stack. Most don’t notice until they are most of the way through.

It starts with “the agent should remember things.” You add a sync from Slack. Then one from your CRM. Then you need the same person from both sources to be one person. Then someone asks why the agent said 40 when the dashboard says 38.

None of that is an AI problem. It is ingestion, modelling, and reconciliation, and data engineering spent fifteen years working it out.

So take the analogy seriously, because most of it transfers. Connectors are change data capture. The append-only log is your raw landing zone. Entities are the modelled tables. Behaviors are dbt models on a schedule. The app is the BI layer on top. If you have built a warehouse, you already know most of the shape an agent memory layer should have, and you should reuse that instinct instead of inventing new vocabulary.

The classic pipeline in six numbered stages: OLTP sources, CDC and ETL, the data warehouse of fact tables, dimensions and aggregates, analysts, executives, and end users. Along the bottom, the manual actions those people take re-enter the source systems as new operational events, never as a record of the decision itself

It breaks in four places. Those four are the rest of this post.

Writing the decision back

Follow the pipeline above to the end. An analyst reads a dashboard, writes a brief, an executive decides something.

Then the decision leaves. Some of what people do next comes back as raw activity, a new row or a new ticket, but the decision itself never does. It lands in Slack, or a board deck, or nowhere. The warehouse never learns it happened, so next quarter’s model has no way to know the March number was thrown out for a reason.

Warehouses do push data back out. Reverse ETL is a mature category, you could model decisions as a table and sync them, and some teams do. What makes it rare is that nothing upstream captures the decision in the first place. It happens in a meeting or a Slack thread and reaches the warehouse only if somebody remembers to write it down.

Capture is the hard part. A store whose readers are agents has to catch the decision where it is made, not backfill it afterwards.

So we close the loop. When an agent wants to do something consequential, one transaction writes two things: a pending run, and an event saying a human needs to approve it.

The human’s answer does not update that event. It appends a new one pointing at it, stamped with who decided. Later the outcome appends again, carrying whatever came back. The approval, the rejection, the arguments a reviewer edited before letting it through: all of it lands in the same store the agent reads from tomorrow.

I wrote about the store underneath that loop in The Context Layer Is an Event Store.

Everything downstream is only a cache

The instinct from warehouse-land is medallion: bronze for raw, silver for cleaned, gold for business-level aggregates. Three tiers, each a physical table you maintain, each authoritative for its own layer.

We materialize too, so this is not projection versus materialization. Anyone who has built a read model knows those are not alternatives. The difference is what the materialized table is allowed to claim.

The clearest way to explain it is with one we deleted.

For a day in June we had a trigger-maintained table holding the latest value of every entity field. Its own comment demoted it, calling itself a rebuildable cache and naming the events log as the source of truth. Then we dropped it, because no read path consumed it and the producer feeding it was pure overhead on the hot write path. That took one migration. The events it had emitted stayed in the log, because we never rewrite the log, and they are still there if we want the projection back.

Try that with a silver table. It is authoritative for its layer, so deleting it loses data and rebuilding it is a project. Ours was a cache, and a cache is allowed to be wrong. The shape is closer to what Jay Kreps called kappa architecture than to a medallion stack: one immutable log, everything downstream derived from it, nothing downstream load-bearing.

This sounds academic until you have to correct something. When silver is authoritative, fixing a bad row means a backfill and a reconciliation, and “what did gold say last Tuesday” often has no answer at all. Here a correction is an append pointing at what it replaced, anything derived recomputes from the log, and the old value is still on disk. You pay storage for the ability to answer questions about your own past. That is a good trade when the thing you are explaining is an agent’s mistake.

Every read has to carry who it is for

This is the difference that surprised me most, and the one I would push hardest on if you are building this.

Warehouses can do it, and I do not want to pretend otherwise. Snowflake row access policies and dynamic masking run in the query path, enforced by the engine rather than by whichever dashboard someone happened to open. The mechanism is there and it is good.

What changes is the consumer. Those policies key off a session identity, the role running the query, which works when every human logs in as themselves.

An agent breaks that. One process serves everybody, and the identity that matters is whoever the agent is answering for, not the credential it connected with. Get it wrong once and a coworker’s Gmail turns up in an answer, with no dashboard boundary to catch it on the way out.

So every read carries an explicit scope instead of inheriting one from the connection. Org scoping, per-connection visibility, and per-resource membership all become part of the SQL that runs. A connection backed by a personal OAuth login is forced private by a database trigger. When a membership sync goes stale the gate fails closed, so a stalled sync hides data rather than leaking it.

None of this is beyond a warehouse. Bind a request-scoped identity, let engine-level policies do the work, and you have built the same thing the right way. What a warehouse gives you is permission to defer it, because a missing policy usually shows up as a dashboard nobody can open. An agent turns that same gap into a fluent sentence containing someone else’s data. So the scope has to be mandatory on every read, not something you add when the compliance review lands.

You cannot re-run a Behavior

The closest thing we have to a dbt model is a Behavior: a declared unit of work that runs on a cadence, reads the log, and writes derived events back. Underneath, a tick enqueues a row on a job queue, and that row moves through pending, claimed, and running, with heartbeat-based reaping when a worker dies. If you have configured a Snowflake Task or an Airflow DAG, you know this shape already.

The familiarity stops there.

Re-running a dbt model is usually cheap and usually safe. Not always: incremental models and snapshots carry state, sources move underneath you, and a full refresh on a large table costs real money. But the worst case is a wrong table or a bigger bill, which is why the ecosystem can treat re-running as the default recovery. Something looks off, you run it again.

A Behavior takes actions in systems we do not own. It sends the email. No --full-refresh unsends it, and no amount of compute buys the message back.

That one property is why everything around it looks stricter than a warehouse’s scheduler. A run is claimed rather than merely scheduled. The claim query only ever selects work a human already approved. An approval is a durable event rather than a flag somebody flips in a UI. The orchestration is heavier because a failed run is not a stale table.

What the warehouse still does better

If your question is analytical, the warehouse wins and it is not close. Columnar storage and decades of query optimization beat a Postgres event log at scanning a billion rows for one aggregate, and the dbt ecosystem around it is better than anything we would build. We do not compete there. Metrics that need the warehouse read the warehouse, live, over a connector, and the answer gets composed rather than copied.

So the claim is narrower than replacing anything. Agent memory should borrow warehouse practice for ingestion, modelling, and provenance, because that work is done and it is good. What it cannot borrow is the shape. The loop has to close. Derived state stays a cache. The scope travels with the read. The transform is unsafe to repeat.

None of those four come from AI being special. They come from having a reader that acts on what it finds, in systems nobody here owns.

Your systems still own the truth. This store keeps what was observed, who is allowed to see it, and what every agent did about it.

Start with the docs, or read how the memory layer works.