Why agent memory leaks permissions, and why prompt-filtering can't fix it

We're connecting agents to the systems where companies keep their real knowledge: Google Drive, Gmail, Salesforce, Notion, the ticket queue. And we're giving those agents memory, a place to write down what they learn so the next session or the next agent doesn't start from scratch.

That memory store is a new kind of thing, and it has a problem almost nobody is looking at. It's where knowledge pulled out of permissioned systems piles up, disconnected from the permissions it came from. The moment an agent writes a memory, it has copied a fact out from behind an ACL, and most memory layers have no idea that ACL ever existed.

The leak

This isn't an exotic edge case. It's the default behavior.

An agent working for the Acme account team reads Acme's renewal contract. It's allowed to; that's the job. It writes down a memory: "Acme's renewal is $61k, deal-desk floor is $58k." The memory goes into the shared store, embedded and tagged with whoever the agent was acting as.

Three weeks later, a different session runs a semantic recall for "renewal pricing benchmarks." Maybe it's an agent helping a different customer, maybe a different rep entirely. The Acme memory is a strong match, so it comes back, and it ends up in a reply.

Nobody attacked anything. The source contract said account-team-only. The memory said nothing at all. The permission got laundered at the write step: the memory layer turned a permissioned document into an unpermissioned fact, then handed that fact to whoever asked a close-enough question. It's the same family of problem as the Copilot oversharing audits and the EchoLeak bug (CVE-2025-32711): knowledge crossing a trust boundary it was never cleared for. A shared memory layer just makes it systematic instead of occasional.

Why the obvious fixes don't hold

Three fixes get reached for. None of them close it.

The first is to namespace by user_id, which is what most memory tools do. Memory is scoped to the user who wrote it. This breaks the first time it meets a real company, because enterprise access isn't organized per user. It's organized by group, by role, by who a document is shared with. Alice and Bob can both see the roadmap because they're both in Engineering. A per-user namespace either cuts Alice off from her own team's shared memory, or you widen the namespace until the boundary is gone. And it never captures the part that matters, which is not who wrote the memory but who is allowed to see the source it came from.

The second is to filter after retrieval: pull the top-k by similarity, then drop the hits the caller can't see. That has two problems. It quietly wrecks recall, because you asked for ten results and got three, and you have no way to know whether the seven that got dropped were the ones you wanted. And the filter is only as good as the ACL data you attached after the fact, which tends to be stale, partial, or missing.

The third is to tell the model not to leak. You put a line in the system prompt: never reveal documents the user can't access. This is the most common approach and the least sound.

A prompt is a request, not a guarantee.

A language model is a probabilistic text generator, and it has no concept of a permission it can be made to honor. A prompt injection gets around it, so does a jailbreak, and so does an unlucky sample. You can't ask the thing you're trying to contain to also be the thing that contains it. That's the reason prompt-filtering doesn't work: access control doesn't belong in the model, no matter how carefully you word the request.

Where it has to live

If the model can't be trusted to hold the line, the line has to sit underneath the model. It runs in the index, before anything is retrieved, as a hard filter rather than a polite request or a cleanup pass at the end.

In practice that's three pieces. You inherit the source system's own ACLs, the Drive shares and group memberships and CRM sharing rules, into a permission graph, so the memory layer knows what the document knew. You resolve the caller's identity into the exact set of permission tokens they're cleared for, following every group they belong to. And you compile that set into the retrieval query as a required filter that runs inside the index, before ranking, before the model sees anything. There's no LLM call on that path and no live authorization check, because it all resolves ahead of time. If the caller holds no matching token, the memory simply isn't in the results.

That buys you one guarantee.

The model can't leak what the index never handed it.

A prompt injection has nothing to work with, because the filter already ran, one layer below where the prompt can reach.

The honest part

I'm building an open-source project in this area, called Verity, so keep that in mind while you read. But the failure mode is real whether or not you ever look at my code, and it's worth building around either way.

It's a hard problem, so here's what I'm not going to claim. Permission fidelity is only as good as what a source API exposes. Google Drive gives you true per-item sharing; some systems don't, and when that happens the right move is to fail closed and over-hide rather than guess. A "zero leaks" number is only as strong as the test behind it, and mine comes from an adversarial harness run against planted sentinels. It's reproducible and I stand behind it, but it's a test suite, not a third-party audit. And the permission-blind memory tools, mem0 and Zep and Letta, aren't doing anything wrong. They're built for personal and consumer memory, where there's no boundary to cross in the first place.

But if you're putting an agent on top of shared enterprise data, with several customers and several teams reading from one memory store, permission-awareness isn't something you add later. It has to be there on day one, because the leak is the normal case, not the corner case.

There's a one-command harness that tries to trigger the leak and tells you whether it happened. Run it against your own setup and tell me where it falls over: github.com/RunAlphaLoop/verity.