Zero leaks isn't a benchmark. Here's the one we ran.

Every memory vendor claims zero leaks. It's the easiest number in the world to say and the hardest to back. The question that matters is never "did you leak?" It's "measured against what?" A leak test with no oracle is a vibe with a green checkmark.

An oracle, or it didn't happen

For a permission-aware memory layer, a leak is one thing: you surfaced a record to a user who wasn't allowed to see it. To count leaks you need a second opinion you trust more than your own. That second opinion is the oracle.

Most "zero leaks" claims skip it. They plant a sentinel in a synthetic fixture, check that the sentinel doesn't come back, and call it a day. That's a real test and we run one too, but it grades your own homework against secrets you planted yourself. It proves your enforcement is internally consistent. It says nothing about whether your reconstruction of someone else's permission model is correct.

For a Salesforce connector, that reconstruction is exactly the interesting part. Verity never asks Salesforce at read time, since that would put a live ReBAC call on the read path, which we forbid. It reconstructs Salesforce's sharing model into a materialized index ahead of time and filters against that. So the thing under test is whether our offline reconstruction agrees with Salesforce's live answer.

A leak test with no oracle is a vibe with a green checkmark.

We used Salesforce's own access API

Salesforce ships an API that answers the exact question we care about: UserRecordAccess. Give it a user and a record and it tells you whether that user can read that record, computed by Salesforce's own sharing engine. Org-wide defaults, role hierarchy, sharing, View-All, all of it, resolved the way Salesforce would resolve it for that user.

So the harness is simple to describe. Take a set of (user, record) pairs. Ask UserRecordAccess for each. Ask the Verity index for each. Line the two answers up and look at the disagreements.

There are only two kinds of disagreement, and they aren't symmetric. A leak is Verity showing a record UserRecordAccess said hide. An over-hide is Verity hiding a record UserRecordAccess said show. A security tool has to drive the first set to empty. It's allowed to carry some of the second.

The join that quietly decides everything

Before any of this means anything, you have to know that Verity's "user" and Salesforce's "user" are the same person. Get that wrong and the whole audit is theater, because you're comparing two different people's access and calling it agreement.

The tempting join key is email, and it's the wrong one. Email isn't stable across SSO. It gets reused, aliased, and changed. Matching on it is how you manufacture the exact leak you were trying to prevent.

Verity's Salesforce crosswalk resolves on FederationIdentifier, to the SSO alias, to the canonical identity, not on User.Email. That's the administratively asserted link between the Salesforce account and the IdP subject the caller actually presents. When that link can't be made, the correct behavior is to deny, not to guess.

Matching on email is how you manufacture the exact leak you were trying to prevent.

What we measured

On a real Salesforce trial org, reconciling Verity's index decisions against UserRecordAccess, the leak set was empty. Zero cases where Verity surfaced a record Salesforce said the user could not see.

The over-hide set was not empty, and that's worth saying out loud instead of hiding. Where Verity hid records Salesforce would have shown, the misses traced to two places: inactive accounts and accounts with no FederationIdentifier, whose access can't be mapped to a caller identity, and the reconstruction of org-wide View-All grants.

Over-hiding is the safe direction to be wrong. Hide a record the user was allowed to see and they lose a result, then go find it in Salesforce directly. Show a record they weren't allowed to see and that's the failure this whole project exists to prevent. When the reconstruction is uncertain, it fails closed.

Over-hiding is the safe direction to be wrong.

What this number is not

This is a test harness, not a third-party audit. The result is reproducible and it holds: run it and you get the same empty leak set. But a reproducible harness isn't an auditor's letter, and it ran against a trial org with synthetic data, not a production org at scale. Nobody outside the project has signed anything.

The audit only covers what we could reconcile against the oracle, which is not the whole Salesforce sharing model. Org-wide defaults, role hierarchy, object sharing, and View-All reconstruction are in. Sharing-rule and territory reconstruction are deferred until a real org that has those features configured can measure them. Until then we don't claim fidelity on the parts UserRecordAccess never got to check. Where a source's API doesn't expose enough to reconstruct access faithfully, the honest move is the same one the over-hides show: fail closed and over-hide.

Run the harness against your own org and tell us where it breaks: github.com/RunAlphaLoop/verity