The clean install that went dark

We have a launch gate: install Verity from scratch on a throwaway VM, run the quickstart, watch it work. The install came up clean. Every health check was green. Then we ran a recall and it returned nothing. Not an error. Nothing.

Everything green, answers empty

The quickstart is five minutes: download the dev binary, point an agent at it, ingest a few documents, ask a question. On the throwaway VM it all went fine. The server started. The SpiceDB process came up. The web UI loaded. The ingest ran without complaint.

Then the first recall came back with an empty result set. We tried a broader query. Empty. We tried a query we knew matched a document that had just been indexed. Empty. The data was clearly in the index; the rows were sitting in the storage layer. But every query path ran through the same freshness fence, so every one of them came back empty.

On an empty result your first instinct is that something broke upstream. Nothing had. Nothing threw, nothing logged an error at the level we were watching, and the process stayed healthy the whole time. The system was behaving exactly as designed, and the design had a hole in it.

Why the read path would rather say nothing

Verity's read path is fail-closed on freshness. Before it serves a recall, it wants to know that the permission data behind the index is current. If it cannot confirm that, it will not serve a possibly-stale answer. It returns empty.

This is deliberate. A permission-aware memory layer that serves stale ACLs is worse than one that serves nothing, because stale ACLs are how a revoked user keeps reading a document they should no longer see. So a staleness fence sits above the storage layer, and when the fence is not satisfied that the world is fresh, it blanks the result.

Freshness is confirmed, not guessed. It comes from a live change stream out of the authorization datastore. SpiceDB, the ReBAC engine, has a push Watch API: as permission tuples change, it emits events. Verity consumes that stream to keep its materialized visibility tokens current, and the fence uses the stream's connected state as its freshness signal. If the watch is connected and caught up, the fence is satisfied and recall runs.

A permission-aware memory layer that serves stale ACLs is worse than one that serves nothing.

Following the empty result back to its cause

Here is the chain we walked. The fence blanks recall because it cannot confirm freshness. It cannot confirm freshness because the watch never reports connected. The watch never reports connected because it has not received an event. And it has not received an event because the datastore is brand new and completely idle. There have been zero permission writes, so there is nothing for the watch to emit.

On an idle datastore the watch sits there correctly waiting, having done nothing wrong, producing no evidence that it is alive. The fence, seeing no evidence of liveness, makes the safe call: treat the permission data as infinitely stale, fail closed, return empty.

So a healthy install with a working watch, a working index, and correctly ingested data returned empty for every query, forever, until the first permission write happened to arrive and wake the stream up. On a fresh quickstart that write might never come. The install had gone dark on its very first read and would have stayed dark.

Prime the stream so silence means something

The fix, commit 7cb80c9, is a startup watch probe. On boot, before the read path is live, Verity primes the watch so the stream establishes and reports connected even when the datastore has never seen a write. Now the fence has a positive signal to read: the watch is connected, it is caught up to the current revision, freshness is confirmed. An empty datastore now yields correct, non-empty recall.

The probe does not weaken the fence. A stalled or disconnected watch still fails the freshness check and still blanks recall, which is what we want. What changed is that a healthy-but-idle watch now looks different from a stalled one, because we force it to announce itself instead of inferring its state from silence.

Absence of events is not a signal

The general lesson is worth more than the specific commit. If you build a fail-closed freshness fence whose input is a change stream, you have to make "no data yet" distinguishable from "stale data." An unprimed liveness signal conflates the two, because both look like an empty stream.

The fence was correct. Failing closed on unconfirmed freshness is the right call for anything on a permission-sensitive read path. The defect was upstream of the fence, in what the fence was allowed to treat as a freshness signal. It was reading the absence of events as the absence of liveness, and on a quiet system those are not the same thing.

So here is the rule we now hold: freshness needs a positive liveness signal, not the mere absence of events. If your only evidence that a stream is alive is that it keeps sending you things, then a correctly-quiet stream and a dead stream are indistinguishable, and a fail-closed system will treat both as failure. Give the stream a way to say "I am here and I am caught up" out loud, at startup, before anyone reads.

Freshness needs a positive liveness signal, not the mere absence of events.

The honest part

Fail-closed on the read path means Verity will sometimes return empty when you would rather it return something stale-but-useful. That is the deliberate trade, and it is not free. On a watch reconnect or a cold start, the safe answer is the empty one and a user sees a gap. We chose that over serving an answer built on permission data we cannot vouch for.

The bug here was never the fence. The fence did its job. We handed it a liveness signal we had not primed, so it could not tell a fresh-and-idle system from a stale one, and it made the safe call on bad evidence. Fixing it meant giving the fence better evidence, not making the fence more permissive.

One thing to keep honest: this was caught by our own launch gate on a throwaway VM, not by an outside auditor or a production incident. There are no production users yet. The gate exists because a clean-install path is easy to never actually test, and this bug is what happens when you don't.

Run the from-scratch quickstart yourself and try to break it: github.com/RunAlphaLoop/verity