Ingestion · deep dive

Getting data in

Verity ingestion is a layered funnel: many doors, one structural choke point. Every door demands an explicit visibility decision and terminates at the same fail-closed gate in the Rust write path. This page walks every entry point with a working example, from the five-minute CLI wow to source-fidelity native connectors.

The BYOT doctrine: bring your own token

Verity ships zero OAuth apps. Bring-your-own-token is the only auth mode of the open-source core. Every connector quickstart reads the same way: create a key, private app, service account, or self-registered OAuth client in your own tenant, and paste it into Verity. The July 2026 coverage survey confirmed this is viable for 20/20 surveyed systems — no vendor-registered app is required for a customer to grant first-party access.

Three engineering consequences are absorbed once in the ingest SDK rather than per connector:

• Credential lifecycle, four shapes: static key/PAT, client-credentials minting, refresh-token rotation, and service-account JWT — plus expiry telemetry and a 401-invalidate-retry-once hook.
• Webhook receivability, three ways: a public HTTPS ingest endpoint; Socket-Mode delivery (Slack); or a delta/poll fallback so no source is ever push-blocked, including firewalled self-hosters.
• Credential wizards, not OAuth clients: verity-cli connect slack opens the app-from-manifest flow; verity-cli connect github uses a pasted fine-grained PAT once, then never stores it.

The layered funnel

Entry points, in adoption order. Each reuses the same push lane, HMAC scope handles, and fail-closed quarantine. The two lanes — convenience (snapshot-grade, admin-assigned visibility) and truth (source-fidelity ACLs, push freshness) — are labeled everywhere and never blurred.

The one choke point

No SDK, sink, CLI flag, or endpoint can bypass the gate that governs all of them:

1 · verity-cli

The developer front door. One binary drives a running server over REST — it holds no database access and no enforcement logic of its own.

dev — the whole local plane, one command

cargo run --release -p verity-cli -- dev
# docker Postgres + the server + a "dev" tenant + an org-wide scope handle + a written config.
# Fully idempotent — re-run it anytime.

add --visibility — ingest with an explicit visibility

add accepts a file, a directory (recursive over .txt/.md/.json/.csv/.html, capped at 200 files), an http(s) URL, or stdin (-). --visibility is required by the argument parser and has no default — Verity never guesses who may see a memory. Omitting it is a usage error that names the invariant. Under the hood, add mints a short-lived scope whose principals are exactly your --visibility tokens and uploads under that handle.

verity-cli add ./docs --visibility 1                       # a directory, org-wide token
verity-cli add report.md --visibility 7,11 --entity account:acme  # narrow visibility + entity tag
echo "quick note" | verity-cli add - --visibility 1              # stdin

query — scoped hybrid recall

verity-cli query "what do we know about pricing?"   # uses the handle saved by `dev`
verity-cli query "renewal risks" -k 5 --json       # custom k, raw JSON hits

Every hit already passed your scope's visibility pre-filter in the index — there is no post-hoc filtering step to forget.

webhook mint — any JSON-POSTing system as a source

verity-cli webhook mint my-system --visibility 1
# The token in the returned URL is the credential, shown exactly once.
# Ingested memory carries source "webhook:my-system".

Watch what the gate refuses with verity-cli tail — it polls the quarantine and prints the raw payloads Verity declined to index permissively.

2 · Canonical envelope endpoints

The curl-in-60-seconds lane. Two shapes cover most structured ingestion.

POST /v1/episodes — an observation

Writes a durable observation under a scope handle. Visibility is inherited from the handle's principals; entities must be inside the handle's entity scope (omit to inherit the whole scope).

curl -s -X POST "$VERITY/v1/episodes" -H 'content-type: application/json' -d '{
  "scope_handle": "'"$HANDLE"'",
  "observation": "Acme confirmed the renewal moves to their Q4 board meeting.",
  "entities": ["account:acme"]
}'
# → { "episode_id": "0192..." }  — immediately searchable via /v1/recall

POST /v1/ingest/debezium — the CDC truth lane

Accepts a Debezium change envelope (or a JSON array of them) and turns it into a deterministic keyed L1 upsert. This endpoint is on the trusted connector plane: it is gated by the admin bearer token, not a scope handle. The ?tenant_id= query param names the tenant; ?pk= overrides the primary-key field (default id).

curl -s -X POST "$VERITY/v1/ingest/debezium?tenant_id=$TENANT" \
  -H 'content-type: application/json' \
  -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"payload":{
       "after":{"id":"opp-1","amount":84000,"stage":"negotiation"},
       "source":{"connector":"postgresql","db":"crm","table":"opportunities","ts_ms":1752000000000},
       "op":"u"}}'
# → { "facts_inserted":0, "facts_superseded":3, "facts_unchanged":0, "facts_retired":0 }

An op:"u"/"c" upserts each field of the row image; op:"d" retires the entity. Facts written this way carry mirrored provenance and record a freshness sample. The value becomes queryable at GET /v1/records/postgresql:crm.opportunities/opp-1/amount — measured at 31 ms POST-to-queryable on the dev profile.

3 · Minted scoped webhook URLs

Any system that can POST JSON — GitHub, Stripe, Zapier, n8n, cron jobs, internal services — becomes a push source with no connector and no OAuth. The visibility policy is bound at mint time into the URL; the blast radius is one URL, instantly revocable.

curl -s -X POST "$VERITY/v1/webhooks" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"tenant_id":"'"$TENANT"'","name":"demo-system","visibility":[11],
       "entity_scope":["account:acme"]}'
# → { "webhook_id":"0192...", "url":"/wh/n8sQ...raw-token" }

curl -s -X POST "$VERITY/wh/n8sQ...raw-token" -H 'content-type: application/json' \
  -d '{"content":"Acme signed the pilot agreement for the fall rollout.",
       "entities":["account:acme"]}'
# → 200 { "episode_id":"...", "chunks_indexed":1, "facts_written":0 }

curl -s -X POST "$VERITY/wh/n8sQ...raw-token" -H 'content-type: application/json' \
  -d '{"weird":"vendor shape we do not understand"}'
# → 202 { "quarantined": true }   (view it: verity-cli tail  /  GET /v1/admin/quarantine)

Facts and chunks from a webhook carry admin-assigned provenance — the visibility was an explicit admin decision at mint, not a mirrored source ACL. Revoke with DELETE /v1/webhooks/{id}; the URL stops resolving immediately.

4 · File drop

POST /v1/files is a multipart upload: scope_handle, an optional comma-separated entities, and the file part. The server splits the text into paragraph chunks and embeds each. Visibility is inherited from the scope handle the file is uploaded under.

curl -s -X POST "$VERITY/v1/files" \
  -F "scope_handle=$HANDLE" \
  -F "entities=account:acme" \
  -F "file=@handbook.md"
# → { "episode_id":"...", "chunks_indexed": 12 }

5 · Framework sinks

Six thin adapters plug into each framework's documented memory extension point, turning hundreds of community loaders into Verity sinks. Every one takes a required visibility_policy constructor argument with no default — loaders strip source ACLs by construction, so this lane is always policy-based (admin-assigned provenance), and anything arriving without a policy hits quarantine. Bypass is impossible, not discouraged.

LlamaIndex

from verity_llamaindex import VerityVectorStore

store = VerityVectorStore(
    verity_url="http://127.0.0.1:7717",
    tenant_id=TENANT,
    visibility_policy=[1],          # REQUIRED — no default; missing → quarantine
    admin_token=ADMIN_TOKEN,
)
# every LlamaHub reader now writes into permission-aware Verity memory

LangChain

from verity_langchain import VerityVectorStore

store = VerityVectorStore(verity_url=URL, tenant_id=TENANT,
                          visibility_policy=[1], admin_token=ADMIN_TOKEN)
retriever = store.as_retriever()   # or verity_langchain.VerityRetriever

LangGraph

from verity_langgraph import VerityStore

store = VerityStore(verity_url=URL, tenant_id=TENANT,
                    visibility_policy=[1], admin_token=ADMIN_TOKEN)
# put() ingests; search() mints an entity-scoped handle and recalls; get() reads the brief

CrewAI

from verity_crewai import VerityStorage   # StorageBackend (CrewAI 1.0)

storage = VerityStorage(verity_url=URL, tenant_id=TENANT,
                        visibility_policy=[1], admin_token=ADMIN_TOKEN)

Google ADK

from verity_adk import VerityMemoryService

memory = VerityMemoryService(verity_url=URL, tenant_id=TENANT,
                             visibility_policy=[1], admin_token=ADMIN_TOKEN)
# add_session_to_memory(session) ingests; search_memory(...) recalls

OpenAI Agents SDK

from verity_openai_agents import VeritySession

session = VeritySession("conv-42", verity_url=URL, tenant_id=TENANT,
                        visibility_policy=[1], admin_token=ADMIN_TOKEN)
# add_items() ingests; get_items() recalls, ordered by item seq

6 · MCP write tools

Paste one MCP config into Claude Code or Cursor and the agent itself becomes the universal connector — zero installs for users already in an agent. Four write tools reach the same choke point; visibility is inherited from the agent's scope handle, so there is no way to write something the caller couldn't already see.

memory_remember

Append a one-line observation (immutable episode + immediately searchable chunk). For durable knowledge learned during the task.

memory_ingest_text

Ingest a document-sized piece of text verbatim — pasted docs, transcripts, reports held in-context.

memory_ingest_file

Read a local UTF-8 text file (.txt/.md/.json/.csv/.html, ≤ 512 KB) and ingest it.

memory_ingest_url

Fetch a public http(s) URL (10 s timeout, 2 MB cap), reduce HTML to text, and ingest it.

// an agent's tool call — the scope_handle comes from memory_open_scope,
// which fixes identity from server config, never from arguments
{ "tool": "memory_ingest_text",
  "arguments": {
    "scope_handle": "vs_9f2...",
    "content": "Meeting notes: Acme wants net-30 terms and a Q4 close...",
    "entities": ["account:acme"] } }

See the API reference for all 14 MCP tools.

7 · Native connectors (the truth lane)

Native connectors give source-fidelity ACL envelopes and push freshness — the graduation path and the moat. Each is configured with a first-party credential you create in your own tenant. The Python ingest plane runs them; connectors are never on the read path.

HubSpot — private-app token, ACL tier C

Auth is a private-app token created in your own portal (~2 min), read from HUBSPOT_PRIVATE_APP_TOKEN. HubSpot exposes no per-record ACL API, so it is ACL tier C: the connector cannot mint a faithful ACL envelope and instead requires an admin-assigned visibility_policy with no default. Every emitted event carries it; provenance is admin-assigned. The truth lane polls the CRM search API on hs_lastmodifieddate (contacts use lastmodifieddate — a documented quirk); the push lane maps UI-configured v3 webhook payloads.

export HUBSPOT_PRIVATE_APP_TOKEN=pat-na1-...
python -m verity_ingest.connectors.hubspot --once --visibility 1,2
python -m verity_ingest.connectors.hubspot --webhook-file payload.json --visibility 1,2

Google Drive — service account, ACL tier A (mirrored)

Auth is your own service account with domain-wide delegation configured in your admin console; the key path comes from GOOGLE_APPLICATION_CREDENTIALS. Drive is ACL tier A — permissions.list is best-in-class — so this is the mirroring proof. The connector polls changes.list, and for each changed file fetches metadata, then the ACL, then — only if the ACL resolves — content (Google Docs exported as text; text/* and JSON downloaded directly). ACL-before-content is the rule.

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
python -m verity_ingest.connectors.gdrive --once            # --dry-run prints request bodies

Because the ACL is mirrored, Drive documents carry mirrored provenance — a Verity scope inherits exactly the Drive permissions on each file. This is how a Drive ACL becomes an in-index pre-filter (see Permissions → ACL inheritance).

Salesforce — customer-created Connected App, ACL tier A (hardest)

Auth is a Connected App you create in your own org with the client-credentials flow (SF_MY_DOMAIN / SF_CLIENT_ID / SF_CLIENT_SECRET). Salesforce's client-credentials response carries no expires_in and no refresh token, so the token is cached until a 401 triggers the shared retry-once hook. The truth lane runs SOQL on LastModifiedDate for Account, Contact, and Opportunity.

Debezium / Postgres CDC — the built-in truth lane

Anything Debezium speaks (Postgres, MySQL, …) posts change envelopes to POST /v1/ingest/debezium (see above). ACL fidelity is mirrored when the schema declares principal columns via a manifest, or admin-assigned under a static policy. For a lightweight on-ramp, a copy-paste pg_net/Supabase trigger POSTs row changes to a minted webhook URL — CDC-lite into the same lane (31 ms POST-to-queryable).

8 · Source manifests (connectors as config)

A source manifest turns any REST/webhook source into a reviewable YAML file — no connector code, no vendor OAuth. The manifest is data, not code: the Rust runtime (crates/verity-manifest) interprets it with hard evaluator limits and a fail-closed ACL gate; nothing in a manifest can execute or touch the network.

The shipped, fixture-verified example is examples/linear.yaml:

manifest_version: 1
source:
  name: linear                       # [a-z0-9_-]; becomes the L1 `source`
  tier: B                            # A | B | C — gate-enforced ACL contract
  webhook:
    signature:
      scheme: hmac_sha256            # failed signature → 401, no ingestion at all
      header: Linear-Signature
      secret_ref: secret://linear-webhook-secret
entities:
  - type: issue
    route:
      when: "type = 'Issue' and action in ['create','update']"
      operation: upsert              # upsert only in v1
    primary_key: "data.id"           # deterministic PK → idempotent replays
    valid_from: "data.updatedAt"     # dot-path or $now()
    map:
      title: "data.title"
      state: "data.state.name"
    content: "data.description"      # optional: text → a retrieval chunk
acl_policy:                          # absent ⇒ activation refused; runtime quarantines all
  mode: map                          # map | static | quarantine
  identity_namespace: source_native_id
  principals: "data.team.members[].id"
  approximation: true                  # mandatory for Tier B; note surfaces to the approver
  note: "Team membership approximates issue visibility."

acl_policy: three modes, tier contracts, human gate

The tier contract is gate-enforced: Tier A must be map with approximation:false; Tier B must be map with approximation:true and a human-readable note; Tier C must be static. Activating a manifest requires an explicit approved_by recorded in the audit log. An LLM may draft everything except the acl_policy — an unreviewed manifest can only quarantine, structurally.

# upload (draft) → activate (human gate) → bind to a webhook
curl -s -X POST "$VERITY/v1/manifests" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"tenant_id":"'"$TENANT"'","yaml":"'"$(cat linear.yaml)"'"}'
curl -s -X POST "$VERITY/v1/manifests/$ID/activate" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"tenant_id":"'"$TENANT"'","approved_by":"admin@corp.example"}'
curl -s -X POST "$VERITY/v1/webhooks" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"tenant_id":"'"$TENANT"'","name":"linear","visibility":[1],"manifest_id":"'"$ID"'"}'

ACL provenance & the quarantine rule

Every fact and chunk carries exactly one provenance tag, surfaced on every read:

• mirrored — from a real source ACL API (Drive permissions.list).
• approximated — from a proxy for the real ACL (team membership).
• admin-assigned — from an explicit policy at ingest (webhook, CLI, sink, static manifest).
• quarantined — no usable visibility could be derived; held for review.

Entry-point comparison

Pick a door by the trade-off you want. Freshness and ACL fidelity climb together toward the truth lane; effort climbs with them.