Connecting · per-system runbooks

Connecting your systems

Ingestion is the mechanism reference — every door and how the write path treats it. This page is the operator's counterpart: “I have HubSpot / Salesforce / a Google Drive / a Postgres database / a pile of docs / an internal system — walk me through connecting it and getting my existing data in for the first time.” One runbook per system, grounded in the real connector source.

The universal shape

Verity ships zero OAuth apps. Bring-your-own-token is the only auth mode of the open-source core: you create a first-party credential in your own tenant — a HubSpot private app, a Salesforce Connected App, a Google service account — and paste it into Verity. Your tokens never rest in a third-party cloud. Every runbook below reads the same five steps:

Common prerequisites

Every runbook assumes a running Verity server and a tenant. Create both once:

# local all-in-one: docker Postgres + server + a "dev" tenant + a scope handle
cargo run --release -p verity-cli -- dev

# or, against an existing server, mint a tenant on the trusted admin plane
export VERITY_URL=http://127.0.0.1:7717
export VERITY_ADMIN_TOKEN=...              # the server's admin bearer, if configured
export TENANT=$(curl -s -X POST "$VERITY_URL/v1/admin/tenants" \
  -H 'content-type: application/json' -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"name":"acme-corp"}' | jq -r .tenant_id)

The Python ingest plane (connectors) reads three shared env vars, plus each connector's own credential:

VERITY_URL

The server base URL the connector's sink POSTs to. Debezium-lane connectors (HubSpot, Salesforce) default to http://127.0.0.1:7717; the Drive connector defaults to http://localhost:8080 — set it explicitly to your server.

VERITY_TENANT_ID

The tenant UUID from above. Required by the Debezium sink; the Drive runner falls back to "default".

VERITY_ADMIN_TOKEN

Sent as the bearer token on the trusted connector plane (/v1/ingest/debezium, /v1/ingest/documents, /v1/admin/principals) when the server requires it.

Install the ingest SDK: pip install verity-ingest (the Drive connector needs the extra: pip install 'verity-ingest[gdrive]' for google-auth).

HubSpot

The flagship connector. HubSpot exposes no per-record ACL API, so it is ACL tier C: the connector cannot mint a faithful ACL envelope and instead requires an admin-assigned visibility_policy with no default. Provenance on every emitted fact is admin-assigned.

1 · Create the credential

In your own HubSpot portal, go to Settings → Integrations → Private Apps, create a private app, grant it the CRM read scopes you want mirrored (contacts, companies, deals), and copy its token — it looks like pat-na1-.... This is BYOT: no vendor-hosted app, ~2 minutes.

2 · Configure

export HUBSPOT_PRIVATE_APP_TOKEN=pat-na1-...
export VERITY_URL=http://127.0.0.1:7717
export VERITY_TENANT_ID=$TENANT
export VERITY_ADMIN_TOKEN=...              # if the ingest plane requires it

3 · Run the initial backfill

--once runs one truth-lane poll cycle. With no saved cursor this is a full backfill: it searches contacts, companies, and deals on the last-modified property (hs_lastmodifieddate; contacts use lastmodifieddate — a documented HubSpot quirk) from epoch, ascending, paginated at 100/page, honoring the search API's 429 Retry-After. Each non-null property becomes one deterministic keyed L1 fact via POST /v1/ingest/debezium; the last-modified timestamp becomes valid_from. The cursor (max last-modified seen) is written to the state file at the end.

python -m verity_ingest.connectors.hubspot --once --visibility 1,2
# poll: 148 fact event(s), cursor -> 2026-07-09T18:04:57.000Z -> {written:148,...}
# cursor persists to .verity/hubspot_cursor (override: --state-file / $HUBSPOT_STATE_FILE)

4 · Verify

Facts land keyed (source="hubspot", table=<object>, id=<record id>, field) — the L1 partition is hubspot:<object>. Point-read a field, then confirm scoping: a scope whose principals include your policy tokens sees it; one that does not sees nothing.

# mint a scope handle whose principals include a policy token (e.g. 1)
IN=$(curl -s -X POST "$VERITY_URL/v1/scopes" -H 'content-type: application/json' \
  -d "{\"tenant_id\":\"$TENANT\",\"principals\":[1],\"entity_scope\":[]}" | jq -r .scope_handle)

# the record is queryable at its keyed L1 address
curl -s "$VERITY_URL/v1/records/hubspot:deals/<deal-id>/amount?scope_handle=$IN" | jq .value
# → "84000"

# a scope holding NO policy token sees nothing — fail closed
OUT=$(curl -s -X POST "$VERITY_URL/v1/scopes" -H 'content-type: application/json' \
  -d "{\"tenant_id\":\"$TENANT\",\"principals\":[999],\"entity_scope\":[]}" | jq -r .scope_handle)
curl -s -o /dev/null -w '%{http_code}\n' "$VERITY_URL/v1/records/hubspot:deals/<deal-id>/amount?scope_handle=$OUT"
# → 404  (out-of-scope value is not visible to this principal set)

5 · Ongoing sync

Two options. Re-run --once on a cron (the saved cursor makes it incremental), or push: HubSpot v3 webhook subscriptions are UI-configured under the private app and deliver to a Verity-minted webhook URL. Decode a recorded delivery with --webhook-file (only *.propertyChange events map to facts):

python -m verity_ingest.connectors.hubspot --webhook-file payload.json --visibility 1,2

For a supervised schedule instead of cron, see Ongoing sync (Temporal).

Salesforce

Salesforce is ACL tier A (the *Share tables are readable) but the hardest reconstruction of the 20 surveyed systems. The connector does not reconstruct full effective visibility; enforcement uses the same fail-closed admin-assigned visibility_policy as HubSpot, and share rows ride along as approximated additive metadata.

1 · Create the credential

In your own org, create a Connected App and enable the OAuth client-credentials flow with a run-as integration user. This survived the post-Sept-2025 crackdown that made vendor-distributed apps harder — customer-created stays easy. Copy the consumer key and secret. Note your My Domain (e.g. acme → acme.my.salesforce.com). Salesforce's client-credentials response carries no expires_in and no refresh token, so the token is cached until a 401 (INVALID_SESSION_ID) triggers the shared retry-once hook.

2 · Configure

export SF_MY_DOMAIN=acme                    # or the full acme.my.salesforce.com host
export SF_CLIENT_ID=...                       # Connected App consumer key
export SF_CLIENT_SECRET=...                   # Connected App consumer secret
export VERITY_URL=http://127.0.0.1:7717
export VERITY_TENANT_ID=$TENANT
export VERITY_ADMIN_TOKEN=...

3 · Run the initial backfill

One cycle runs SOQL through GET /services/data/v62.0/query for Account, Contact, and Opportunity with WHERE LastModifiedDate > <cursor> ORDER BY LastModifiedDate ASC — with no cursor, that is a full backfill from epoch, following nextRecordsUrl (queryMore) pagination. Each non-null field becomes one keyed L1 fact. For Accounts changed in the window, the connector additionally fetches AccountShare rows and attaches best-effort share principals (user:005…, group:00G…) as approximated metadata — a failed share fetch logs a warning and never gates the facts.

python -m verity_ingest.connectors.salesforce --once --visibility 1,2
# poll: 312 fact event(s), cursor -> 2026-07-09T18:04:57.000+0000 -> {written:312,...}
# --no-shares skips the AccountShare fetch; cursor -> .verity/salesforce_cursor

4 · Verify

Records land at the salesforce:<sobject> partition, keyed by the Salesforce Id. Same scoped check as HubSpot — in-policy scope sees it, out-of-policy scope gets a 404:

curl -s "$VERITY_URL/v1/records/salesforce:Account/<account-id>/Name?scope_handle=$IN" | jq .value
# → "Acme Corporation"
curl -s "$VERITY_URL/v1/records/salesforce:Opportunity/<opp-id>/Amount?scope_handle=$IN" | jq .value
# → "84000.0"     (out-of-policy scope → 404, fail closed)

5 · Ongoing sync

Re-run --once on a cron or Temporal schedule; the saved cursor makes each cycle incremental (truncated to whole seconds, so a ≤1s window may replay safely into keyed upserts). The push lane — Salesforce CDC over the Pub/Sub API (gRPC) — is a documented no-op in this poll-first connector; the truth lane reconciles everything. Live-org validation awaits a design partner (see Status & honesty).

Google Drive

Drive is ACL tier A and the mirroring proof: permissions.list is best-in-class, so documents carry mirrored provenance — a Verity scope inherits exactly the Drive permissions on each file. This is the one connector with no admin-assigned --visibility: visibility comes from the source ACL, resolved through the principal registry.

1 · Create the credential

In your own Google Cloud project, create a service account, download its JSON key, and enable domain-wide delegation in your Workspace admin console for the read-only scope https://www.googleapis.com/auth/drive.readonly. (Without DWD, the service account itself must be granted access, e.g. added to shared drives.) No vendor OAuth app, ever.

2 · Configure

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
export VERITY_URL=http://127.0.0.1:7717     # the Drive runner otherwise defaults to :8080
export VERITY_TENANT_ID=$TENANT
export GDRIVE_DELEGATED_SUBJECT=admin@acme.example  # the workspace user to impersonate (DWD)
export GDRIVE_ANYONE_MAPS_TO=org:everyone     # optional; default: anyone-shared quarantines

3 · Run the initial backfill

The first --once bootstraps: with no cursor, poll fetches a start page token via changes.getStartPageToken and returns with no events — history before the token is the reconciliation crawl's job, not the change feed's. The crawl (files.list over trashed = false) is what pulls your existing corpus: per file it fetches metadata, then the ACL via permissions.list, then — only if the ACL resolves — content (Google Docs exported as text/plain; text/* and JSON downloaded directly). Preview the exact request bodies first with --dry-run:

python -m verity_ingest.connectors.gdrive --once --dry-run   # prints POST /v1/ingest/documents bodies
python -m verity_ingest.connectors.gdrive --once             # deliver for real; cursor -> .verity/gdrive_cursor.json
# gdrive: delivered 91 request(s); cursor -> .verity/gdrive_cursor.json

Each Drive permission maps to a canonical principal, resolved to an int token through POST /v1/admin/principals (the HttpRegistry) — or a local --principal-map JSON for offline runs:

4 · Verify

Mirrored Drive docs are recall-searchable, pre-filtered by the mirrored ACL. A caller whose resolved principals intersect the file's Drive permissions gets hits; a caller who is not on the file gets zero — the ACL became an in-index pre-filter (see Permissions → ACL inheritance).

# a scope for a principal on the file → the doc is recalled
curl -s -X POST "$VERITY_URL/v1/recall" -H 'content-type: application/json' \
  -d "{\"scope_handle\":\"$IN\",\"text\":\"onboarding runbook\",\"k\":5}" | jq length
# → 3
# a scope for a principal NOT on the file → 0 hits (mirrored ACL excluded it in the index)
curl -s -X POST "$VERITY_URL/v1/recall" -H 'content-type: application/json' \
  -d "{\"scope_handle\":\"$OUT\",\"text\":\"onboarding runbook\",\"k\":5}" | jq length
# → 0

5 · Ongoing sync

Run without --once to loop at --interval seconds (default 300); each cycle drives changes.list from the saved page-token cursor and re-emits changed files (permission drift shows up as re-emitted envelopes; the server diffs). The changes.watch push lane needs a public HTTPS endpoint and 7-day channel renewal and is a documented no-op today. For a supervised schedule, use Temporal below.

Postgres / MySQL

Two on-ramps for a relational database, both landing in the same deterministic keyed L1 path. Pick by whether you need history and ordering.

The pg_net trigger (CDC-lite, convenience)

One copy-paste trigger POSTs row changes to a minted webhook URL — no replication slot, no Kafka, no connector process. Works on Supabase (pg_net preinstalled) or any Postgres with pg_net installed. The full snippet is docs/snippets/pg-net-trigger.md; the shape:

verity-cli webhook mint pg:orders --visibility 1

create extension if not exists pg_net;
-- verity_row_to_facts() + AFTER INSERT OR UPDATE trigger — see the snippet file
perform net.http_post(
  url  := 'https://verity.example.com/wh/<token>',
  body := jsonb_build_object('facts', facts),
  headers := '{"Content-Type": "application/json"}'::jsonb);

verity-cli query "order 1042"

The Debezium envelope (the truth lane)

Already running CDC, or need ordering, snapshots (backfill), and deletes? Point a Debezium Server HTTP sink (or a small Kafka consumer) at the admin ingest endpoint. Debezium's initial snapshot is your backfill; it accepts a single envelope or a JSON array, upserts row facts through the same bi-temporal path, and handles op:"d" by retiring the entity's facts.

curl -s -X POST "$VERITY_URL/v1/ingest/debezium?tenant_id=$TENANT&pk=id" \
  -H 'content-type: application/json' \
  -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"op":"u","source":{"connector":"postgresql","table":"orders","ts_ms":1752000000000},
       "after":{"id":"1042","status":"paid","amount":99.00}}'
# → { "facts_inserted":2, "facts_superseded":0, "facts_unchanged":0, "facts_retired":0 }

Bulk documents

Loading an existing document corpus — a folder of markdown, a wiki export, PDFs already extracted to text — is the convenience lane: snapshot-grade, no per-object ACLs, so visibility is always an explicit admin decision (admin-assigned).

verity-cli add

add takes a file, a directory (recursive over .txt/.md/.json/.csv/.html, capped at 200 files), an http(s) URL, or stdin. --visibility is required by the parser and has no default — Verity never guesses who may see a memory. It mints a short-lived scope whose principals are exactly your tokens and uploads under it.

verity-cli add ./handbook --visibility 1                        # a whole directory, org-wide
verity-cli add onboarding.md --visibility 7,11 --entity account:acme  # narrow + entity tag

Framework sinks

If your corpus already loads through a framework, six thin adapters turn its readers into Verity sinks — LlamaIndex, LangChain, LangGraph, CrewAI, Google ADK, OpenAI Agents. Every one takes a required visibility_policy with no default (loaders strip source ACLs by construction, so this lane is always policy-based; missing policy quarantines). Point hundreds of community loaders at permission-aware memory:

from verity_llamaindex import VerityVectorStore

store = VerityVectorStore(
    verity_url="http://127.0.0.1:7717",
    tenant_id=TENANT,
    visibility_policy=[1],          # REQUIRED — no default; missing → quarantine
    admin_token=ADMIN_TOKEN,
)
# every LlamaHub reader now writes into permission-aware Verity memory

See Ingestion → framework sinks for all six adapters. For file uploads without a framework, POST /v1/files is a multipart upload under a scope handle (paragraph-chunked, embedded). Verify with a scoped verity-cli query.

Any internal system

A homegrown app, a cron job, an ops tool — anything that can POST JSON becomes a source in ~60 seconds, no connector code and no OAuth. The visibility policy is bound at mint time into the URL; the blast radius is one URL, instantly revocable.

curl -s -X POST "$VERITY_URL/v1/webhooks" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"tenant_id":"'"$TENANT"'","name":"ops-tool","visibility":[11],
       "entity_scope":["account:acme"]}'
# → { "webhook_id":"0192...", "url":"/wh/n8sQ...raw-token" }

curl -s -X POST "$VERITY_URL/wh/n8sQ...raw-token" -H 'content-type: application/json' \
  -d '{"content":"Acme signed the pilot agreement for the fall rollout.",
       "entities":["account:acme"]}'
# → 200 { "episode_id":"...", "chunks_indexed":1, "facts_written":0 }

Provenance is admin-assigned (the visibility was an explicit admin decision at mint). Unknown shapes return 202 {"quarantined":true} — preserved for review, never permissively indexed. Revoke instantly with DELETE /v1/webhooks/{id}. For a reviewable, structured mapping over the same endpoint (with tier-enforced ACL policy), graduate to a source manifest.

Ongoing sync (Temporal)

Once a backfill has landed, promote the connector from ad-hoc --once runs to a supervised schedule. The orchestration plane runs a Temporal worker plus one Schedule per connector; the cursor lives in workflow state instead of a .verity/*_cursor file, and delivery stays at-least-once (the next cursor is returned only after the sink succeeds).

# which connectors sync, and how often — fail closed: EMPTY means nothing syncs
export VERITY_CONNECTORS=hubspot,salesforce,gdrive
export VERITY_SYNC_INTERVAL=300              # default poll seconds
export VERITY_SYNC_INTERVAL_HUBSPOT=60       # per-connector override
# the orchestration runners read each connector's OWN policy env, no default:
export HUBSPOT_VISIBILITY=1,2
export SALESFORCE_VISIBILITY=1,2

# 1) run the worker (scale horizontally by running more on the same task queue)
python -m verity_ingest.orchestration.worker

# 2) apply the schedules (dry-run without --apply; overlap policy SKIP)
python -m verity_ingest.orchestration.schedules --apply
# verity-sync-hubspot: created   verity-sync-salesforce: created   verity-sync-gdrive: created

System comparison

The one-glance table: credential, the initial-backfill command, freshness, and the ACL contract for each system.

Status & honesty

Verity is pre-1.0 and every claim on this site is measured, never quoted. Where a connector is fixture-verified vs live-validated:

• Fixture-verified. The HubSpot, Salesforce, and Google Drive connectors' deterministic mappings (search/query pages → facts, permissions.list → ACL envelope, webhook decode) run against recorded fixtures in CI. The request bodies for /v1/ingest/documents and /v1/admin/principals are pinned by fixture tests; full live-server integration lands as those endpoints finish.
• Awaiting a design partner. Salesforce live-org validation — and the CDC Pub/Sub (gRPC) push lane — is not yet exercised against a real org. The share-derived principals are approximated additive metadata, not enforced visibility, until the identity crosswalk and the implicit-sharing gap are closed. We say so rather than imply live coverage.
• Forthcoming. A backfill-progress dashboard is not yet built; today the connectors emit best-effort heartbeats to POST /v1/admin/connector-status (source, items synced, newest event time, cursor) — readable at GET /v1/admin/connector-status — and a heartbeat failure never fails a sync.