cli (7)
# vauth — Service Overview
Service registry metadata: owner `identity`; repo `verkada/Verkada-Backend`. Code-level evidence confirms vauth changes land in this repo (e.g., `fix(vauth)` PR #199618) [4].
## Summary
vauth is the authentication service of the Verkada Backend. Its tokens are the currency of service-to-service and device authentication: consumers such as vautomations' `withDeviceAuth` middleware accept "any valid vauth token with a UUID-shaped entity ID" [9]. vauth also stores identity-related state, notably `ACCESS_ORG_MEMBER` role grants, which carry a `hidden` flag in vauth [6], and it hosts HTTP endpoints, including internal GET endpoints [3] and a token-creation endpoint for the Pass App SDK [12].
## Responsibilities
- **Token validation surface for consumers.** Other services authenticate callers against vauth tokens. Before a fix, vautomations' `withDeviceAuth` authenticated the device JWT but did not verify the caller was an access controller; the remediation adds `VerifyACU` so that a valid vauth token for a non-ACU entity returns 401 from the device rconfig endpoint [9].
- **Storage of role grants.** vauth holds `ACCESS_ORG_MEMBER` role grants with a `hidden` attribute; grants marked `hidden = true` are silently omitted from generated access controller (aconfig) output, which caused silent access denial affecting ~2,998 users across 50 orgs on prod1 [6].
- **Hosting endpoints.** vauth serves internal GET endpoints (`lockdown_internal_get`, `area_internal_get`) [3] and an unpublished public endpoint to create Pass App SDK-scoped tokens [12], enabled by a new public-SDK scope shipped under a feature flag [15].
- **Error reporting.** vauth 401 responses carry a parseable failure reason; PR #199618 in Verkada-Backend fixed behavior that discarded this reason [4].
## Architecture
- **Codebase.** vauth code lives in `verkada/Verkada-Backend`; fixes are merged to master (PR #165882, CI passed) [3] [4].
- **Caching layer.** vauth runs memcached clusters in prod1 ("v2 clusters"); Terraform sets `memcached_connections_overhead` to 500 on them (COR-24162) [7].
- **Infrastructure as code.** Alarm configuration for `services/vauth` environments is managed in the `verkada/Terraform` repo, including OpsGenie SNS ARNs in alarm action lists [1].
- **Scope of evidence.** The evidence does not describe vauth's overall deployment topology, token issuance mechanism, or data stores beyond the memcached tuning noted above; treat those as unknown.
## Interfaces
- **`lockdown_internal_get` and `area_internal_get`** — internal GET endpoints in vauth (fix locations: lines 530 and 556 respectively) [3]. Known caller: vlive, fetching Scenarios action list filters [3].
- **Pass App SDK-scoped token creation** — an "unpublished" public endpoint in vauth, called by the 3rd-party SDK backend, via `POST` with body `{user_id: <TDB>}` [12]. This required a new public-SDK scope on the backend, gated by a feature flag, to create a Pass App token (CSRF) in vauth [15].
- **Tokens consumed by other services** — vautomations validates device JWTs via `withDeviceAuth`, where valid vauth tokens were (and remain, post-`VerifyACU` for ACUs) the accepted credential for the device rconfig endpoint [9]. The device-facing rconfig download endpoint (`GET /access/v2/device/automations/device/config/1`) lives in vautomations, not vauth [18].
- **401 failure reason** — surfaced to callers rather than discarded, per PR #199618 [4].
## Operations
- **Alarms.** Infra Advisor dropped an empty OpsGenie SNS ARN from the alarm action lists of `services/vauth(environment)` in Terraform [1].
- **Capacity tuning.** `memcached_connections_overhead` was set to 500 on prod1 v2 memcached clusters (COR-24162) [7].
- **Incident history.** ACBE-6926: bare `site_ids[0]` accesses in `lockdown_internal_get` and `area_internal_get` crashed with `IndexError` when a lockdown or area had no parent site, logging "Error processing message: list index out of range". Prod1 impact before the fix: 50+ errors per 24h across 12 unique lockdown IDs. Fixed in PR #165882 (guards added, unit tests added, merged to master) [3].
- **Known data-quality issue.** Hidden `ACCESS_ORG_MEMBER` grants in vauth are excluded from aconfig generation, silently denying access (~2,998 users / 50 orgs on prod1); the analysis is pinned to commit `1d036758bb3365ac4780042030f6bdda53936cbd` on origin/master [6]. The evidence documents the problem but not a fix.
## Open questions
- Does `vcerberus` call the Pass App SDK-scoped token endpoint? This is an explicit "Questions / TBD" item in the source issue [12].
- Has the hidden-`ACCESS_ORG_MEMBER` aconfig exclusion been fixed? The evidence records the investigation and affected scale but no remediation [6].
- What is vauth's token issuance mechanism and protocol? The evidence shows consumers validating vauth tokens [9] but says nothing about how tokens are minted, signed, or refreshed.
- Is vauth the "platform authentication adapter that validates the agreed proxy/token assertion" referenced in the gaia reference documentation? That description is not explicitly tied to vauth [2], so the relationship is unknown.
- What is the full public/internal endpoint surface of vauth beyond the endpoints named above [3] [12], and what are its non-prod1 operational characteristics (clusters, regions, alarm coverage)? The evidence only covers prod1 memcached tuning [7] and Terraform-managed alarm lists [1]; everything else is unknown.vdevai (6)
# vauth — Service Overview
Owner: identity · Repo: verkada/Verkada-Backend
## Summary
vauth is an identity-owned service in the `verkada/Verkada-Backend` monorepo that runs as a Kubernetes deployment and recently gained a dedicated `vauth-authz` container serving authorization routes via gunicorn [1]. It operates in at least `prod-ap-syd` and `prod2`, where a companion `vaccel-vauth` deployment is scaled to match vauth's replica counts [2]. Recent engineering activity in the evidence covers authorization/permissions handling [1][5], pod lifecycle tuning [3], mitigation of worker memory growth [5], and a planned internal reconciliation endpoint [11].
## Responsibilities
- **Authorization serving:** a `vauth-authz` container was added to vauth, backed by `vauth/vauth/authz_routes.py` and a dedicated gunicorn entrypoint (`gunicorn_authz.py`) [1].
- **Permissions evaluation:** `vauth/vauth/lib/permissions_manager.py` contains org-entity permission queries (e.g., `_get_org_entities_query` around L2770), which append path-match expressions over `DirectoryPath.path` [5].
- **Incident download authorization:** a completed fix (CVS-8137, team CV & Search) addressed an "incident download loophole for archive" under the "Incident Management support for Access Control" project [21]. The evidence does not specify which enforcement path was involved.
- **Endpoint clustering:** an internal Notion page titled "vAuth Endpoint Clusters" exists (tagged `v3_api`); only the title and outline are visible in the evidence, so the exact clustering scheme is unknown [4].
## Architecture
- **Kubernetes deployment:** vauth runs as a deployment configured through `vauth/k8s/` manifests (e.g., `deployments.k`, staging base/overlay YAML) and `vauth/vaccel-configs/staging/...` configs [1][3].
- **Containers:** the main vauth container plus the `vauth-authz` container, the latter served by gunicorn [1][5].
- **Scale and topology:** in `prod-ap-syd`, vauth runs at 15/60 replicas; in `prod2`, 3/60; the `vaccel-vauth` deployment's replica counts were raised to match vauth in both regions [2]. The terminationGracePeriod issue notes that replica "triples are perZone splits," indicating per-zone replica layout [3].
- **Lifecycle:** pod `terminationGracePeriodSeconds` was raised from 60 to 90 on the vauth deployment [3].
- **Relationship to vaccel-vauth:** the evidence only shows replica-count matching [2]; the functional role of vaccel-vauth is not described in the evidence.
## Interfaces
- **Authorization routes:** implemented in `vauth/vauth/authz_routes.py` and served by the `vauth-authz` container [1]. The specific route paths are not included in the evidence.
- **Internal reconciliation endpoint (planned):** COR-24072 proposes an internal `supplements:reconcile` endpoint, blocked by COR-24068 ("add supplementrolemanagerreconcile") [11]. Its completion status is not in the evidence.
- **Endpoint cluster catalog:** the "vAuth Endpoint Clusters" Notion page [4]; its contents (which endpoints belong to which cluster, and how clusters are used) are unknown from the evidence.
- No public API surface for vauth is documented in the provided evidence.
## Operations
- **Graceful shutdown:** `terminationGracePeriodSeconds` on the vauth deployment was increased from 60 to 90 (COR-23897) [3].
- **Memory growth mitigation:** vauth-authz gunicorn workers exhibited memory growth in `prod2`; the mitigation lowered gunicorn `max-requests` for the `vauth-authz` container in `prod2` to 3000/300, mirroring the earlier COR-24107 change (prod-ap) [5]. The evidence points to the org-entities permission query appending regex-style path operations as a visible code path in the issue [5]. Whether this mitigation is permanent is not stated.
- **Replica scaling:** `vaccel-vauth` was raised from 1/1 to 15/60 in `prod-ap-syd` and 1/1 to 3/60 in `prod2`, explicitly to match vauth (COR-23722) [2].
## Open questions
- What are vauth's full responsibilities beyond the authz container and the permissions-manager code? The evidence only samples these areas [1][5].
- What is `vaccel-vauth`, and why must its replica counts track vauth's? Only the scaling change is documented [2].
- Which HTTP routes does `authz_routes.py` expose, and which callers consume them? [1]
- Did the internal `supplements:reconcile` endpoint (COR-24072) ship after its dependency COR-24068, and what does SupplementRoleManager reconcile cover? [11]
- What do the "vAuth Endpoint Clusters" contain, and are clusters used for routing, ownership, or capacity decisions? [4]
- What was the root cause of the vauth-authz worker memory growth in `prod2`, and is the `max-requests` reduction a durable fix or a stopgap? [5]
- Are there vauth deployments in environments other than `prod-ap-syd` and `prod2`? The evidence only mentions these two [2][3].
- How does vauth's role in the Access Control incident download fix (CVS-8137) relate to its general authorization routes? The issue title alone does not say [21].