BYOK — bring your own key

Why key custody matters

With BYOK, calls to OpenAI, Anthropic, Groq, and the rest bill your provider account under your rate limits and data-processing agreements — EvalGuard never resells inference or marks up tokens. Just as important, we minimize how much of the secret we can ever see: on write the plaintext is pushed into an encrypted vault and our own database only keeps a pointer plus the last four characters. No API response, log line, Postgres page, or backup snapshot carries the plaintext back out.

Configure a key — CLI

The evalguard keys commands are the safest path: the plaintext is read from stdin or a file, so the key never lands in your shell history. Only metadata comes back.

# Add or rotate a key. Plaintext is read from stdin (Ctrl-D to end)…
echo "$OPENAI_API_KEY" | evalguard keys add openai \
  --org "$EVALGUARD_ORG_ID" --label "prod-openai"

# …or from a file, never touching the shell history:
evalguard keys add anthropic --key-file ./anthropic.key --org "$EVALGUARD_ORG_ID"

# Scope a key to one project (wins over the org-wide default at call time):
evalguard keys add groq --project <projectId> --key-file ./groq.key

# List keys in the org — metadata only, plaintext stays in the vault:
evalguard keys list --org "$EVALGUARD_ORG_ID"
#   Provider   Label        Last-4  Scope        Rotated
#   openai     prod-openai  …AB12   org-default  —

# Revoke (destructive — --dry-run previews the target first):
evalguard keys remove <key-id> --org "$EVALGUARD_ORG_ID" --dry-run
evalguard keys remove <key-id> --org "$EVALGUARD_ORG_ID" --yes

Adding a key for a provider that already has one rotates it in place — the vault row is rewrapped under a fresh data-encryption key and the same id is kept.

Configure a key — HTTP API

The CLI is a thin wrapper over three routes. Writes require an owner or admin role — a read-only API key cannot create, rotate, or revoke provider keys.

# Create or rotate (201 on create, 200 on rotate)
POST /api/v1/provider-keys
Authorization: Bearer <evalguard-api-key>
{
  "orgId":     "<uuid>",
  "provider":  "openai",        // canonical id; aliases resolve (claude→anthropic, kimi→moonshot)
  "apiKey":    "sk-...",        // plaintext — goes straight to the vault
  "projectId": null,            // null = org-wide default; a uuid = per-project override
  "label":     "prod-openai"
}

# List — metadata only, NEVER the plaintext or the vault pointer
GET /api/v1/provider-keys?orgId=<uuid>[&projectId=<uuid>]
→ { "keys": [{ "id", "provider", "project_id", "label",
               "key_last4", "created_at", "rotated_at" }], "total": 1 }

# Revoke — the DB cascade also drops the underlying vault secret
DELETE /api/v1/provider-keys?id=<uuid>&orgId=<uuid>

The accepted provider list is derived live from the core provider registry (listProviders()), so it stays in lockstep with what the gateway can actually call — no hand-maintained allowlist to drift. An unknown id returns a 400 UNKNOWN_PROVIDER with the supported set echoed back.

How it's encrypted

Writes go to Supabase Vault (pg_sodium): each secret gets a per-row, libsodium-wrapped data-encryption key, and the key-encryption key lives in Supabase's managed KMS. Our tables store only the vault_secret_id pointer and key_last4. Reads bridge back through a SECURITY DEFINER RPC (vault_read_secret), because the vaultschema isn't exposed over the data API directly.

writeProviderSecret(plaintext) ──► vault_create_secret()  (or vault_update_secret on rotate)
                                    │  per-row DEK, wrapped by managed KMS KEK
                                    ▼
provider_keys row: { vault_secret_id, key_last4, provider, project_id, org_id }
                                    │
readProviderSecret(row) ──► vault_read_secret(vault_secret_id) ──► plaintext (server-side only)

A legacy AES-256-GCM path still decrypts rows created before the vault migration: ciphertext packed as salt || iv || ct+authTag, key derived with PBKDF2-SHA-256 at 310,000 iterations from the BYOK_SECRETenv var (min 32 chars). Newer "v2" blobs bind the org+provider as GCM additional-authenticated-data so a ciphertext can't be swapped across tenants, and BYOK_SECRET_PREVIOUS lets you rotate the secret with zero downtime. All of this is hidden behind readProviderSecret(), so business code never sees which path a given row uses.

How the gateway & evals use it

Every consumer resolves a key through one function, resolveProviderSecret(orgId, provider, projectId), which picks the project-specific row over the org default, decrypts it, and memoizes the result for ~30s per tenant so bursts don't hammer the vault. A rotation via the write path invalidates that cache immediately.

resolveProviderSecret(org, "openai", projectId)
  1. project-scoped provider_keys row (if projectId given and present)
  2. org-default row (project_id IS NULL)
  3. → decrypt via vault (or legacy AES-GCM)  → plaintext
  4. null  → caller falls back to a server env-var provider key, if any

• Evals & generation — AI eval-suite generation, chat completions, RAG auto-eval, and adaptive red teaming resolve the stored key first, then fall back to a server env key. AI-Generate is BYOK-only: with no key it returns 422 and points you to Settings → Provider Keys (or the free Template Library).
• Gateway proxy— on the primary hop the caller's own Authorization header is forwarded as-is. The stored BYOK key comes into play for fallbackproviders: when the primary fails over, the gateway resolves that org's key for the fallback provider (a fallback with no configured key is simply skipped rather than sent an unauthenticated request).

Honest edges

• Multi-credential providers (AWS Bedrock SigV4, Google Vertex service accounts, Azure OpenAI, SageMaker) need a structured credential the single-input vault can't hold as one string. The signed-bridge read path (resolveSignedBridgeCreds) and request signing exist and are fail-closed, but there is no write/UI seam to populate those creds yet — so this path is inert behind a default-off flag until that lands.
• Keyless local runtimes (Ollama, vLLM, LM Studio, and other local endpoints) accept a vault row that records the endpoint URL, but core's local-runtime factories currently start from their built-in default endpoints; threading the stored URL through as the base URL is a follow-up.
• Access control — provider_keys is protected by row-level security: org members can read key metadata, but only owners and admins can create, rotate, or revoke. The API handlers assert that role explicitly so a scoped read-only key can't bypass RLS via the admin client.