Gateway / Proxy

AI Gateway

OpenAI-compatible HTTP proxy with inline firewall, observability, and FinOps. No SDK changes — change one base URL.

Speaks the OpenAI Chat Completions wire format. EvalGuard supports 77 LLM providers (91 total integrations) across the platform; the gateway proxies to 15 upstream providers (14 OpenAI-compatible + Anthropic via a dedicated bridge). Works as a drop-in for the OpenAI SDK in every language we publish: TypeScript, Python, Go — and from any language that can speak HTTP.

Architecture

The gateway is inline in the data path. Your apps, agents, and MCP tools call one OpenAI-compatible endpoint; every request runs through the control plane — firewall, eval, routing, observability — before it reaches the upstream model, and the response is re-inspected on the way back.

your stack

Your appSDK · REST

AI agentstool-calling

MCP toolsservers

EvalGuardcontrol plane live

Firewall

Eval

Gateway

Observe

every call inspected · 3.67ms p95

any provider

OpenAIGPT-4o · o-series

AnthropicClaude family

GeminiGoogle

Mistralopen models

+90+ providersone API

request in inspected & verified out

The 91-provider chip counts platform-wide SDK integrations; the gateway itself proxies to 15 upstream providers (see above).

What you get

Inline firewall

PII, prompt-injection, jailbreak, and 439 DLP patterns blocked before the request leaves your network — see /trust/latency for p50/p95/p99.

Trace + cost

Every request lands in /dashboard/traces with token counts, latency, model, end-user, and a per-call cost figure pulled from your model_registry.

Smart routing + cache

Similarity response cache (cosine over a local lexical vector), circuit breaker, fallback chains across the 15 gateway-proxied providers, and per-API-key budget enforcement.

Three install paths

The gateway IS our hosted SaaS — there's no standalone binary because cloud users don't install anything (option 1). Self-host users run the same image we run in prod via Docker (option 2) or Helm (option 3); that image is private, so both need the licence grant and GHCR pull token described in self-hosting.

1. Cloud (zero install — recommended)

Point your existing SDK base URL at evalguard.ai. The SDK doesn't change — every OpenAI / Anthropic / Mistral / Groq / Together / Fireworks request is firewalled, traced, cost-tracked, and replayable.

OpenAI SDK — Python

import openai
client = openai.OpenAI(
    base_url="https://evalguard.ai/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

OpenAI SDK — TypeScript

const client = new OpenAI({
  baseURL: "https://evalguard.ai/api/v1/gateway/proxy",
  defaultHeaders: { "X-EvalGuard-Key": "eg_live_..." },
});

Anthropic SDK — same pattern

client = anthropic.Anthropic(
    base_url="https://evalguard.ai/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

2. Self-host with Docker (licensed image)

For on-prem / regulated environments. Runs the same image we run in production. The image is private: self-hosting requires a licence grant, and the pull token is issued with it — email sales@evalguard.ai. Anonymous pulls of ghcr.io/evalguardai/evalguard return 401. Once you are logged in, the gateway lives at /api/v1/gateway/proxy on the container.

terminal

# 1. Authenticate. The image is private — an anonymous pull is a 401.
#    The pull token ships with the self-host licence grant, alongside
#    monorepo read access (sales@evalguard.ai). See /docs/self-hosting.
echo "$GHCR_PULL_TOKEN" | docker login ghcr.io -u "$GHCR_USERNAME" --password-stdin

# 2. Run it — exposes the gateway on :3000
docker run -d \
  --name evalguard-gateway \
  -p 3000:3000 \
  -e DATABASE_URL="postgres://..." \
  -e REDIS_URL="redis://..." \
  -e EVALGUARD_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
  ghcr.io/evalguardai/evalguard:latest

Then point your SDK at it

client = openai.OpenAI(
    base_url="http://localhost:3000/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

3. Helm chart — Kubernetes (production self-host)

Same image, deployed via the bundled Helm chart at /helm/evalguard. Includes worker + Redis + horizontal pod autoscaler; Postgres/Supabase is an external dependency you supply via env.NEXT_PUBLIC_SUPABASE_URL and secrets.supabaseServiceKey.

terminal

# Install with custom values. Use --strict so an unknown
# --set key errors instead of being silently dropped.
helm install evalguard ./helm/evalguard \
  --set image.tag=latest \
  --set ingress.enabled=true \
  --set 'ingress.hosts[0].host=gateway.your-corp.com' \
  --set 'ingress.hosts[0].paths[0].path=/' \
  --set 'ingress.hosts[0].paths[0].pathType=Prefix' \
  --set secrets.encryptionKey=$(openssl rand -base64 32) \
  --set env.REDIS_URL=$REDIS_URL \
  --set env.NEXT_PUBLIC_SUPABASE_URL=$SUPABASE_URL \
  --set secrets.supabaseServiceKey=$SUPABASE_SERVICE_KEY

Why no `npx` install?

The gateway is a stateful service — it needs Postgres + Redis to track traces, costs, budgets, and firewall rules. A short-lived npx binary would lose every trace at process exit, defeating the purpose. For local dev without a cloud account, run the licensed Docker image (option 2, grant required) or use the cloud sandbox at evalguard.ai/signup with the included free tier.

Agent guardrails

Two inline, dependency-free guardrails aimed at agentictraffic — failure modes the content/PII guardrails don't cover. Both are fail-closed and default-inert: they do nothing until you enable them for a project with a gateway_guardrail_config row. With no row, the proxy runs its inline firewall only — zero behavior change. Both are local presets: they make no external call, so — unlike partner adapters — they carry no secretRef.

Data-Not-Instructions

data-not-instructions

Request side

Enforces the “data is not instructions” boundary: flags or blocks untrusted content (a retrieved doc, a pasted block, a prior tool result) that carries embedded tool-call syntax (<tool_call>, <function_calls>, <invoke>, or a fenced block declaring a "tool" key) or a classic override directive (“ignore previous…”, “disregard the above”, “you are now…”, a fake SYSTEM: role marker, “new instructions:”) — the payload trying to act as instructions rather than inert data.

Tool-Call Circuit Breaker

tool-call-circuit-breaker

Response side

Trips when the SAME tool call (name + canonical, key-sorted args) repeats with no progress within a session — a looping / runaway agent. Mutating calls trip sooner than read-only ones (with the default budget of 3, read-only trips after 3 repeats and mutating after 2). The read-only budget is tunable via config.maxRepeats.

On a detection, the on_flag verdict decides what happens: block aborts the request; flag (and redact, which has no safe auto-rewrite for this class) annotates the traffic with x-guardrail-flagged headers and lets it through. The API/CLI default when creating a row is block. Separately, both controls are tagged fail-closed (failMode: "closed"): if the guardrail itself crashes or times out, the request is blocked, never passed unscanned.

Enable them

Enabling is admin-only and org-scoped. The management endpoint upserts on (project_id, vendor), so re-running updates in place instead of stacking duplicates. The change takes effect on the very next gateway request (the loader cache is busted on write).

API — POST /api/v1/gateway/guardrails

Enable data-not-instructions (request-side, block) — no secretRef

curl -X POST https://evalguard.ai/api/v1/gateway/guardrails \
  -H "Authorization: Bearer eg_live_..." \
  -H "content-type: application/json" \
  -d '{
    "orgId": "00000000-0000-0000-0000-000000000000",
    "projectId": "11111111-1111-1111-1111-111111111111",
    "vendor": "data-not-instructions",
    "onFlag": "block",
    "checkRequest": true,
    "enabled": true,
    "priority": 12
  }'

Enable tool-call-circuit-breaker (response-side, block) — maxRepeats tunes the read-only budget

curl -X POST https://evalguard.ai/api/v1/gateway/guardrails \
  -H "Authorization: Bearer eg_live_..." \
  -H "content-type: application/json" \
  -d '{
    "orgId": "00000000-0000-0000-0000-000000000000",
    "projectId": "11111111-1111-1111-1111-111111111111",
    "vendor": "tool-call-circuit-breaker",
    "onFlag": "block",
    "enabled": true,
    "priority": 12,
    "config": { "maxRepeats": 3 }
  }'

Sending a secretRef for either local preset is rejected with 400 SECRET_REF_NOT_ALLOWED. The circuit breaker declares its own response-side hook, so checkResponse is not needed for it; with maxRepeats: 3, read-only tool calls trip after 3 repeats and mutating calls after 2.

CLI — evalguard guardrail set

terminal — idempotent on (project, vendor)

# Data-is-not-instructions boundary on the request side, blocking.
evalguard guardrail set \
  --org 00000000-0000-0000-0000-000000000000 \
  --project 11111111-1111-1111-1111-111111111111 \
  --vendor data-not-instructions \
  --on-flag block --check-request --enabled

# Tool-call loop breaker on the response side, blocking, budget 3.
evalguard guardrail set \
  --org 00000000-0000-0000-0000-000000000000 \
  --project 11111111-1111-1111-1111-111111111111 \
  --vendor tool-call-circuit-breaker \
  --on-flag block --enabled --config '{"maxRepeats":3}'

# List / remove.
evalguard guardrail list --project 11111111-1111-1111-1111-111111111111
evalguard guardrail rm   --project 11111111-1111-1111-1111-111111111111 --vendor data-not-instructions

The CLI enforces the same local-vs-vendor rule client-side: passing --secret-ref with a local guardrail fails fast before any round-trip.

SDK — TypeScript (fetch)

enable-guardrails.ts

const res = await fetch("https://evalguard.ai/api/v1/gateway/guardrails", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.EVALGUARD_API_KEY}`,
    "content-type": "application/json",
  },
  body: JSON.stringify({
    orgId: "00000000-0000-0000-0000-000000000000",
    projectId: "11111111-1111-1111-1111-111111111111",
    vendor: "data-not-instructions", // or "tool-call-circuit-breaker"
    onFlag: "block",                  // block | redact | flag
    enabled: true,
    priority: 12,
    // No secretRef — these are local, dependency-free presets.
  }),
});
const { data } = await res.json();

Troubleshooting

The proxy returns OpenAI-style error envelopes — { "error": { "message", "type" } }— so your existing SDK's error handling surfaces them unchanged. The most common failures and their fixes:

`401` authentication_error — key missing, malformed, revoked, or expired

The gateway authenticates via the X-EvalGuard-Key header, and EvalGuard keys start with eg_. A missing header, a key that doesn't start with eg_, an unknown key, or one that has been revoked / has expired all return 401. Set the key in default_headers / defaultHeaders, not the Authorization header — the latter is forwarded to the upstream provider as-is.

401 responses (message varies by cause)

{ "error": { "message": "Authentication required. Provide an X-EvalGuard-Key header.", "type": "authentication_error" } }
{ "error": { "message": "Invalid API key format. Keys must start with 'eg_'",       "type": "authentication_error" } }
{ "error": { "message": "Invalid API key",                                          "type": "authentication_error" } }
{ "error": { "message": "API key has been revoked",                                 "type": "authentication_error" } }
{ "error": { "message": "API key has expired",                                      "type": "authentication_error" } }

`400` invalid_request— “Cannot determine target provider”

The gateway routes by the model name in the request body (e.g. gpt-4o resolves to OpenAI, a claude- prefix to Anthropic). If the model prefix isn't recognized, set an explicit x-evalguard-provider header, or a custom endpoint via x-evalguard-base-url (which must be HTTPS and resolve to an allow-listed public host).

400 response

{ "error": { "message": "Cannot determine target provider", "type": "invalid_request" } }

`429` rate_limit_error — throttled

Unauthenticated try-before-you-buy traffic (no X-EvalGuard-Key) is capped at 5 requests/minute and 200/day per IP — authenticate for higher limits. Per-key requests-per-minute, tokens-per-minute, and max-parallel caps also return 429, with a retry-after header (seconds) to back off on. A 402 instead means a per-key or per-team budget cap was hit — see Budgets.

429 responses

{ "error": { "message": "Rate limit exceeded. Use an EvalGuard API key for higher limits.", "type": "rate_limit_error" } }
{ "error": { "message": "API key rate limit exceeded",                                       "type": "rate_limit_error" } }

Self-host: connection & configuration errors

On the Docker / Helm path, a 401 with "Server configuration error" means the container can't reach Supabase — NEXT_PUBLIC_SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY must both be set so keys can be verified. The image also needs DATABASE_URL, REDIS_URL (traces / budgets / rate limits), and EVALGUARD_ENCRYPTION_KEY. Point your SDK at the proxy path on the published port, not the container root.

401 — missing Supabase config

{ "error": { "message": "Server configuration error", "type": "authentication_error" } }

The gateway lives at http://localhost:3000/api/v1/gateway/proxy on the Docker container (option 2). A 404 usually means the base URL is missing the /api/v1/gateway/proxy suffix.