Skip to content

Gateway / Proxy

AI Gateway

OpenAI-compatible HTTP proxy with inline firewall, observability, and FinOps. No SDK changes — change one base URL.

Speaks the OpenAI Chat Completions wire format. EvalGuard supports 77 LLM providers (91 total integrations) across the platform; the gateway proxies to 15 upstream providers (14 OpenAI-compatible + Anthropic via a dedicated bridge). Works as a drop-in for the OpenAI SDK in every language we publish: TypeScript, Python, Go — and from any language that can speak HTTP.

Architecture

The gateway is inline in the data path. Your apps, agents, and MCP tools call one OpenAI-compatible endpoint; every request runs through the control plane — firewall, eval, routing, observability — before it reaches the upstream model, and the response is re-inspected on the way back.

your stack

Your appSDK · REST
AI agentstool-calling
MCP toolsservers
EvalGuardcontrol plane live
Firewall
Eval
Gateway
Observe
every call inspected · 2.57ms p95

any provider

OpenAIOpenAIGPT-4o · o-series
AnthropicAnthropicClaude family
GeminiGeminiGoogle
MistralMistralopen models
+90+ providersone API
request in inspected & verified out

The 91-provider chip counts platform-wide SDK integrations; the gateway itself proxies to 15 upstream providers (see above).

What you get

Inline firewall

PII, prompt-injection, jailbreak, and 235 DLP patterns blocked before the request leaves your network — see /trust/latency for p50/p95/p99.

Trace + cost

Every request lands in /dashboard/traces with token counts, latency, model, end-user, and a per-call cost figure pulled from your model_registry.

Smart routing + cache

Semantic cache (cosine similarity), circuit breaker, fallback chains across all 91 provider integrations, and per-API-key budget enforcement.

Three install paths

The gateway IS our hosted SaaS — there's no standalone binary because cloud users don't install anything (option 1). Self-host users get the same image we run in prod via Docker (option 2) or Helm (option 3).

1. Cloud (zero install — recommended)

Point your existing SDK base URL at evalguard.ai. The SDK doesn't change — every OpenAI / Anthropic / Mistral / Groq / Together / Fireworks request is firewalled, traced, cost-tracked, and replayable.

OpenAI SDK — Python
import openai
client = openai.OpenAI(
    base_url="https://evalguard.ai/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)
OpenAI SDK — TypeScript
const client = new OpenAI({
  baseURL: "https://evalguard.ai/api/v1/gateway/proxy",
  defaultHeaders: { "X-EvalGuard-Key": "eg_live_..." },
});
Anthropic SDK — same pattern
client = anthropic.Anthropic(
    base_url="https://evalguard.ai/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

2. Self-host with Docker (one command)

For on-prem / regulated environments. Pulls the same image we run in production. The gateway lives at /api/v1/gateway/proxy on the published container.

terminal
# One command — pulls the latest image, exposes the gateway on :3000
docker run -d \
  --name evalguard-gateway \
  -p 3000:3000 \
  -e DATABASE_URL="postgres://..." \
  -e REDIS_URL="redis://..." \
  -e EVALGUARD_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
  ghcr.io/evalguardai/evalguard:latest
Then point your SDK at it
client = openai.OpenAI(
    base_url="http://localhost:3000/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

3. Helm chart — Kubernetes (production self-host)

Same image, deployed via the bundled Helm chart at /helm/evalguard. Includes worker + Redis + horizontal pod autoscaler; Postgres/Supabase is an external dependency you supply via env.NEXT_PUBLIC_SUPABASE_URL and secrets.supabaseServiceKey.

terminal
# Install with custom values. Use --strict so an unknown
# --set key errors instead of being silently dropped.
helm install evalguard ./helm/evalguard \
  --set image.tag=latest \
  --set ingress.enabled=true \
  --set 'ingress.hosts[0].host=gateway.your-corp.com' \
  --set 'ingress.hosts[0].paths[0].path=/' \
  --set 'ingress.hosts[0].paths[0].pathType=Prefix' \
  --set secrets.encryptionKey=$(openssl rand -base64 32) \
  --set env.REDIS_URL=$REDIS_URL \
  --set env.NEXT_PUBLIC_SUPABASE_URL=$SUPABASE_URL \
  --set secrets.supabaseServiceKey=$SUPABASE_SERVICE_KEY

Why no npx install?

The gateway is a stateful service — it needs Postgres + Redis to track traces, costs, budgets, and firewall rules. A short-lived npx binary would lose every trace at process exit, defeating the purpose. For local dev without a cloud account, run the Docker image (option 2) or use the cloud sandbox at evalguard.ai/signup with the included free tier.