Back to Docs

AI Gateway

OpenAI-compatible HTTP proxy with inline firewall, observability, and FinOps. No SDK changes — change one base URL.

Speaks the OpenAI Chat Completions wire format and proxies to 77 LLM providers. Works as a drop-in for the OpenAI SDK in every language we publish: TypeScript, Python, Go — and from any language that can speak HTTP.

What you get

Inline firewall

PII, prompt-injection, jailbreak, and 103 DLP patterns blocked before the request leaves your network — see /trust/latency for p50/p95/p99.

Trace + cost

Every request lands in /dashboard/traces with token counts, latency, model, end-user, and a per-call cost figure pulled from your model_registry.

Smart routing + cache

Semantic cache (cosine similarity), circuit breaker, fallback chains across 91 providers, and per-API-key budget enforcement.

Three install paths

The gateway IS our hosted SaaS — there's no standalone binary because cloud users don't install anything (option 1). Self-host users get the same image we run in prod via Docker (option 2) or Helm (option 3).

1. Cloud (zero install — recommended)

Point your existing SDK base URL at evalguard.ai. The SDK doesn't change — every OpenAI / Anthropic / Mistral / Groq / Together / Fireworks request is firewalled, traced, cost-tracked, and replayable.

# Python — OpenAI SDK
import openai
client = openai.OpenAI(
    base_url="https://evalguard.ai/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

# TypeScript / JavaScript — OpenAI SDK
const client = new OpenAI({
  baseURL: "https://evalguard.ai/api/v1/gateway/proxy",
  defaultHeaders: { "X-EvalGuard-Key": "eg_live_..." },
});

# Anthropic SDK — same pattern
client = anthropic.Anthropic(
    base_url="https://evalguard.ai/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

2. Self-host with Docker (one command)

For on-prem / regulated environments. Pulls the same image we run in production. The gateway lives at /api/v1/gateway/proxy on the published container.

# One command — pulls the latest image, exposes the gateway on :3000
docker run -d \
  --name evalguard-gateway \
  -p 3000:3000 \
  -e EVALGUARD_DB_URL="postgres://..." \
  -e EVALGUARD_REDIS_URL="redis://..." \
  -e EVALGUARD_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
  ghcr.io/evalguardai/evalguard:latest

# Then point your SDK at it:
client = openai.OpenAI(
    base_url="http://localhost:3000/api/v1/gateway/proxy",
    default_headers={"X-EvalGuard-Key": "eg_live_..."},
)

3. Helm chart — Kubernetes (production self-host)

Same image, deployed via the bundled Helm chart at /helm/evalguard. Includes worker + Redis + Postgres dependencies + horizontal pod autoscaler.

# Add the chart, install with custom values
helm install evalguard ./helm/evalguard \
  --set image.tag=latest \
  --set ingress.host=gateway.your-corp.com \
  --set encryption.key=$(openssl rand -base64 32) \
  --set postgres.connectionString=$DB_URL \
  --set redis.connectionString=$REDIS_URL

Why no npx install?

The gateway is a stateful service — it needs Postgres + Redis to track traces, costs, budgets, and firewall rules. A short-lived npx binary would lose every trace at process exit, defeating the purpose. For local dev without a cloud account, run the Docker image (option 2) or use the cloud sandbox at evalguard.ai/signup with the included free tier.