Sourced & dated buyer's guide
AI Gateway & Eval Buyer's Guide.
Side-by-side capability matrix for the 8 most common platforms in AI gateway, eval, and red-team RFPs. Real numbers from each vendor's public docs and GitHub — verified 2026-05-07.
SOC 2 evidence engineISO 42001 mappedEU AI ActGDPR
Last verified 2026-05-07. Portkey was acquired by Palo Alto Networks on 2026-04-30 — buyers should weigh acquisition risk + future bundling under Prisma AIRS.
Capability matrix
Eight platforms, one honest scorecard.
| Capability | EvalGuard | Portkey | Langfuse | Helicone | LangSmith | LiteLLM | OpenRouter | Vercel AI Gateway |
|---|---|---|---|---|---|---|---|---|
| Eval scorers (built-in) | 200+ | 0 (evals SDK is OpenAI passthrough) | Custom (no library) | Custom | Custom + LLM-as-judge | n/a | n/a | n/a |
| Red-team plugins | 300+ | 0 (delegated to partners) | 0 | 0 | 0 | 0 | 0 | 0 |
| Attack strategies | 100+ | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Compliance frameworks | 50 (EU AI Act, ISO 42001, NIST AI RMF, OWASP LLM Top 10, MITRE ATLAS, +45) | 0 framework code mappings (SOC2 / ISO / HIPAA service certs only) | 0 | 0 | 0 | 0 | 0 | 0 |
| Total guardrails (scorers + red-team) | 588 | ~40 (20 deterministic + 21 partner) | Custom | 0 | Custom | 0 | 0 | 0 |
| LLM firewall p95 | 2.57ms (published, /trust/latency) | No published number | n/a | n/a | n/a | n/a | n/a | n/a |
| Provider integrations | 90+ typed | 89 directories ('1,600+ via aliases') | via LiteLLM (100+) | Proxy-based (broad) | via LangChain | 100+ | 200+ | OpenAI + Anthropic + Google |
| Self-hosted (Docker + Helm) | Yes (full platform) | OSS gateway only — observability, prompt mgmt, RBAC, persistent cache, OTel exporter, PII model are SaaS-locked | Yes | Yes | Enterprise tier only | Yes | No (SaaS-only) | No (Vercel-tied) |
| OSS license | Apache 2.0 (full) | Apache 2.0 (gateway frame; persistence/SaaS closed) | MIT | Apache 2.0 | Closed-source | MIT | Closed-source | Closed-source |
| Acquisition / vendor risk | Independent, bootstrapped | ⚠️ Acquired by Palo Alto Networks (Apr 30, 2026) | YC W23, independent | YC W23, independent | LangChain Inc subsidiary | Independent | Independent | Vercel platform-tied |
| Cache key normalization | Field-aware by default (role+content+model+tenantId) | SHA256(entire body) — top_p/user/stream all force misses | n/a | n/a | n/a | Basic | n/a | n/a |
| Conditional/metadata routing DSL | $eq, $ne, $gt, $lt, $in, $nin, $regex, $exists, $contains, $and, $or, $not | Same MongoDB-ish DSL | n/a | n/a | n/a | Basic config rules | Provider preferences | Limited |
| Budget caps with auto-disable | Yes (per-tenant dailyBudgetUsd, enforced inline) | Enterprise tier only | n/a (observability only) | Cost tracking, no auto-disable | n/a | Yes | Per-key spending limit | Vercel platform billing |
| Sticky load balancing | Yes (per-tenant session affinity) | Yes (Feb 2026) | n/a | n/a | n/a | No | No | No |
| Quality-cost routing (auto-downgrade for simple queries) | Yes (gateway/quality-cost-router.ts) | No first-class primitive | n/a | n/a | n/a | No | Manual model preferences | No |
| SDK languages | TS, Python, Go | TS, Python | TS, Python | TS, Python | TS, Python, Go, Java | Python | REST only | TS only |
| Public head-to-head benchmarks | Yes (/trust/latency 20K runs, NeMo head-to-head) | None published | None | None | None | None | None | None |
| Pricing transparency | Public ($49/mo Pro, $199/mo Team, /pricing) | Public + Enterprise quote (PII anonymizer + BAA quote-only) | Public | Public | Public + Enterprise quote | Public + Enterprise | Pay-per-token | Vercel platform pricing |
Methodology
- EvalGuard numbers are drift-checked at build time against the live registries (
packages/core/src/counts.ts). Latency is measured at /trust/latency with reproducible scripts. - Portkey numbers are from a code audit of all 33 Portkey-AI repos on GitHub (verified 2026-05-07) — including the grep-confirmed absence of red-team and compliance-mapping code.
- Other competitors are from their public docs + GitHub READMEs. Where marketing was vague (e.g. "1,600+ models"), we counted actual provider directories in their codebases.
- Acquisition riskis a procurement consideration we surface explicitly. Portkey's announced acquisition by Palo Alto Networks (closes Q4 FY26) is documented at paloaltonetworks.com.
Try EvalGuard Free
No credit card required. Apache 2.0, self-hosted in 5 minutes.