EvalGuard vs Portkey AI.
Open-source LLM gateway with observability, prompt management, and partner-guardrail integrationsPortkey (portkey.ai) is an LLM gateway with observability, prompt management, and 19 partner-guardrail integrations (21 plugin directories, two of which are Portkey's own). Their open-source claim is gateway-frame only: LLM-response cache persistence, semantic cache, circuit breaker, budget caps and the OTel exporter read injection points the OSS tree never sets, so they resolve only in their closed control plane. Verified by grep across the Portkey-AI/gateway source at commit 669825cbe (2026-05-25).
across the 26-row feature matrix · see “where Portkey AIleads” below
Competitor data (GitHub stars, downloads, feature counts, funding / acquisition status) verified as of 2026-08-09. EvalGuard's own counts are sourced live from the drift-checked registry.
Coverage at a glance
EvalGuard vs Portkey AI, by the numbers
Where both platforms publish a number, here's the gap. Our values come straight from the drift-checked registry; Portkey AI's are quoted as published.
| Feature | EvalGuard | Portkey AI |
|---|---|---|
| Eval Scorers | 200+ | 0 (OpenAI passthrough) |
| Red Team Plugins | 300+ | 0 |
| Attack Strategies | 100+ | 0 |
| Compliance Frameworks | 50 with code mappings | 0 (4 service certs) |
| Total Guardrails | 550+ (200+ scorers + 300+ plugins) | ~40 (20+21 partner) |
| Firewall p95 latency | 3.67ms (published) | Not published |
| LLM Providers | 90+ typed | 71 typed (1600+ models aliased) |
| OSS Scope | Apache 2.0 (SDKs + CLI) | Gateway frame only |
| Cache key | Field-aware | SHA256(whole body + URL) |
| Similarity cache | Yes (local lexical cosine, no embedding call) | Yes (hosted/enterprise, embedding-backed) |
| Conditional routing | Yes (MongoDB-style) | Yes (MongoDB-style) |
| Budget caps + auto-disable | Yes | Hosted/enterprise only |
| Sticky load balancing | Yes | Yes (hosted) |
| Circuit breaker | Yes | OSS-scaffolded only |
| Canary testing | Yes | No |
| Quality-cost routing | Yes | No |
| Format translation | Yes | Yes |
| Partner adapters | 200+ native, 0 partner | 19 partners (21 plugins) |
| MCP support | Yes (eval + server) | Yes (gateway, Jan 2026) |
| Skills Registry | No | Yes |
| Secret References | No | Yes |
| Native PII model | Local (sovereign) | Opt-in, prompt text sent to their API |
| Terraform provider | Yes | Yes |
| SDK languages | 4 published (TS/Py/Go/Java) | 2 (TS/Py) |
| Public benchmarks | Yes (/trust/latency) | None |
| Pricing transparency | Public tiers | Enterprise quote-cliff |
Why choose EvalGuard over Portkey AI
- 550+ guardrails (200+ scorers + 300+ red-team plugins) vs Portkey's ~40 (20 deterministic + 21 partner) — 14.8x breadth on safety + offensive testing
- Apache 2.0 SDKs + CLI, and the self-hostable platform is complete — Portkey's LLM-response cache, circuit breaker, budget caps and OTel exporter read Hono context keys their OSS tree never sets, so those paths resolve only in their closed control plane (verified by code audit grep)
- 50 compliance framework code mappings (EU AI Act, NIST AI RMF, ISO 42001, MITRE ATLAS, OWASP LLM Top 10, +45) vs Portkey's 4 service certifications with no code controls
- Field-aware cache keys (only role+content+model+tenantId) — Portkey's cache key is SHA-256 over the whole JSON body plus the URL, so a changed top_p or user field forces a miss
- Local PII model (Presidio + regex) keeps data sovereign — Portkey's own PII guardrail posts your prompt text to api.portkey.ai for inspection
- Quality-cost routing auto-downgrades simple queries to cheaper models — Portkey has no first-class cost-optimization router
- 4 published SDKs (TS/Python/Go/Java) from one monorepo vs Portkey's 2 (TS/Python)
- Public reproducible benchmark methodology at /trust/latency (3.67ms p95 over 20K runs) plus a published NeMo Guardrails head-to-head — we have not found a comparable guardrail-latency measurement published by Portkey
Where Portkey AI leads
- 21 partner-guardrail integrations (Aporia, Pillar, Patronus, Bedrock Guardrails, Azure Content Safety, etc.) — customers with existing contracts can chain them through Portkey's gateway today; we ship 200+ native scorers but no partner-webhook adapters yet
- Skills Registry (managed Anthropic-format skills synced to Claude Code/Cursor/Codex/Copilot) is a net-new product category we don't have a parallel for
- External secret manager references (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault) — we ship Vault-backed BYOK encryption for our own secrets, but don't yet allow referencing customer-managed secrets by ARN/path
- Venture-backed with a large public partner ecosystem — 19 named guardrail vendors integrate with their gateway, which is reach an independent OSS competitor doesn't yet match
Ready to switch from Portkey AI?
Start free. No credit card required. Migrate in minutes.