Test, red-team, and monitor LLM agents in production.
Find every way your LLM agent can be jailbroken or leak data — before your customers do — then ship the compliance evidence to prove it. BYOK, self-hostable, no vendor lock-in.
Free plan — 50,000 traces/month. No credit card required.
Works with your entire stack
One platform for every model you ship.
By the numbers
Built for production from day one
One platform
Six products. Zero blind spots.
Eval
Score every model change before it ships.
- 201 built-in scorers + custom graders
- Datasets, versioning, and CI pass/fail gates
- LLM-as-judge, embedding, exact-match, RAG
Open & verifiable
Don't take our word for it — run the proof yourself.
Every claim below is independently checkable today. No vanity logos.
Open-source SDKs
Apache-2.0 packages on npm & PyPI — TypeScript, Python & Go. Read the published source and pin any version.
Browse on npmReproducible benchmarks
2.57ms firewall p95 and the first independent NeMo-Guardrails head-to-head — methodology you can re-run.
See the numbersSelf-host & BYOK
Run the whole platform in your own VPC and bring your own model keys — no data leaves your boundary.
Self-hosting guideWant to shape the roadmap and get white-glove onboarding?
Become a design partnerBuilt for high-stakes industries
Proof, not promises
The widest coverage, at the lowest latency.
More red-team coverage than any open-source tool — running behind a firewall whose p95 stays flat as traffic scales.
Red-team coverage vs the field
Built-in attack plugins. More vectors → more real findings, before your customers hit them.
built-in attack plugins
Competitor plugin counts from public repos & docs, verified 2026-05-30; EvalGuard's is live from the registry. Methodology: /trust.
Firewall p95 stays flat under load
A naive in-band scanner degrades as RPS climbs. EvalGuard's pre-filter holds ~2.57ms.
Three lines. Production-grade eval.
No infra. No vendor lock-in. The SDK runs in Node, edge, browser, and CI — same code path, same scorers, same results.
- TypeScript-first SDK with full type inference
- 91 providers via the unified gateway
- Streams results — no batch-waiting
- Edge-runtime compatible (Vercel + Cloudflare)
import { evaluate } from "@evalguard/core";
const result = await evaluate({
input: "What's the capital of France?",
output: response,
assertions: [
{ type: "answer-relevance", threshold: 0.8 },
{ type: "hallucination" },
{ type: "pii-leak" },
],
});
if (!result.passed) {
console.log(result.failingCriteria);
}Lifecycle
From your first eval to the production firewall.
One workspace covers the full lifecycle — pre-launch evals, red-team scans, runtime guardrails, and audit-ready compliance.
01 — Evaluate
Test every prompt before it reaches production.
Run 201 built-in scorers across faithfulness, relevance, toxicity, and more. Create custom LLM-as-judge evaluators. Catch regressions before your users do.
- Pairwise A/B testing + ELO model leaderboards
- Drop into CI — fail the build under threshold
- Custom LLM-as-judge with any grading rubric
02 — Secure
Red team your AI with 257 attack plugins.
Automated adversarial testing across 47 strategies — prompt injection, jailbreak, PII extraction, and more. OWASP LLM Top 10 compliance reports instantly.
- Adaptive UCB1 multi-turn attacker
- OWASP LLM + Agentic Top 10, auto-mapped
- Per-finding evidence + suggested fix
03 — Debug
Chrome DevTools for your AI agents.
Visualize every step of your agent's reasoning chain. Detect infinite loops, identify tool call failures, and pinpoint where things went wrong.
- Full span tree: retrieve → llm → tool
- Infinite-loop + failed-tool detection
- Cost & latency attributed per step
04 — Monitor
Real-time observability for every LLM call.
Track latency, cost, and quality in real time. Set alerts on drift, spikes, and anomalies. Get notified before your users complain.
- OTel-native spans, ClickHouse rollups
- Drift, spike & anomaly alerting
- Live latency / cost / quality dashboards
Receipts
What auditors, security teams, and ML engineers actually ask for.
Real attack coverage, real eval scoring, real evidence exports — not slides.
Evaluation Engine
Security Scanner
Agent Debugging
LLM Firewall
Monitoring
Compliance Evidence
AI Gateway
NL→Eval Pipeline
Adaptive Red Teaming
Quickstart
Up and running in minutes.
Three steps. No infrastructure to manage.
Install the CLI
# Install the CLI globally
npm install -g @evalguard/cliRun your first evaluation
# Run an evaluation
evalguard eval evalguard.yaml \
--model gpt-4oShip with confidence
# Add to CI/CD pipeline
evalguard gate --min-score 0.9
> All checks passed. Deploying...Personas
Built for your role.
Tailored workflows for every stakeholder in the AI pipeline.
For CISOs
- Automated OWASP LLM Top 10 compliance
- Real-time vulnerability dashboard
- SOC 2 readiness & GDPR audit documentation
- Policy enforcement across all AI endpoints
For Engineering Leads
- CI/CD quality gates for LLM outputs
- Cost optimization with caching & routing
- Team-wide evaluation dashboards
- Incident root cause analysis
For ML Engineers
- 201 pre-built + custom evaluation metrics
- A/B model comparison with confidence intervals
- Trace-level debugging for agent chains
- Dataset versioning with golden test sets
Enterprise
Built for Enterprise.
Enterprise-grade security, compliance, and deployment options from day one.
SOC 2 control-evidence engine — live
GDPR Compliant
SSO / SAML
Self-Hosted
RBAC + Audit Log
Enterprise SLA
FAQ
Frequently asked questions.
How is EvalGuard different from other LLM evaluation tools?
Most tools cover one layer — eval, security, or observability. EvalGuard unifies all six (eval, firewall, gateway, observability, red-team, compliance) on one platform so signals compose end-to-end. You don't stitch evals + Helicone + Promptfoo + a homegrown firewall; you run one workspace with one auth, one bill, one SLA. We also ship 5× the red-team coverage (257 attack plugins vs 50–60 for the nearest open-source tool) and the only adaptive multi-turn red team that productionizes UCB1 bandit attack-strategy selection.
Can I use EvalGuard with any LLM provider?
Yes — 91 typed providers in the gateway today (OpenAI, Anthropic, Gemini, Bedrock, Azure, Mistral, Cohere, DeepSeek, xAI, Together, Replicate, OpenRouter, Groq, Perplexity, and more). Same SDK call shape across all of them. BYOK key vault, automatic failover, semantic caching, and cost tracking are uniform — you don't write provider-specific code.
How long does it take to get started?
Minutes. Run `pip install evalguardai` (or `npm install -g @evalguard/cli`), drop an `evalguard.yaml` in your repo, and run `evalguard eval --model gpt-4o`. The free tier has no credit card, no time limit. Add `evalguard gate --min-score 0.9` to your CI/CD pipeline and you have eval gates on every PR.
Is my data secure? Do you store prompts and completions?
Yes and by default no. All data is encrypted at rest (AES-256-GCM) and in transit (TLS 1.3). Prompts and completions are NOT stored unless you explicitly enable trace logging per project. When enabled, retention is configurable per org and append-only audit logs track every access. Enterprise plans support BYOK envelope encryption, VPC deployment, and full air-gapped mode.
What does the free tier include?
The most generous free tier in AI evals: 50,000 traces/month (5× Helicone/Portkey, matches Langfuse), 100 red-team scans, all 257 attack plugins, all 201 scorers + custom, AI Gateway (route/cache/failover) — yes, free — pairwise eval, ELO leaderboard, annotation queues (50/mo), full 5-layer firewall, OTel observability, 30-day retention, unlimited projects, 5 team members. No credit card. Upgrade to Pro ($49/mo) for 500K traces, FinOps dashboard, and the Prompt IDE.
How does the security scanner work?
Point it at your model endpoint (OpenAI URL, Anthropic, your hosted model, anything). It runs 257 attack plugins across 47 strategies — prompt injection, jailbreak, PII extraction, data exfiltration, multi-turn crescendo, role-play escalation. The adaptive red-team uses UCB1 to focus on attacks that are actually working. Output is an OWASP LLM Top 10 + OWASP Agentic Top 10 compliance report plus per-finding evidence (request, response, attack vector, suggested fix).
Can I create custom evaluation metrics?
Yes. Two paths: (1) Write a custom LLM-as-judge with any grading rubric — pass a prompt template, EvalGuard handles the eval-loop + scoring. (2) Drop in a TypeScript / Python function that takes (input, output, expected) and returns a score 0–1. Both run through the same eval engine, same reports, same CI gates as the 201 built-in scorers.
Do you offer enterprise features?
Yes — a live SOC 2 control-evidence engine (exportable to your auditor today; third-party attestation is on our roadmap), GDPR with 7 EU residency regions, SSO/SAML/SCIM with Okta + Azure + Google, RBAC with 5 roles + WORM audit logs (7-year retention), VPC + on-prem deployment, named SRE + 1h SLA response, 99.95% uptime target. See /enterprise for the full feature list.
Compliance
Audit reports your CISO will sign.
EU AI Act risk classification, ISO 42001 statement-of-applicability, SOC 2 evidence collector, OWASP LLM/Agentic Top 10 — 33 frameworks mapped out of the box.
Early-access feedback
What testers noticed first.
Anonymized real feedback from our beta circle.
“Swapped new OpenAI() for the wrapper and the rest of the pipeline didn’t change. Traces and cost-ledger started populating on their own, and the eval-on-response flag gave me a per-call quality score without me wiring anything up. Slotted into our pipeline in an afternoon — most observability bolt-ons eat a sprint.”“Dropped the firewall inline in front of an agent stack we were piloting. It caught the model echoing back a customer email it shouldn’t have, before our review caught it. p95 stayed under 3ms— first inline guardrail I’ve benched that didn’t add a latency tier.”
Ready to ship better AI?
Free forever — 50,000 traces/month, AI Gateway included.Pro plans start at $49/mo.
Start evaluating, securing, and monitoring your AI in production today.