Keep your metrics. Add security and guardrails.
DeepEval is a great open-source, pytest-style metrics library — GEval, AnswerRelevancy, Faithfulness, Hallucination, and the rest. EvalGuard imports both your evaluate() results and your golden datasets, then adds hosted red-team scans (300+ plugins), a runtime LLM firewall, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where DeepEval stops, EvalGuard keeps going
DeepEval is a first-class open-source metrics library — the cleanest pytest-style eval developer experience out there, and the metrics are genuinely good. EvalGuard overlaps on metrics, goldens, and CI, then extends into the hosted security, runtime firewall, and compliance work you'd otherwise buy separately.
| Capability | DeepEval | EvalGuard |
|---|---|---|
| Open-source metrics (GEval, AnswerRelevancy, Faithfulness, Hallucination) | Yes | Yes — 200+ scorers (LLM-as-judge, pairwise, rubric) |
| Pytest-native / CI integration | Yes | Yes — `gate` command + CI action, fail-closed |
| Golden datasets & synthetic data | Yes | Yes — versioned datasets + test generators |
| Open source & self-hosting | Yes | Yes — Apache-2.0 core, self-host available |
| Red-team & security scans | Via DeepTeam | Yes — 300+ attack plugins |
| Hosted dashboards & RBAC | Confident AI | Yes — governed dashboards, roles, audit log |
| Runtime LLM firewall / guardrails | Not in the OSS package — a Confident AI feature | Yes — real-time input + output firewall |
| AI gateway / BYOK proxy | Connects to OpenRouter / Portkey / LiteLLM; doesn't ship one | Yes — BYOK gateway, 15 proxied providers, similarity response cache |
| SOC 2 evidence automation | Not a documented product feature | Yes — live evidence engine + audit log |
DeepEval claims on this page were verified against their source — github.com/confident-ai/deepeval @ bd10fa61c (v4.1.5, 2026-08-05) — on 2026-08-10. The “Not offered” row was settled by searching that tree for guardrail modules and classes by filename and by content — there are none, and their own README still lists Guardrails as an unchecked roadmap item. Capabilities move; check their current docs before you decide. “SOC 2 evidence automation” describes a product capability, not any vendor’s own certification status.
The red-team cell is their own wording: the deepeval/red_teaming/package contains one file, a README reading “The Red Teaming module is now in DeepTeam for deepeval-v3.0 onwards”. Hosted dashboards — and guardrails — are Confident AI, their commercial platform, not the Apache-2.0 package.
Migration paths
DeepEval is two things. So the move is two paths.
Everything stays local until you choose to run it — we never touch your DeepEval account or Confident AI project. Convert what you export with the EvalGuard CLI.
Results — bring your evaluate() runs
Serialize a DeepEval EvaluationResult (its test_resultslist) to JSON, then convert it to neutral-shape spans — every metric's score, threshold, pass/fail, and reason maps over. A failed assertion stays a normal run; only a real evaluation error is flagged as an error.
Golden datasets — convert and run
Export a DeepEval golden dataset, convert it to a runnable EvalGuard config — inputs, expected outputs, and RAG context all carry over — then run it keyless on your machine, or on the EvalGuard cloud.
# Python: dataset.save_as('json', directory) → a JSON list of goldensnpx @evalguard/cli import:deepeval deepeval-dataset.json -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoYour DeepEval metrics (GEval and friends) are Python code, so they don't travel inside a dataset export — the import attaches a runnable equals scorer when your goldens carry an expected output, and points you at llm-grader / semantic-similarity to re-create a GEval-style rubric. The echo provider runs the whole loop with no API key — swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large golden dataset or a suite of custom GEval metrics, send us your DeepEval export and we'll help you map it and validate the first run. Free.