Keep your metrics. Add security and guardrails.
DeepEval is a great open-source, pytest-style metrics library — GEval, AnswerRelevancy, Faithfulness, Hallucination, and the rest. EvalGuard imports both your evaluate() results and your golden datasets, then adds hosted red-team scans (300+ plugins), a runtime LLM firewall, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where DeepEval stops, EvalGuard keeps going
DeepEval is a first-class open-source metrics library — the cleanest pytest-style eval developer experience out there, and the metrics are genuinely good. EvalGuard overlaps on metrics, goldens, and CI, then extends into the hosted security, runtime firewall, and compliance work you'd otherwise buy separately.
| Capability | DeepEval | EvalGuard |
|---|---|---|
| Open-source metrics (GEval, AnswerRelevancy, Faithfulness, Hallucination) | Yes | Yes — 200+ scorers (LLM-as-judge, pairwise, rubric) |
| Pytest-native / CI integration | Yes | Yes — `gate` command + CI action, fail-closed |
| Golden datasets & synthetic data | Yes | Yes — versioned datasets + test generators |
| Open source & self-hosting | Yes | Yes — Apache-2.0 core, self-host available |
| Red-team & security scans | Via DeepTeam | Yes — 300+ attack plugins with threat-feed sync |
| Hosted dashboards & RBAC | Confident AI | Yes — governed dashboards, roles, audit log |
| Runtime LLM firewall / guardrails | Not offered | Yes — real-time input + output firewall |
| AI gateway / BYOK proxy | Not offered | Yes — BYOK gateway, 90+ providers, semantic cache |
| SOC 2 evidence engine | Not offered | Yes — live evidence engine + audit log |
Comparison reflects each product's core offering; DeepEval offers red-teaming via its companion DeepTeam project and hosted dashboards via Confident AI — check their latest docs for details.
Migration paths
DeepEval is two things. So the move is two paths.
Everything stays local until you choose to run it — we never touch your DeepEval account or Confident AI project. Convert what you export with the EvalGuard CLI.
Results — bring your evaluate() runs
Serialize a DeepEval EvaluationResult (its test_resultslist) to JSON, then convert it to neutral-shape spans — every metric's score, threshold, pass/fail, and reason maps over. A failed assertion stays a normal run; only a real evaluation error is flagged as an error.
Golden datasets — convert and run
Export a DeepEval golden dataset, convert it to a runnable EvalGuard config — inputs, expected outputs, and RAG context all carry over — then run it keyless on your machine, or on the EvalGuard cloud.
# Python: dataset.save_as('json', directory) → a JSON list of goldensnpx @evalguard/cli import:deepeval deepeval-dataset.json -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoYour DeepEval metrics (GEval and friends) are Python code, so they don't travel inside a dataset export — the import attaches a runnable equals scorer when your goldens carry an expected output, and points you at llm-grader / semantic-similarity to re-create a GEval-style rubric. The echo provider runs the whole loop with no API key — swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large golden dataset or a suite of custom GEval metrics, send us your DeepEval export and we'll help you map it and validate the first run. Free.