Keep your RAG metrics. Add security and guardrails.
Ragas is a great open-source RAG-evaluation library — faithfulness, answer relevancy, context precision, context recall, and the rest. EvalGuard imports both your evaluate() results and your evaluation datasets, then adds hosted red-team scans (300+ plugins), a runtime LLM firewall, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where Ragas stops, EvalGuard keeps going
Ragas is a first-class open-source RAG-metrics library — the cleanest way to measure faithfulness and context quality on a retrieval pipeline, and the metrics are genuinely good. EvalGuard overlaps on metrics, datasets, and CI, then extends into the hosted security, runtime firewall, and compliance work you'd otherwise buy separately.
| Capability | Ragas | EvalGuard |
|---|---|---|
| Open-source RAG metrics (faithfulness, answer relevancy, context precision / recall) | Yes | Yes — 200+ scorers, incl. RAG + LLM-as-judge |
| Reference-free & reference-based scoring | Yes | Yes — reference, rubric, and pairwise scorers |
| Evaluation datasets & synthetic test generation | Yes | Yes — versioned datasets + test generators |
| Open source & self-hosting | Yes | Yes — Apache-2.0 core, self-host available |
| CI gating (fail-closed on a regression) | DIY in pytest | Yes — `gate` command + CI action, fail-closed |
| Red-team & security scans | Not offered | Yes — 300+ attack plugins with threat-feed sync |
| Hosted dashboards & RBAC | Ragas app (limited) | Yes — governed dashboards, roles, audit log |
| Runtime LLM firewall / guardrails | Not offered | Yes — real-time input + output firewall |
| AI gateway / BYOK proxy | Not offered | Yes — BYOK gateway, 90+ providers, semantic cache |
| SOC 2 evidence engine | Not offered | Yes — live evidence engine + audit log |
Comparison reflects each product's core offering; Ragas continues to add features (including a hosted app) — check their latest docs for details.
Migration paths
Ragas is two things. So the move is two paths.
Everything stays local until you choose to run it — we never touch your Ragas project. Convert what you export with the EvalGuard CLI. Both the old (question / contexts / ground_truth) and new (user_input / retrieved_contexts / reference) Ragas field names are supported.
Results — bring your evaluate() runs
Serialize a Ragas EvaluationResult (its .to_pandas() records) to JSON, then convert it to neutral-shape spans — every metric column (faithfulness, answer relevancy, context precision / recall) maps into ragas.metric.*, and your retrieved contexts and reference are preserved. Ragas rows are scores, not pass/fail — no run is flagged as an error unless it genuinely errored.
Evaluation datasets — convert and run
Export a Ragas EvaluationDataset, convert it to a runnable EvalGuard config — inputs, references, and RAG retrieved_contexts all carry over — then run it keyless on your machine, or on the EvalGuard cloud.
# Python: json.dump(dataset.to_list(), f) → a JSON list of samplesnpx @evalguard/cli import:ragas ragas-dataset.json -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoYour Ragas metrics (faithfulness and friends) are computed by Python at evaluate()time, so they don't travel inside a dataset export — the import attaches a runnable equals scorer when your samples carry a reference, and points you at semantic-similarity / llm-grader (and the RAG scorers) to re-create a faithfulness / relevancy check. The echo provider runs the whole loop with no API key — swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large RAG evaluation dataset or a suite of custom Ragas metrics, send us your Ragas export and we'll help you map it and validate the first run. Free.