Keep your RAG metrics. Add security and guardrails.
Ragas is a great open-source RAG-evaluation library — faithfulness, answer relevancy, context precision, context recall, and the rest. EvalGuard imports both your evaluate() results and your evaluation datasets, then adds hosted red-team scans (300+ plugins), a runtime LLM firewall, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where Ragas stops, EvalGuard keeps going
Ragas is a first-class open-source RAG-metrics library — the cleanest way to measure faithfulness and context quality on a retrieval pipeline, and the metrics are genuinely good. EvalGuard overlaps on metrics, datasets, and CI, then extends into the hosted security, runtime firewall, and compliance work you'd otherwise buy separately.
| Capability | Ragas | EvalGuard |
|---|---|---|
| Open-source RAG metrics (faithfulness, answer relevancy, context precision / recall) | Yes | Yes — 200+ scorers, incl. RAG + LLM-as-judge |
| Reference-free & reference-based scoring | Yes | Yes — reference, rubric, and pairwise scorers |
| Evaluation datasets & synthetic test generation | Yes | Yes — versioned datasets + test generators |
| Open source & self-hosting | Yes | Yes — Apache-2.0 core, self-host available |
| CI gating (fail-closed on a regression) | `ragas evals` CLI, but no score-threshold gate | Yes — `gate` command + CI action, fail-closed |
| Red-team & security scans | Not in the open-source repo | Yes — 300+ attack plugins |
| Hosted dashboards & RBAC | Ragas App (hosted, not source-visible) | Yes — governed dashboards, roles, audit log |
| Runtime LLM firewall / guardrails | Not in the open-source repo | Yes — real-time input + output firewall |
| AI gateway / BYOK proxy | Not in the open-source repo | Yes — BYOK gateway, 15 proxied providers, similarity response cache |
| SOC 2 evidence automation | Not a documented product feature | Yes — live evidence engine + audit log |
Ragas claims on this page were verified against their source — github.com/vibrantlabsai/ragas (formerly explodinggradients/ragas), full clone, main @ 298b682 — on 2026-08-10. Rows reading “not in the open-source repo” were searched case-insensitively across that whole tree — src/ and the docs/ that builds docs.ragas.io — for red team, guardrail, firewall, prompt injection, gateway and BYOK. Capabilities move; check their current docs before you decide. “SOC 2 evidence automation” describes a product capability, not any vendor’s own certification status.
Three rows say “not in the open-source repo” rather than “not offered”, and the distinction is deliberate: Ragas also runs a hosted app whose source we cannot read, so an absence in the repository is the widest claim we can stand behind. Check their current product before you rely on any of the three.
Migration paths
Ragas is two things. So the move is two paths.
Everything stays local until you choose to run it — we never touch your Ragas project. Convert what you export with the EvalGuard CLI. Both the old (question / contexts / ground_truth) and new (user_input / retrieved_contexts / reference) Ragas field names are supported.
Results — bring your evaluate() runs
Serialize a Ragas EvaluationResult (its .to_pandas() records) to JSON, then convert it to neutral-shape spans — every metric column (faithfulness, answer relevancy, context precision / recall) maps into ragas.metric.*, and your retrieved contexts and reference are preserved. Ragas rows are scores, not pass/fail — no run is flagged as an error unless it genuinely errored.
Evaluation datasets — convert and run
Export a Ragas EvaluationDataset, convert it to a runnable EvalGuard config — inputs, references, and RAG retrieved_contexts all carry over — then run it keyless on your machine, or on the EvalGuard cloud.
# Python: json.dump(dataset.to_list(), f) → a JSON list of samplesnpx @evalguard/cli import:ragas ragas-dataset.json -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoYour Ragas metrics (faithfulness and friends) are computed by Python at evaluate()time, so they don't travel inside a dataset export — the import attaches a runnable equals scorer when your samples carry a reference, and points you at semantic-similarity / llm-grader (and the RAG scorers) to re-create a faithfulness / relevancy check. The echo provider runs the whole loop with no API key — swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large RAG evaluation dataset or a suite of custom Ragas metrics, send us your Ragas export and we'll help you map it and validate the first run. Free.