Keep your scans and test-sets. Add guardrails and compliance.
Giskard is a great open-source evals + red-teaming + test-generation framework for agentic systems — the LLM vulnerability scan, conformity checks, and RAGET test generation. EvalGuard imports both your SuiteResult / scan results and your Scenario test-sets, then adds a hosted red-team platform (300+ plugins), a runtime LLM firewall, an AI gateway, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where Giskard stops, EvalGuard keeps going
Giskard is a first-class open-source project — its LLM vulnerability scan, conformity judges, and test generation are genuinely good, and the async-first design is clean. EvalGuard overlaps on scans, evals, and test-sets, then extends into the hosted security platform, runtime firewall, AI gateway, and compliance work you'd otherwise buy separately.
| Capability | Giskard | EvalGuard |
|---|---|---|
| Open-source LLM vulnerability scan & red-team | Yes | Yes — 300+ attack plugins with threat-feed sync |
| Test / scenario generation (RAGET, adversarial) | Yes | Yes — versioned datasets + test generators |
| Conformity / semantic / comparison checks | Yes | Yes — 200+ scorers (LLM-as-judge, pairwise, rubric) |
| Open source & self-hosting | Yes | Yes — Apache-2.0 core, self-host available |
| CI gating | JUnit XML output | Yes — `gate` command + CI action, fail-closed |
| Hosted dashboards & RBAC | Giskard Hub | Yes — governed dashboards, roles, audit log |
| Runtime LLM firewall / guardrails | Not offered | Yes — real-time input + output firewall |
| AI gateway / BYOK proxy | Not offered | Yes — BYOK gateway, 15 proxied providers, semantic cache |
| SOC 2 evidence engine | Not offered | Yes — live evidence engine + audit log |
Comparison reflects each product's core offering; Giskard offers hosted dashboards via the Giskard Hub and JUnit-based CI output — check their latest docs for details.
Migration paths
Giskard is two things. So the move is two paths.
Everything stays local until you choose to run it — we never touch your Giskard project or the Giskard Hub. Convert what you export with the EvalGuard CLI.
Results — bring your scans and suite runs
Serialize a Giskard SuiteResult (from a suite run or a vulnerability_scan) with to_hub_format(), then convert it to neutral-shape spans — every check's status, metrics, and message maps over, per scenario. A failed check stays a normal run; only a real check error is flagged as an error.
Test-sets — convert and run
A Giskard test-set is a ScenarioJSONL file (one scenario per line). Convert it to a runnable EvalGuard config — each scenario's generator prompt, category, tags, and checks carry over — then run it keyless on your machine, or on the EvalGuard cloud.
# e.g. giskard-scan scenarios, or your own Scenario test-set on disk (one Scenario per line)npx @evalguard/cli import:giskard giskard-testset.jsonl -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoYour Giskard checks (conformity judges, comparison and semantic graders) are Python objects, so they don't travel inside a test-set — the import attaches a runnable equals scorer when a scenario carries a reference expectation, and points you at llm-grader (for a conformity rule) / semantic-similarity to re-create the check's intent. The echo provider runs the whole loop with no API key — swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large Scenario test-set or a suite of custom conformity checks, send us your Giskard export and we'll help you map it and validate the first run. Free.