Keep your scans and test-sets. Add guardrails and compliance.
Giskard is a great open-source evals + red-teaming + test-generation framework for agentic systems — the LLM vulnerability scan, conformity checks, and adversarial scenario generation. EvalGuard imports both your SuiteResult / scan results and your Scenario test-sets, then adds a hosted red-team platform (300+ plugins), a runtime LLM firewall, an AI gateway, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where Giskard stops, EvalGuard keeps going
Giskard is a first-class open-source project — its LLM vulnerability scan, conformity judges, and test generation are genuinely good, and the async-first design is clean. EvalGuard overlaps on scans, evals, and test-sets, then extends into the hosted security platform, runtime firewall, AI gateway, and compliance work you'd otherwise buy separately.
| Capability | Giskard | EvalGuard |
|---|---|---|
| Open-source LLM vulnerability scan & red-team | Yes | Yes — 300+ attack plugins |
| Adversarial scenario / test generation | Yes | Yes — versioned datasets + test generators |
| Conformity / semantic / comparison checks | Yes | Yes — 200+ scorers (LLM-as-judge, pairwise, rubric) |
| Open source & self-hosting | Yes | Yes — Apache-2.0 core, self-host available |
| CI gating | JUnit XML output | Yes — `gate` command + CI action, fail-closed |
| Hosted dashboards & RBAC | Giskard Hub | Yes — governed dashboards, roles, audit log |
| Runtime LLM firewall / guardrails | Not offered | Yes — real-time input + output firewall |
| AI gateway / BYOK proxy | Not offered | Yes — BYOK gateway, 15 proxied providers, similarity response cache |
| SOC 2 evidence automation | Not a documented product feature | Yes — live evidence engine + audit log |
Giskard claims on this page were verified against their source — github.com/Giskard-AI/giskard @ 22a6bc362 (v3.0.0b1, 2026-07-06) — on 2026-08-10. The two “Not offered” rows were settled by searching that whole tree — every file type, not just Python — for guardrail, firewall, gateway, proxy and BYOK modules; the only hit is an internal dev-process doc. Capabilities move; check their current docs before you decide. “SOC 2 evidence automation” describes a product capability, not any vendor’s own certification status.
The CI-gating row is their to_junit_xml in libs/giskard-checks/src/giskard/checks/export/junit.py. Giskard's root licence is Apache-2.0, but four of its five shipped sub-packages declare MIT — check the package you actually install rather than taking one label for the repo.
Migration paths
Giskard is two things. So the move is two paths.
Everything stays local until you choose to run it — we never touch your Giskard project or the Giskard Hub. Convert what you export with the EvalGuard CLI.
Results — bring your scans and suite runs
Serialize a Giskard SuiteResult (from a suite run or a vulnerability_scan) with to_hub_format(), then convert it to neutral-shape spans — every check's status, metrics, and message maps over, per scenario. A failed check stays a normal run; only a real check error is flagged as an error.
Test-sets — convert and run
A Giskard test-set is a ScenarioJSONL file (one scenario per line). Convert it to a runnable EvalGuard config — each scenario's generator prompt, category, tags, and checks carry over — then run it keyless on your machine, or on the EvalGuard cloud.
# e.g. giskard-scan scenarios, or your own Scenario test-set on disk (one Scenario per line)npx @evalguard/cli import:giskard giskard-testset.jsonl -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoYour Giskard checks (conformity judges, comparison and semantic graders) are Python objects, so they don't travel inside a test-set — the import attaches a runnable equals scorer when a scenario carries a reference expectation, and points you at llm-grader (for a conformity rule) / semantic-similarity to re-create the check's intent. The echo provider runs the whole loop with no API key — swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large Scenario test-set or a suite of custom conformity checks, send us your Giskard export and we'll help you map it and validate the first run. Free.