Migrate from Promptfoo in an afternoon.
Your Promptfoo YAML imports in one command. You get 300+ red-team plugins, 200+ scorers, and the same CLI ergonomics — plus firewall, gateway, observability, and FinOps in the same platform. One bill. Zero stitching.
Three reasons the calculus changed
Test every model you ship on, from one config
Jailbreaks, prompt injection and policy violations behave differently on every model, so the same suite has to run everywhere you deploy. EvalGuard reaches 90+ providers from one config — no per-vendor rewrite.
300+ attack plugins
Every OWASP LLM Top 10 category plus indirect prompt injection, data exfiltration, multi-turn jailbreaks, PII leakage, policy violations, and many more. Plus 100+ adversarial strategies and 439 DLP patterns.
For scale: Promptfoo's public red-team plugin registry held 155 plugins when we counted it from their source at commit db03327, re-verified 2026-08-10.
Eval + firewall + gateway + observability + FinOps — same workspace
Promptfoo does eval and red team well, and ships its own OpenTelemetry tracing. What EvalGuard puts next to it is the runtime tier — a first-party firewall on the request path, a BYOK gateway, and cost attribution — behind one auth, one bill, one SLA.
Promptfoo claims on this page were verified against their source — github.com/promptfoo/promptfoo @ db03327 — on 2026-08-10. “Doesn’t ship a gateway” was checked against that tree: their two gateway files are provider clients for Cloudflare AI Gateway and MLflow, not a gateway of their own. Capabilities move; check their current docs before you decide.
Your promptfoo.yaml maps over cleanly
Most fields need no rename, and the standard assert types are mapped to EvalGuard scorers automatically. The fields that differ are listed below — only the two arbitrary-code assertion types (javascript, python) need manual handling.
| Promptfoo | EvalGuard | Notes |
|---|---|---|
| providers: [openai:gpt-4o] | model: gpt-4o | Provider auto-detected from model prefix. |
| prompts: [...] | prompt: "{{input}}" | Inline prompt template; {{input}} interpolation supported. |
| tests: [...] | cases: [...] | Same shape: { input, expectedOutput, metadata }. |
| assert: [{type: contains, value: 'X'}] | scorers: ["contains"] | Simple string array. 200+ built-in scorers; contains / regex-match / semantic-similarity / etc. |
| assert: [{type: llm-rubric}] | scorers: ["llm-grader"] | Assertion-type names are mapped automatically (is-json → json-valid, model-graded-closedqa → llm-grader, similar → semantic-similarity). |
| assert: [{type: bleu|rouge-n|webhook}] | scorers: ["bleu" | "rouge-n" | "webhook"] | All three are built-in EvalGuard scorers under the same names Promptfoo uses — no rename needed. |
| assert: [{type: javascript|python}] | manual — see note | Arbitrary code — no built-in scorer. eval:local skips these with a warning; write a custom scorer instead. |
| assert config via type+value | scorerOptions: { contains: { value: 'X' } } | Per-scorer config is a separate object. |
| redteam: {plugins: [...]} | `evalguard scan` command | Red team lives in a separate config for scans; same platform, separate flow. |
CLI commands
| Promptfoo CLI | EvalGuard CLI |
|---|---|
| promptfoo eval | evalguard eval |
| promptfoo eval --no-cache | evalguard eval:local |
| promptfoo view | evalguard view |
| promptfoo redteam run | evalguard scan |
| promptfoo share | evalguard share |
| promptfoo init | evalguard init |
In four commands
- 1Install the CLI
npm i -g @evalguard/cli - 2Authenticate (optional — only for the cloud dashboard)
evalguard login --key <your-eg_key> - 3Import your Promptfoo config
evalguard import:promptfoo promptfoo.yaml - 4Run the eval locally (uses your provider key, no account needed)
evalguard eval:local evalguard.config.json
eval:local runs on your machine with your own provider key (e.g. OPENAI_API_KEY) — no EvalGuard account needed. To run on the cloud instead (shared dashboards, run history), log in and use evalguard eval --project <id>.
Stuck on a Promptfoo feature that doesn't map cleanly? Tell us — we'll add the shim within a business day.
Keep the eval suite. Add everything around it.
Move in an afternoon. Free forever tier — 50K traces/month, unlimited projects, AI Gateway included.
Start free — no credit card