Migrate from Promptfoo in an afternoon.
Your Promptfoo YAML imports in one command. You get 300+ red-team plugins (2× Promptfoo's 157), 200+ scorers, and the same CLI ergonomics — plus firewall, gateway, observability, and FinOps in the same platform. One bill. Zero stitching.
Three reasons the calculus changed
Keep your red-team tooling model-vendor neutral
When you test models for jailbreaks, prompt injection, or policy violations, the tool you rely on shouldn't be tied to any single model provider. EvalGuard is independent — 90+ providers, zero model-vendor ownership.
300+plugins vs Promptfoo's 157
Every OWASP LLM Top 10 category plus indirect prompt injection, data exfiltration, multi-turn jailbreaks, PII leakage, policy violations, and many more. Kept up to date with threat-intel feed sync — not static. Plus 100+ adversarial strategies and 235 DLP patterns.
Eval + firewall + gateway + observability + FinOps — same workspace
Promptfoo does eval. Then you stitch Helicone for observability, Portkey for gateway, Langfuse for traces. EvalGuard replaces all of them with one auth, one bill, one SLA.
Your promptfoo.yaml maps over cleanly
Most fields need no rename, and the standard assert types are mapped to EvalGuard scorers automatically. The fields that differ are listed below — only the two arbitrary-code assertion types (javascript, python) need manual handling.
| Promptfoo | EvalGuard | Notes |
|---|---|---|
| providers: [openai:gpt-4o] | model: gpt-4o | Provider auto-detected from model prefix. |
| prompts: [...] | prompt: "{{input}}" | Inline prompt template; {{input}} interpolation supported. |
| tests: [...] | cases: [...] | Same shape: { input, expectedOutput, metadata }. |
| assert: [{type: contains, value: 'X'}] | scorers: ["contains"] | Simple string array. 200+ built-in scorers; contains/regex/similarity/etc. |
| assert: [{type: llm-rubric}] | scorers: ["llm-grader"] | Assertion-type names are mapped automatically (is-json → json-valid, model-graded-fact → factuality, similar → semantic-similarity). |
| assert: [{type: bleu|rouge|webhook}] | scorers: ["bleu" | "rouge-n" | "webhook"] | All three are built-in EvalGuard scorers — reference them directly (Promptfoo's rouge → rouge-n). |
| assert: [{type: javascript|python}] | manual — see note | Arbitrary code — no built-in scorer. eval:local skips these with a warning; write a custom scorer instead. |
| assert config via type+value | scorerOptions: { contains: { value: 'X' } } | Per-scorer config is a separate object. |
| redteam: {plugins: [...]} | `evalguard scan` command | Red team lives in a separate config for scans; same platform, separate flow. |
CLI commands
| Promptfoo CLI | EvalGuard CLI |
|---|---|
| promptfoo eval | evalguard eval |
| promptfoo eval --no-cache | evalguard eval:local |
| promptfoo view | evalguard view |
| promptfoo redteam run | evalguard scan |
| promptfoo share | evalguard share |
| promptfoo init | evalguard init |
In four commands
- 1Install the CLI
npm i -g @evalguard/cli - 2Authenticate (optional — only for the cloud dashboard)
evalguard login --key <your-eg_key> - 3Import your Promptfoo config
evalguard import:promptfoo promptfoo.yaml - 4Run the eval locally (uses your provider key, no account needed)
evalguard eval:local evalguard.config.json
eval:local runs on your machine with your own provider key (e.g. OPENAI_API_KEY) — no EvalGuard account needed. To run on the cloud instead (shared dashboards, run history), log in and use evalguard eval --project <id>.
Stuck on a Promptfoo feature that doesn't map cleanly? Tell us — we'll add the shim within a business day.
Your tests don't belong to the vendor you're testing
Move in an afternoon. Free forever tier — 50K traces/month, unlimited projects, AI Gateway included.
Start free — no credit card