Migrate from Promptfoo in an afternoon.
Your Promptfoo YAML imports in one command. You get 250+ red-team plugins (5× Promptfoo), 200+ scorers, and the same CLI ergonomics — plus firewall, gateway, observability, and FinOps in the same platform. One bill. Zero stitching.
Three reasons the calculus changed
Your red-team tool shouldn't be owned by the model you're testing
OpenAI now owns Promptfoo. That's a conflict if you rely on it to test OpenAI models for jailbreaks, prompt injection, or policy violations. EvalGuard is independent — 91 providers, zero model-vendor ownership.
250+plugins vs Promptfoo's ~50
Every OWASP LLM Top 10 category plus indirect prompt injection, data exfiltration, multi-turn jailbreaks, PII leakage, policy violations, and many more. Kept up to date with threat-intel feed sync — not static. Plus 40+ adversarial strategies and 200 DLP patterns.
Eval + firewall + gateway + observability + FinOps — same workspace
Promptfoo does eval. Then you stitch Helicone for observability, Portkey for gateway, Langfuse for traces. EvalGuard replaces all of them with one auth, one bill, one SLA.
Your promptfoo.yaml maps over cleanly
Most fields need no rename, and the standard assert types are mapped to EvalGuard scorers automatically. The fields that differ are listed below — five code/metric assertion types (javascript, python, webhook, rouge, bleu) need manual handling.
| Promptfoo | EvalGuard | Notes |
|---|---|---|
| providers: [openai:gpt-4o] | model: gpt-4o | Provider auto-detected from model prefix. |
| prompts: [...] | prompt: "{{input}}" | Inline prompt template; {{input}} interpolation supported. |
| tests: [...] | cases: [...] | Same shape: { input, expectedOutput, metadata }. |
| assert: [{type: contains, value: 'X'}] | scorers: ["contains"] | Simple string array. 200+ built-in scorers; contains/regex/similarity/etc. |
| assert: [{type: llm-rubric}] | scorers: ["llm-grader"] | Assertion-type names are mapped automatically (is-json → json-valid, model-graded-fact → factuality, similar → semantic-similarity). |
| assert: [{type: javascript|python|webhook|rouge|bleu}] | manual — see note | No 1:1 scorer; eval:local skips these with a warning. Use a custom scorer / webhook action / semantic-similarity. |
| assert config via type+value | scorerOptions: { contains: { value: 'X' } } | Per-scorer config is a separate object. |
| redteam: {plugins: [...]} | `evalguard scan` command | Red team lives in a separate config for scans; same platform, separate flow. |
CLI commands
| Promptfoo CLI | EvalGuard CLI |
|---|---|
| promptfoo eval | evalguard eval |
| promptfoo eval --no-cache | evalguard eval:local |
| promptfoo view | evalguard view |
| promptfoo redteam run | evalguard scan |
| promptfoo share | evalguard share |
| promptfoo init | evalguard init |
In four commands
- 1Install the CLI
npm i -g @evalguard/cli - 2Authenticate
evalguard login --key <your-eg_key> - 3Import your Promptfoo config
evalguard import:promptfoo promptfoo.yaml - 4Run the eval
evalguard eval
Stuck on a Promptfoo feature that doesn't map cleanly? Tell us — we'll add the shim within a business day.
Your tests don't belong to the vendor you're testing
Move in an afternoon. Free forever tier — 50K traces/month, unlimited projects, AI Gateway included.
Start free — no credit card