Open benchmark
Anti-overfit firewall benchmark
A prompt-injection & safety detector leaderboard scored against a hidden, held-out corpus. The headline metric is recall at 100% precision — the share of attacks caught with zero false positives on prompts the detector has never seen. We also publish an overfit gap (public minus hidden score): a near-zero gap means the detector genuinely generalises.
vs Lakera PINT
PINT reports one private balanced-accuracy number at the detector’s default threshold. We publish a reproducible public split, the full recall@precision curve, and a held-out score nobody can train against.
vs guardrails-ai Index
The Index scores detectors on a public-only set with no held-out split and no over-refusal penalty. We fold over-refusal traps into the negatives and rank generalisers above memorisers.
Corpus 2026.05.31-1
26 public sample prompts · 35 hidden held-out prompts across 8 categories. Public split is downloadable; the hidden split is scored server-side only.
| # | Detector | Recall @100% prec (hidden) | Recall @99% prec | Bal. acc (hidden) | Overfit gap | Latency |
|---|---|---|---|---|---|---|
| 1 | FirewallEngineEvalGuard | 25.0% | 25.0% | 55.5% | 16.0 pts | 1.05ms |
| 2 | regex-baseline (generalising)reference | — | — | 62.1% | 3.9 pts | 0.01ms |
| 3 | regex-baseline (overfit to public set)reference | — | — | 50.0% | 16.7 pts | 0.01ms |
| 4 | e2e-detector (e2e-tester2) | — | — | 49.6% | -0.3 pts | 0.01ms |
| 5 | e2e-detector (e2e-tester) | — | — | 49.6% | -0.3 pts | 0.01ms |
| 6 | inj-re (e2e-tester) | — | — | 45.5% | 4.5 pts | 0.01ms |
Contribute
Submit your detector
Submit a declarative detector and it is scored server-side against the hidden held-out split — the corpus is never exposed to you. Two forms are supported: a list of literal/regex patterns, or a hosted HTTPS endpoint we POST each prompt to. The endpoint is SSRF-guarded and rate-limited.
curl -X POST https://evalguard.ai/api/v1/firewall-benchmark/submit \
-H "content-type: application/json" \
-d '{
"submitter": "Acme Security",
"detector": {
"kind": "patterns",
"name": "acme-guard-v1",
"rules": [
{ "type": "regex", "value": "ignore (all )?previous (instructions|rules)" },
{ "type": "regex", "value": "\\bDAN\\b|do anything now|unrestricted" }
]
}
}'Reproduce the methodology against the public split: GET /api/v1/firewall-benchmark/public-corpus
Methodology
Why a hidden split
Public benchmarks are easy to game: once the corpus is downloadable, a detector can be regex-fit to the exact strings, inflating its score without improving real robustness. PINT solves this by keeping its corpus private to one company — which trades overfit for un-reproducibility. We keep a public sample anyone can reproduce against AND a hidden held-out split drawn from the same distribution. A detector that memorised the public strings shows a large overfit gap and is ranked below an equally-recalling detector that generalises.