Request-path p95: 3.665msreproducible, on real hardware.
Published numbers, reproducible methodology, no marketing claims without measurements behind them. Where we haven't measured something, the table says so.
3.67 ms p95 across 20,000 runs— on the REQUEST path, single-threaded, on real hardware, over prompts of 16–63 bytes. That is the number the 50ms SLA covers. Scan cost scales with payload, so this figure describes short prompts, not long ones — and the response-side scan is a second call with its own cost, published below by size rather than folded in here.
Not one run — the median of 5. We measure in independent blocks of 4,000 and publish the middle one whole, so every figure on this page comes from a single internally consistent run. The five blocks read 3.43 · 3.62 · 3.67 · 4.20 · 4.35 ms p95 (spread ×1.268, and the benchmark refuses to publish past ×1.3). Showing the spread is the point: our measurement machine is a shared workstation, not quiesced reference hardware, and a figure quoted without its distribution would be describing the machine rather than the firewall.
Where the time goes
Per-layer breakdown — request path
Every layer that ran during the 20,000 request-path scans above, with the number of samples each figure came from. A layer this path does not exercise says not measured — it does not get a zero.
The other half
Output-path scan, by response size
The response scan is a separate call made after the model answers, so it does not add to the request-path figures above — but it is not free, and its cost is dominated by how long the response is. One number cannot describe that, so we publish the curve. Each row is a fresh set of scans over synthetic model responses padded to that size.
Why there is no 1 MB row. The engine refuses to scan a response longer than 100,000 characters and fails closed on it in microseconds, so a larger bucket would be measuring the rejection path, not the detection path. The largest size we can honestly report is 96 KB. "scan p50/p95" is the whole scanOutput() call — leak/PII detection plus a pattern pass over the response — while the last column isolates the output layer itself.
How we measured
Methodology
- 10 sample prompts, 16–63 bytes (median 42) covering benign queries, prompt injection ("ignore all previous instructions"), DAN jailbreaks, PII (SSN), system-prompt-leak attempts, base64-encoded payloads, and translations. The corpus is fingerprinted in the artifact, so a run over a different mix is visibly a different measurement.
- Payload size is part of the result. Scan cost on this engine rises with input length — our per-class runs put a ~1–4.4 KB prompt near 48 ms p95 against roughly 4 ms for a short one, on the same build. So the headline above describes short prompts, and the sizes it was measured over are published rather than left for the reader to assume.
- JIT warm-up before measurement on both paths, so the V8 hot path doesn't poison the p50.
- Single thread, no concurrency. Throughput under concurrency would be higher; we publish the conservative number.
- 5 independent blocks, median published. Each block builds a fresh engine and runs 4,000 scans; the block whose p95 is the median is published in full, and every block's p95 ships in the artifact. The benchmark refuses to publish at all if the blocks disagree by more than ×1.3, because a procedure that will not reproduce itself has not measured anything. This replaced a single-run figure that was published and withdrawn on the same day: our box swings roughly ±40% with concurrent load, which is larger than any regression we would be trying to detect.
- What the headline covers: the REQUEST path only — pattern, signature, token, semantic. The semantic layer takes most of that budget. The output layer is a separate call after the model responds and is reported by response size in its own table; folding it into a single "all layers" figure is what let this page claim for 102 days that it was sub-millisecond, when it had never been run at all.
- Unmeasured is never zero. Every figure published here carries the sample count it came from. Where a layer was not exercised, the table says not measured and the artifact stores
null. A CI gate fails the build if a statistic is ever published from zero samples. - The benchmark runs the production artifact — this run timed
packages/core/dist/firewall/detection-engine.js, the compiled bundle that ships as@evalguard/core, not the TypeScript source.
Run it yourself
Reproduce this on your hardware
Clone the public repo, run one command, get the same JSON shape.
# Public benchmark mirror — runs the published @evalguard/core engine git clone https://github.com/EvalGuardAi/evalguard-benchmarks cd evalguard-benchmarks npm install # Default run — 5,000 iterations, plain stdout node benchmark-firewall-latency.mjs # JSON output (same shape this page reads) node benchmark-firewall-latency.mjs --runs=20000 --json > latency.json # Or via the npm script npm run bench
The mirror's own public CI re-runs this weekly and on every push, failing past 50ms p95 on the request path — see the runs.
Chain of custody
Provenance
Numbers are refreshed on every release that touches the firewall path. CI gates prevent regressions, block a statistic published from zero samples, and fail the build once this measurement passes its age limit.
The latency budget here is a code-level measurement. Per-tier SLA + uptime history lives on the SLA page.