Claims with receipts · 2026-08-09

Engineeringdefensibility.

Every number on this page is verifiable from outside the company — link to the file, the commit, or the workflow that produced it.

15-minute reproducibility walkthrough Public SDKs on GitHub

Claims with receipts

Engineering claims with receipts. Every number on this page is verifiable from outside the company — link to the file, the commit, or the workflow that produced it.

Last updated: 2026-08-09. Source of truth:docs/defensibility-roadmap.md.

Doing diligence? The 15-minute walkthrough at /verify walks you through reproducing every claim below — clone, run, see the same numbers. No accounts needed.

Defensibility scoreboard

Earned

Partial

Calendar-bound

Blocked

We track 14 binary success criteria for the “world's best engineering” claim. Each is verifiable from outside the company. Status:

#1. 7+ day green-main streakCalendar
Calendar — passive accrual.
#2. 0 lint warningsPartial
825 → 288 (-65%). Remaining warnings are being cleared incrementally.
#3. < 100 silent skipsEarned
0 silent skips — all 162 documented across 19 files. Commit `5ca0cb5c`.
#4. Critical-path --strict (95% lines / 90% branches)Earned
api-handler 91.7%, crypto 100%, audit 94.1%. Commit `46994c35`.
#5. Mutation score > 85% on 3 critical filesPartial
Scope expanded from 3 to 8 files, all 8 now floored in scripts/critical-path-mutation-ratchet.cjs. crypto 96.55%, audit 89.66%, statistics 80.34%, rule-builder 58.17%, guardrail-dsl 49.35%, ml-classifier 45.69%, api-handler 44.89%, detection-engine 33.93%. 2 of 8 clear the 85% bar; each floor raises as the file improves (ADR-0031).
#6. SOC 2 third-party attestationBlocked
Evidence engine live + gap analysis done; audit process begins Q4 2026 (see /trust/compliance). See `docs/soc2-starter-pack.md`. Not certified, not under audit.
#7. External synthetic uptime checksEarned
Hourly cron in `.github/workflows/synth-check.yml` probing 3 production endpoints from GitHub-hosted runners; staleness is CI-gated by `scripts/synth-check-freshness.cjs`. Run history is grant-gated (the monorepo is private), not public.
#8. Sustained weekly blog cadence (12 posts)Calendar
Volume bar earned (12/12). The sustained-cadence half is passive accrual and is not claimed as earned.
#9. 3+ OSS packages with downloadsEarned
4 client packages public on npm — @evalguard/sdk, @evalguard/openai, @evalguard/anthropic, @evalguard/otel-sdk. 444 combined downloads in the week measured 2026-07-02; check api.npmjs.org for today's figure.
#10. Public head-to-head benchmarksEarned
3 reproducible benchmarks: firewall-latency (request-path p95=3.67ms), firewall-detection-quality (100%/100%/100% on 200-prompt corpus, doubled + sourdough FP closed 2026-05-06), NeMo Guardrails head-to-head (self-run, script published). Commit 8637e975.
#11. OpenAPI spec — 100% public-route coverageEarned
437 documented, 0 missing (100% coverage). 218 allowlisted (internal admin/health/control-plane). 715 OpenAPI operations across 437 paths.
#12. 12+ engineering blog postsEarned
12 published. /blog index.
#13. 30+ ADRs in repoEarned
38 ADRs. `docs/adr/`. ADR-0001 through ADR-0038.
#14. SBOM + security.txt + vulnerability disclosureEarned
Daily syft + grype CycloneDX SBOM. RFC 9116 security.txt. Public disclosure policy.

The numbers

Active CI regression gates

Receipt: scripts/list-ci-gates.cjs — 21 ratchets + 71 hard invariants

Architecture Decision Records

Receipt: docs/adr/

Public client packages on npm

Receipt: registry.npmjs.org — @evalguard/{sdk,openai,anthropic,otel-sdk}

444

Weekly OSS downloads (as of 2026-07-02)

Receipt: api.npmjs.org — re-check for today's number

437 / 437 (100%)

Public API routes documented

Receipt: apps/web/public/openapi.json — 715 operations / 437 paths; 218 allowlisted internal routes

62.32%

Critical-path mutation score (mean of 8 files)

Receipt: scripts/.mutation-score-baseline.json

3.67 ms

Firewall p95 latency, request path (published)

Receipt: packages/core/src/counts.ts — FEATURE_COUNTS.firewallP95Ms, gate-bound to apps/web/src/app/trust/latency/data.json

288

Lint warnings (down from 825)

Receipt: cleared incrementally, never re-baselined

Silent (undocumented) test skips

Receipt: scripts/.skip-baseline.json

Engineering blog posts

Receipt: /blog

Claims with receipts

Customer audits ask specific questions. Here are 8 of them with our specific answers and the file you can read to verify.

Q1. How is multi-tenant data isolated?

Three layers: Postgres RLS on every owned-resource table, every Supabase query chains `.eq("org_id", orgId)` (belt + suspenders), and a static-analysis CI ratchet asserts every owned-table query carries the predicate.

Receipt: ADR-0014 + scripts/cross-tenant-eq-check.cjs (lower-only floor at 137 chains)available on request

Q2. What's your encryption-at-rest story?

AES-256-GCM with a 12-byte random IV per encryption, GCM auth tag stored alongside ciphertext. Implementation 100% line + branch coverage. Key sourced from secrets manager, wiped from process memory after first load.

Receipt: ADR-0008 + apps/web/src/lib/crypto.ts (96.55% mutation score)available on request

Q3. How fast is the firewall?

Request path: p95 = 3.67 ms across the layers that run on it (pattern + signature + token + semantic), re-measured 2026-08-10 on recorded hardware, at a recorded commit, against the compiled bundle. It is the MEDIAN of 5 independent 4,000-run blocks, not a single run: on our measurement box the identical command spans roughly 2.8–7.0 ms depending only on concurrent load, so we publish the estimator and the block spread rather than whichever draw we got. We do not have quiesced reference hardware, and that is the honest limit on this number. The RESPONSE scan is a separate call whose cost scales with the answer's length — a couple of ms on a short reply, a few hundred near the engine's 100,000-character ceiling — published size-bucketed at /trust/latency rather than folded in here. Public SLA p95 ≤ 50 ms on the request path.

Receipt: scripts/firewall-latency-ratchet.cjs + benchmarks/firewall-latency.md

Q4. What does your CI catch?

92 active CI regression gates — 21 monotone ratchets + 71 hard invariants, enumerated in scripts/list-ci-gates.cjs (run it: every gate listed is asserted to exist AND be invoked by a workflow). Hard zero: RLS coverage, force-dynamic, TODO/FIXME, `as any`, dynamic eval, mass-assignment. Hard validation: lint, security.txt freshness, counts-canonical match, --strict critical-path coverage, firewall latency. Lower-only: skip-count (silent: 0), OpenAPI coverage (0 missing), cross-tenant-eq (137 floor), coverage baseline.

Receipt: .github/workflows/ci.yml — Ratchets jobavailable on request

Q5. How do you prevent BOLA / IDOR?

Auth gate in createApiHandler middleware (97.1% line / 91.7% branch coverage). Cross-tenant `.eq("org_id")` on every owned-resource query (Layer 2 of ADR-0014). Static-analysis ratchet enforces zero regressions.

Receipt: apps/web/src/lib/api-handler.ts + ADR-0014available on request

Q6. Audit log integrity?

Every audit row carries an entry_signature column — HMAC-SHA256 of the row's content fields signed with AUDIT_SIGNING_KEY. Verifier (cron + on-demand /api/v1/audit-logs/verify) recomputes and compares constant-time. Versioned column list. Detects insider tampering even from service-role-key compromise.

Receipt: ADR-0023 + apps/web/src/lib/audit-logger.ts (89.66% mutation score)available on request

Q7. How do you prevent SSRF in webhooks?

assertPublicUrl(url) helper. Blocklist: localhost, 127.0.0.1, 0.0.0.0, 169.254.169.254 (AWS/Azure IMDS link-local), metadata.google.internal, 100.100.100.200 (Alibaba). DNS resolution check for private IPv4 / IPv6 link-local + ULA. Protocol whitelist: http/https only. DNS-rebinding defense via re-resolution at fetch time.

Receipt: ADR-0015 + packages/core/src/security/ssrf-guard.tsavailable on request

Q8. Where do design decisions live?

/docs/adr/ — 38 numbered architecture decision records. Each captures status, date, tags, context (forcing function), decision with alternatives considered, consequences (what it makes easy/hard, review triggers, references). PR-reviewed.

Receipt: docs/adr/README.md — ADR-0001 through ADR-0038available on request

Honest gaps

What we don't yet have

Honest gaps. Each is being addressed; none are being hidden.

api-handler.ts mutation score: 44.89% — the load-bearing API middleware has high line + branch coverage (97.1% / 91.7%) but mutation testing reveals 275 surviving mutants, mostly StringLiteral mutations on log messages and switch-case branch labels. Multi-day work to close. Tracked in scripts/.mutation-score-baseline.json.
Detection corpus is only 200 prompts — recall/precision/F1 are 100% on our committed 200-prompt corpus (100 attacks across 7 categories, 100 benign), but 100% on a small self-assembled corpus is a weaker signal than a mid-90s score on a large public one. Expansion to 500+ via AdvBench / HarmBench / AISafetyLab is the open work item.
SOC 2 third-party attestation — gap analysis + control-to-TSC mapping done (leaning Drata), in docs/soc2-starter-pack.md, and the evidence engine is live; the audit process begins Q4 2026 (see /trust/compliance). We do not describe EvalGuard as SOC 2 compliant until an auditor has signed the report.
External pentest — none commissioned yet. Planned post-Type-1 attestation using a HackerOne or Big-4 firm.
Lint warnings: 288 remaining — down from 825 (-65%), cleared incrementally without re-baselining. Multi-session per-file work to reach zero. Tracked in P1.2 task.

Sources of truth

docs/defensibility-roadmap.md — the 14-criterion scoreboard, updated each round.

docs/adr/ — 38 ADRs covering BYOK encryption, cross-tenant defense, audit signing, ratchet discipline, etc.

.github/workflows/ci.yml — 92 active CI regression gates (21 ratchets), each with deliberate-break verification (per ADR-0028). Reproduce with node scripts/list-ci-gates.cjs.

benchmarks/ — public benchmarks with reproducible measurement scripts.

docs/soc2-starter-pack.md — SOC 2 Type 1 calendar, vendor comparison, control map.

Synthetic uptime probe history — hourly Actions probe of 3 production endpoints, run from outside our infra. History readable with a monorepo read grant.

/blog — 12 engineering blog posts covering audit + ratchets + security postmortems.

Found a discrepancy between this page and the underlying receipt? security@evalguard.ai — we'd rather correct it than leave it.

On the superlative

On “world's best engineering”

We use a 14-criterion scoreboard rather than a marketing superlative because superlatives can't be verified. A “world's best” claim is worth what its receipts are worth. The number above (9 of 14 earned) is honest — externally checkable from this repo. We'd rather earn the claim line by line than assert it.