# EvalGuard

> EvalGuard is an AI evaluation, security-testing, and observability platform for LLM and
> agent applications. Teams use it to run evals, red-team models and agents, guard live
> traffic with an inline LLM firewall, trace agent runs, track spend, and map results to
> governance frameworks (OWASP LLM Top 10, EU AI Act, NIST AI RMF, ISO 42001, SOC 2, HIPAA)
> as audit evidence. It ships an OpenAI-compatible gateway, TypeScript / Python / Go / Java
> SDKs, a CLI, a Terraform provider, and an MCP server.

This file follows the llmstxt.org convention. Every link below is a public documentation
page served anonymously — no account, no API key, no cookie required.

Coding agents: search these docs at `https://evalguard.ai/api/v1/docs/search?q=<query>` (public, no
API key) to self-instrument, or install the MCP server (`@evalguard/mcp-server`) for eval,
security, docs, and trace tools inside your IDE.

## Documentation

- [Documentation](https://evalguard.ai/docs): EvalGuard documentation: API reference, SDK guides, attack plugins, scorers, providers, integrations, self-hosting, and compliance.
- [Agent Graph View](https://evalguard.ai/docs/agent-graph): Node-and-edge DAG of agent execution. Third view mode on the trace detail page, alongside the timeline waterfall and the conversation thread.
- [Agent Memory Governance](https://evalguard.ai/docs/agent-memory-governance): A policy layer over an agent’s durable memory writes — screening every remember / forget / consolidate / rewrite for memory-poisoning, gating autonomous rewrites behind human approval, and sealing each verdict into an append-only, hash-chained decision ledger.
- [API Reference](https://evalguard.ai/docs/api): Most endpoints require Bearer token authentication; a small number of public endpoints (e.g. /demo-eval, /demo-scan, /status/uptime, the jailbreak playground) are unauthenticated.
- [Benchmarks](https://evalguard.ai/docs/benchmarks): 28 of EvalGuard’s 34 benchmark suites detailed here — 17 academic knowledge + reasoning suites and 11 safety/adversarial suites. Each ships a runnable case bank, a deterministic scorer, and reproducible runs.
- [Cost budgets & FinOps](https://evalguard.ai/docs/budget): Cap LLM spend per API key, per team, per run, per project, and per org. Caps are enforced in the gateway proxy: when a key is over budget the proxy returns 402 Payment Required before the upstream provider is ever called.
- [BYOK — bring your own key](https://evalguard.ai/docs/byok): Point EvalGuard at your own LLM-provider API keys. You keep custody of the spend and the credential; we store it encrypted and use it only to make the calls you asked for — gateway proxying, eval generation, red teaming, and moderation.
- [CI/CD Quality Gating](https://evalguard.ai/docs/ci-cd): Block deployments when LLM evaluation scores drop. The evalguard gate CLI command runs an eval, compares the pass rate to a threshold, and exits non-zero when the threshold isn't met — any CI system that respects exit codes will fail the build automatically.
- [CLI Reference](https://evalguard.ai/docs/cli): The 22 core commands (of 102 total — run evalguard --help for the full list) for running evaluations, security scans, and managing your AI testing workflow from the terminal.
- [ClickHouse OLAP layer](https://evalguard.ai/docs/clickhouse): Opt-in dual-write of traces to ClickHouse for fast multi-day aggregations. Postgres stays the OLTP authority; ClickHouse accelerates rollups.
- [Compliance](https://evalguard.ai/docs/compliance): Map your AI system testing to regulatory frameworks. EvalGuard provides automated compliance checks, gap analysis, and remediation guidance.
- [Concepts](https://evalguard.ai/docs/concepts): Mental models you need before reading the API reference — how evaluation modes, thresholds, the regeneration loop, the policy engine, agent checkpoints, red teaming, and the firewall fit together.
- [Dataset Versioning](https://evalguard.ai/docs/dataset-versioning): Immutable per-dataset snapshots. Lock in a frozen case set so re-running an experiment a month later hits the same inputs, bit-for-bit.
- [Fine-tuning](https://evalguard.ai/docs/fine-tuning): Cross-provider fine-tuning ledger. Pin training runs to immutable dataset snapshots for reproducibility; submit + cancel jobs on OpenAI, Vertex, Azure-OpenAI, Fireworks, and Together, and watch progress in one place.
- [AI Gateway](https://evalguard.ai/docs/gateway): OpenAI-compatible HTTP proxy with inline firewall, observability, and FinOps. No SDK changes — change one base URL.
- [Getting Started](https://evalguard.ai/docs/getting-started): Install EvalGuard, run your first evaluation, and connect to your dashboard in under 5 minutes.
- [Go SDK](https://evalguard.ai/docs/go-sdk): 158 methods for Go backends. Covers the core eval / security / gateway / trace surface; the TypeScript SDK has the broadest method coverage.
- [gRPC Gateway](https://evalguard.ai/docs/grpc): Strongly-typed Connect-RPC service over HTTP/1.1 with JSON wire format. Same authoritative gateway logic as the OpenAI-compatible HTTP route; different contract shape for clients that prefer proto-defined services.
- [Install](https://evalguard.ai/docs/install): The canonical install command for every supported language. This is the only page that lists install commands — every other doc, blog, and README links here.
- [Integrations](https://evalguard.ai/docs/integrations): Connect EvalGuard to your existing tools for notifications, issue tracking, CI/CD, and AI framework instrumentation.
- [Java SDK](https://evalguard.ai/docs/java-sdk): A typed JVM client for the EvalGuard /api/v1 surface — evals, security scans, traces, firewall, cost, compliance and prompts — plus a LangChain4j content filter.
- [MCP Vendor Presets](https://evalguard.ai/docs/mcp-presets): One-click registration for 17 popular MCP servers. Each preset bundles the canonical URL, transport, auth shape, and default tool ACLs with sensible risk levels.
- [Migrating from Promptfoo](https://evalguard.ai/docs/migrating-from-promptfoo): EvalGuard is compatible with Promptfoo config files. Most migrations take under 5 minutes.
- [Model scanning](https://evalguard.ai/docs/model-scan): Supply-chain / AI-SPM scanning for model weight files. Static analysis of pickle, PyTorch, and SafeTensors artifacts for deserialization RCE, smuggled payloads, and structural tampering — plus lineage verification and a CycloneDX-ML attestation for the ones that pass.
- [Content Moderation](https://evalguard.ai/docs/moderation): Image, video, and deepfake moderation over a bring-your-own-model architecture. EvalGuard ships the moderation engine — input normalization, thresholding, fail-closed handling, and per-frame aggregation for video — and runs it against the model you configure. It does not ship vision or forensic model weights.
- [OpenTelemetry](https://evalguard.ai/docs/opentelemetry): Point any OTLP/HTTP exporter at EvalGuard. Traces, metrics, and logs ingest — no agent install.
- [Attack Plugins](https://evalguard.ai/docs/plugins): 300+ red team plugins across 30 logical categories, plus 100+ encoding and transformation strategies. 11 of these are safety-taxonomy suites — first-class plugins covering the harm taxonomies from published safety research (AEGIS, BeaverTails, HarmBench, Pliny, ToxicChat, CyberSecEval, UnsafeBench, VLGuard, VLSU, DoNotAnswer, and XSTest), each run on EvalGuard-authored synthetic prompts.
- [Prompt Optimizer](https://evalguard.ai/docs/prompt-optimizer): Closed-loop prompt optimization powered by 4 mutation strategies and meta-LLM scoring. Submit a prompt + an eval dataset and pick a strategy; the optimizer iterates variants, runs each through your evals, and surfaces the highest-scoring prompt — along with a full changelog so you can audit how it got there.
- [Providers](https://evalguard.ai/docs/providers): 77+ first-party LLM providers through a unified interface, plus 800+ models reachable via the gateway-aggregator providers (OpenRouter / LiteLLM / CometAPI). Switch models with one line of config.
- [Python SDK](https://evalguard.ai/docs/python-sdk): 188+ methods covering evals, security, traces, OTLP ingest, Shadow AI, gateway, cost, compliance, and more.
- [Scorers](https://evalguard.ai/docs/scorers): 200+ built-in scorers across 12 categories for evaluating LLM outputs.
- [TypeScript SDK](https://evalguard.ai/docs/sdk): 269 methods covering evals, security, traces, gateway, Shadow AI, AI-SPM, Smart Copilot, cost, compliance, and more. Full TypeScript types included.
- [Self-Hosting](https://evalguard.ai/docs/self-hosting): Deploy EvalGuard on your own infrastructure using Docker Compose or Kubernetes with Helm.
- [Shadow AI discovery](https://evalguard.ai/docs/shadow-ai): Find the unsanctioned AI tools your workforce is already using. Ship egress, SSO, or CASB logs to EvalGuard, and it classifies every domain against a known-AI-tool catalog, rolls sightings up per tool, and ranks them by usage and data risk.
- [SIEM & SOAR integration](https://evalguard.ai/docs/siem): EvalGuard is bidirectional with your SIEM. It exports security events to Splunk, Microsoft Sentinel, and generic webhooks, and it accepts signed inbound SOAR triggers that quarantine a key or open an incident straight from your detection playbook.
- [Skill security scanning](https://evalguard.ai/docs/skill-security): Skills are untrusted third-party code. Their manifest — name, description, and the promptTemplate that is fed to your composing agent verbatim — is the pre-install security boundary. EvalGuard scans that boundary with a deterministic, defense-in-depth pipeline before a skill is ever persisted or served to an installer.
- [Red-team strategies](https://evalguard.ai/docs/strategies): Browse EvalGuard's adversarial transformation strategies — encodings, obfuscations, multi-turn and agentic attacks applied on top of the attack-plugin corpus.
- [Terraform provider](https://evalguard.ai/docs/terraform): Manage EvalGuard projects, API keys, firewall rules, eval schedules, gateway policies, guardrails and agent-memory governance as code. Published on the Terraform Registry as EvalGuardAi/evalguard.
- [Traces & observability](https://evalguard.ai/docs/traces): Every LLM call, tool invocation, retrieval, and agent hop your app makes lands as a span. Spans that share a traceId form one trace — the full record of a single request through your system, top to bottom.
- [EvalGuard for VS Code](https://evalguard.ai/docs/vscode): Inline AI governance in your editor: scan your workspace for LLM-app vulnerabilities and risky agent-instruction files, see findings on the exact line that produced them, and apply AI-suggested fixes — without leaving the editor.

## Concepts

- [Agent checkpoints](https://evalguard.ai/docs/concepts/agent-checkpoints): Three places to insert safety in an agent loop: input injection scan, tool-call gate, tool-result scan. Each catches a different attack class.
- [Evaluation modes — basic vs deep](https://evalguard.ai/docs/concepts/evaluation-modes): Two evaluator depths in EvalGuard: cheap ML-based scorers for high-throughput pre-prod, and LLM-as-judge deep scorers for per-response rigor.
- [Firewall vs scorer](https://evalguard.ai/docs/concepts/firewall-vs-scorer): Sub-3ms inline gate vs LLM-judged eval. When each fires, how they compose, why you want both.
- [Policy engine](https://evalguard.ai/docs/concepts/policy-engine): Declarative rules that map score thresholds and dimension verdicts to actions (block, allow, transform, alert). The control plane between an evaluation and what happens next.
- [Red teaming](https://evalguard.ai/docs/concepts/red-teaming): Evaluators tell you what your model does on the inputs you have. Red teaming tells you what your model does on the inputs you haven’t thought of. The product surface is two orthogonal axes — plugins (what attack class to test) and strategies (how to obfuscate the test).
- [The regeneration loop](https://evalguard.ai/docs/concepts/regeneration-loop): Eval → if-failing-then-regenerate-and-retry. Stop conditions, cost-budget gate, and how the audit row reconstructs the full chain.
- [Scoring thresholds](https://evalguard.ai/docs/concepts/scoring-thresholds): The 0–1 score scale, the 0.8 default threshold, MIN-of-dimensions gate semantics, and how to calibrate thresholds for your domain.

## API

- [API Reference](https://evalguard.ai/docs/api/reference): Auto-generated from the OpenAPI spec on every release.

## SDK

- [@evalguard/core SDK Reference](https://evalguard.ai/docs/sdk/reference): Auto-generated from the TypeScript declaration files on every release.

## Blueprints

- [Autonomous Agent blueprint](https://evalguard.ai/docs/blueprints/autonomous-agent): Scaffold a governed self-improving / autonomous agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on high-risk AND unattended cron/subagent runs, red-team, and OWASP-LLM / MITRE-ATLAS evidence.
- [Coding / SWE Agent blueprint](https://evalguard.ai/docs/blueprints/coding-agent): Scaffold a governed autonomous software-engineering agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop approval on merge/deploy, red-team, and SOC 2 evidence.
- [Computer-Use Agent blueprint](https://evalguard.ai/docs/blueprints/computer-use-agent): Scaffold a governed computer-use / browser-automation agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on every irreversible click, red-team, and OWASP/ATLAS + SOC 2 evidence.
- [Customer-Support Agent blueprint](https://evalguard.ai/docs/blueprints/customer-support-agent): Scaffold a governed customer-support agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on refunds and account changes, red-team, and GDPR / CCPA evidence.
- [Data / Analytics (SQL) Agent blueprint](https://evalguard.ai/docs/blueprints/data-agent): Scaffold a governed enterprise data & analytics agent and ship it: read-only-by-default SQL, gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on writes, red-team, and GDPR / SOC 2 evidence.
- [DevOps / SRE Agent blueprint](https://evalguard.ai/docs/blueprints/devops-agent): Scaffold a governed enterprise DevOps/SRE agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on every prod change, red-team, and SOC 2 evidence.
- [E-commerce Agent blueprint](https://evalguard.ai/docs/blueprints/ecommerce-agent): Scaffold a governed shopper-support agent and ship it: gateway PCI/PII guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on refunds and order changes, red-team, and PCI-DSS / FTC / CCPA / GDPR evidence.
- [Education Agent blueprint](https://evalguard.ai/docs/blueprints/education-agent): Scaffold a governed EdTech tutoring / student-services agent and ship it: gateway student-PII guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on records/grades/minors, red-team, and FERPA/COPPA evidence.
- [Finance Agent blueprint](https://evalguard.ai/docs/blueprints/finance-agent): Scaffold a governed internal finance-ops agent and ship it: gateway PCI/DLP guardrails, per-tool MCP RBAC on Stripe + Postgres, policy rules, human-in-the-loop on money movement, red-team, and SOX/PCI-DSS evidence.
- [Financial Services Agent blueprint](https://evalguard.ai/docs/blueprints/financial-services-agent): Scaffold a governed customer-facing financial-services agent and ship it: gateway PCI/PII guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on money movement, red-team, and SOX/PCI/AML evidence.
- [FinOps Agent blueprint](https://evalguard.ai/docs/blueprints/finops-agent): Scaffold a governed cloud-cost (FinOps) agent and ship it: gateway guardrails, per-tool MCP RBAC on Datadog + cloud cost APIs, policy rules, human-in-the-loop on any resource change or commitment purchase, red-team, and SOC 2 evidence.
- [Governed-Memory Agent blueprint](https://evalguard.ai/docs/blueprints/governed-memory-agent): Scaffold an agent with a persistent, governed memory store and ship it: DLP-scanned writes, per-user/tenant scoping, provenance, poisoning defense, human-in-the-loop consolidation/rewrite/erasure, red-team, and SOC 2 / GDPR erasure evidence.
- [Healthcare Agent blueprint](https://evalguard.ai/docs/blueprints/healthcare-agent): Scaffold a governed patient-support / healthcare-admin agent and ship it: gateway PHI guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on record access, red-team, and HIPAA evidence.
- [HR Agent blueprint](https://evalguard.ai/docs/blueprints/hr-agent): Scaffold a governed internal HR assistant and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on employment-record changes, red-team, and GDPR / India DPDP evidence.
- [Insurance Agent blueprint](https://evalguard.ai/docs/blueprints/insurance-agent): Scaffold a governed insurance member-support / claims-service agent and ship it: gateway PHI guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on claim determinations and record release, red-team, and HIPAA / NAIC / ECOA evidence.
- [IT-Support Agent blueprint](https://evalguard.ai/docs/blueprints/it-support-agent): Scaffold a governed enterprise IT-support agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop approval, red-team, and SOC 2 evidence.
- [Knowledge Assistant blueprint](https://evalguard.ai/docs/blueprints/knowledge-assistant-agent): Scaffold a governed enterprise knowledge assistant and ship it: gateway PII/secret guardrails, per-tool MCP RBAC, entitlement-scoped retrieval, policy rules, human-in-the-loop on writes, red-team, and SOC 2 evidence.
- [Legal-Ops Agent blueprint](https://evalguard.ai/docs/blueprints/legal-agent): Scaffold a governed internal legal-ops assistant and ship it: non-advice guardrails, per-tool MCP RBAC on the document store, policy rules, human-in-the-loop on binding actions, red-team, and GDPR / SOC 2 evidence.
- [Marketing Agent blueprint](https://evalguard.ai/docs/blueprints/marketing-agent): Scaffold a governed marketing content agent and ship it: brand-safety guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop before any send/publish, red-team, and GDPR/CCPA evidence.
- [Pharmacy Agent blueprint](https://evalguard.ai/docs/blueprints/pharmacy-agent): Scaffold a governed retail / mail-order pharmacy agent and ship it: gateway PHI guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on every prescription and controlled-substance action, red-team, and HIPAA / DEA evidence.
- [Real-Estate Agent blueprint](https://evalguard.ai/docs/blueprints/realestate-agent): Scaffold a governed listings / leasing / lending-intake agent and ship it: gateway PII guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on housing and credit decisions, red-team, and Fair Housing / ECOA evidence.
- [Recruiting Agent blueprint](https://evalguard.ai/docs/blueprints/recruiting-agent): Scaffold a governed candidate-screening / recruiting agent and ship it: gateway PII & protected-class-proxy guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on every advance/reject/hire, red-team, and NYC LL144 / EEOC bias-audit evidence.
- [Research Agent blueprint](https://evalguard.ai/docs/blueprints/research-agent): Scaffold a governed deep-research / answer-engine agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on export, source-integrity scoring, and red-team.
- [Sales Agent blueprint](https://evalguard.ai/docs/blueprints/sales-agent): Scaffold a governed enterprise sales agent and ship it: gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop approval on discounts and commitments, red-team, and GDPR/CCPA evidence.
- [Security / SOC Analyst agent blueprint](https://evalguard.ai/docs/blueprints/security-agent): Scaffold a governed SOC analyst agent and ship it: read-only investigation, gateway guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on every response action, red-team, and SOC 2 evidence.
- [Teen-Safety Agent blueprint](https://evalguard.ai/docs/blueprints/teen-safety-agent): Scaffold a governed teen-safety / youth-support agent and ship it: gateway child-safety guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on safeguarding escalations and minor-account actions, red-team, and COPPA / OSA / DSA evidence.
- [Telecom Agent blueprint](https://evalguard.ai/docs/blueprints/telecom-agent): Scaffold a governed carrier / MVNO customer-care agent and ship it: gateway CPNI guardrails, per-tool MCP RBAC, policy rules, human-in-the-loop on SIM/port/plan changes and record access, red-team, and TCPA/CPNI/E911 evidence.

## Integrations

- [GitHub App](https://evalguard.ai/docs/integrations/github-app): Run EvalGuard’s eval + security gates on every pull request. The App reads an evalguard.yaml from the PR head ref, runs matched gates, and posts results as a GitHub Check Run.

## Benchmarks

- [Firewall load benchmark — methodology + published thresholds](https://evalguard.ai/docs/benchmarks/firewall-load): How EvalGuard's inline LLM firewall handles enterprise load. Published thresholds (per-class p95 < 10–50ms, plus a single aggregate p99 < 50ms ceiling across 7 input classes), full k6 methodology, reproducible against your own environment.

## Compliance

- [Evidence Engine](https://evalguard.ai/docs/compliance/evidence): How EvalGuard generates, hashes, chains, signs, and exports audit-ready compliance evidence — from live control checks to a tamper-evident audit log an auditor can verify offline.

## Self Hosting

- [VPC Deployment](https://evalguard.ai/docs/self-hosting/vpc): Run EvalGuard inside an isolated AWS VPC, GCP custom-mode network, or Azure VNet. Optionally publish via AWS PrivateLink, GCP Private Service Connect, or Azure Private Link Service so customers reach EvalGuard without traversing the public internet.

## Machine-readable

- [OpenAPI specification](https://evalguard.ai/openapi.json): Machine-readable OpenAPI 3.0 description of every REST endpoint, with request and response schemas.
- [Canonical feature counts](https://evalguard.ai/canonical-counts.json): Live counts of scorers, attack plugins, red-team strategies, providers, and compliance frameworks, derived from the registries at build time.
- [Docs search (public API)](https://evalguard.ai/api/v1/docs/search?q=run+an+eval): Keyword search across the API reference and the public guides. No API key and no account required, so an agent can search it before it has any credentials of its own.
- [Sitemap](https://evalguard.ai/sitemap.xml): Every public page on evalguard.ai.

## Packages

- [MCP server (npm)](https://www.npmjs.com/package/@evalguard/mcp-server): Exposes eval, security, docs-search, and trace-debug tools over the Model Context Protocol (stdio and HTTP/SSE). The local code-scan, repo/agent-instruction-scan, and AI-BOM tools need no API key.

## Agent instructions

You are reading EvalGuard's documentation index. To answer a question about EvalGuard:

1. Pick the page above whose description matches the task and fetch it at its listed URL.
2. For anything about the REST API — endpoints, parameters, request or response shapes —
   fetch https://evalguard.ai/openapi.json rather than a documentation page. It is the generated
   source of truth and is always current with the deployed API.
3. Every EvalGuard REST endpoint authenticates with `Authorization: Bearer <EVALGUARD_API_KEY>`.
   A user creates a key in the dashboard under Settings → API Keys. Never invent a key, and
   never put one in source control.
4. This index is regenerated from the documentation tree on every deploy, so re-fetch it
   rather than caching a copy indefinitely.

There is no `/llms-full.txt`. EvalGuard's documentation pages are React components, not
Markdown files, so there is no full-text Markdown source to concatenate; a machine-generated
text dump of the page sources would be lower fidelity than the rendered pages this file
already points you at.