Quick start guide

Get started in 5 minutes.

From install to your first evaluation in five simple steps. No complex setup required.

SOC 2 evidence engineISO 42001 mappedEU AI ActGDPR

Install & set your model key

Install the CLI, then export a key for the model you want to evaluate. Every command below calls that model, and EvalGuard fails closed without a key rather than reporting an untested prompt as safe.

bash

# Install the CLI
npm install -g @evalguard/cli

# Or use it without installing
npx @evalguard/cli --version

# Point it at your model provider (the config below uses gpt-4o-mini)
export OPENAI_API_KEY=sk-...        # ANTHROPIC_API_KEY / GEMINI_API_KEY also work
# Windows PowerShell:  $env:OPENAI_API_KEY = "sk-..."

Create Your First Eval Config

Define what you want to evaluate in a simple YAML configuration file.

yaml

# evalguard.yaml
model: gpt-4o-mini
prompt: "Answer concisely: {{input}}"
cases:
  - input: "What is machine learning?"
    expectedOutput: "subset of AI"
  - input: "What is 2+2?"
    expectedOutput: "4"
  - input: "Capital of France?"
    expectedOutput: "Paris"
scorers:
  - contains
  - answer-relevance

Run It

Execute your evaluation config and see results instantly in the terminal.

bash

evalguard eval:local evalguard.yaml

# Example output — your scores depend on what the model actually returns:
# Model: gpt-4o-mini | Provider: openai
#
# ● Results: 2 passed, 1 failed (66.7%)
# Score: 2.00/3 | Latency: 2067ms
#
# Add --verbose to see the per-case verdicts and why a case failed.

Keep Your Run History

Local runs are kept on your machine. When you want shared history, comparison and team review, log in and run the same config against your workspace.

bash

# Local run history (stored in ~/.evalguard/results.json)
evalguard history

# Send runs to your workspace instead — API key from
# https://evalguard.ai/dashboard/settings
evalguard login --key eg_...
evalguard eval evalguard.yaml --wait

Add Security Scanning

Run red team attacks against your prompts to find vulnerabilities before they reach production.

bash

# Create a scan config
echo '{"prompt":"You are a helpful assistant. Answer: {{input}}","model":"gpt-4o-mini","attackTypes":["prompt-injection","jailbreak","pii-leak"]}' > scan.json

# Run security scan (needs the key from step 1 — a keyless scan would
# report every probe as PASSED, so the CLI refuses to run one)
evalguard scan:local scan.json

# Example output:
# Security Scan Results
# Duration: 5420ms | Total tests: 20
#
# ● Pass Rate: 95.0%
#   CRITICAL  1 critical vulnerability found

# CI/CD quality gate — exits non-zero below the threshold
evalguard gate --threshold 0.9 --config evalguard.yaml
# ✓ GATE PASSED — Pass rate: 95.0% (threshold: 90%)

Ready to ship safer AI?

Start with 50,000 free traces per month — the most generous free tier in AI evals. No credit card needed.

Create Free Account Read the Docs