EvalGuard for VS Code

Install

The same .vsix installs in VS Code and in its forks — Cursor, Windsurf, Antigravity — because it targets the standard Extension API. Sideloading is the only path in every one of them right now.

# VS Code, Cursor, Windsurf, Antigravity — all take the same file:
code --install-extension evalguard-vscode-1.0.1.vsix

# Or: Extensions panel → "..." menu → "Install from VSIX..."

# Once published (not yet — these fail today):
#   ext install evalguard.evalguard-vscode
#   code --install-extension evalguard.evalguard-vscode

Activation is lazy: the extension wakes on startup, or as soon as your workspace contains an .evalguard config (.evalguard.yml/.yaml/.json) or a *.prompt.md file.

Set your API key

Run EvalGuard: Set API Key from the command palette and paste a key that starts with eg_ (get one at evalguard.ai/settings). It is stored in VS Code's OS-encrypted SecretStorage, not in plaintext settings — any key found in a legacy plaintext slot is migrated into SecretStorage and the plaintext copy cleared on first run. The status-bar shield shows $(shield) EvalGuard when connected and a warning-tinted EvalGuard (disconnected) until a key is set.

The key is only needed for the network-backed features — the active-file scan, eval runs, and the AI quick-fix. The local Scan Workspace needs no key.

Commands

All commands live under the EvalGuard category in the command palette. evalguard.applyFix is internal (invoked by the quick-fix lightbulb) and hidden from the palette.

evalguard.scanWorkspace   Scan Workspace              — offline, no key
evalguard.scanSecurity    Run Security Scan (Active File) — needs key
evalguard.runEval         Run Evaluation              — needs key
evalguard.setApiKey       Set API Key
evalguard.showDashboard   Show Dashboard              — webview
evalguard.openPlayground  Open Prompt Playground      — webview
evalguard.refreshResults  Refresh Results             — reloads the tree

Scan Workspace (offline)

EvalGuard: Scan Workspace walks every workspace folder and runs two engines:

Code scan      CodeScanner.scanDirectory
               → prompt injection, leaked AI keys, unsafe LLM call
                 patterns, SQLi / XSS / command-injection, path
                 traversal, hardcoded secrets, across TS / JS / Python.

Governance     scanRepoFiles / walkRepoFiles
               → risky agent-instruction files: .cursorrules, CLAUDE.md,
                 mcp.json, SKILL.md, system-prompt files — embedded
                 prompt-injection, exfiltration, tool-bypass patterns.

Findings land in the Problems panelon the engine's real line and column (no whole-file squiggles), most-severe first. The panel opens automatically, and each new scan clears the previous run's diagnostics so resolved findings don't linger. The completion toast reports the split, e.g. 12 finding(s) — 9 code, 3 governance.

Active-file scan & AI quick-fix

EvalGuard: Run Security Scan (Active File)sends the current editor's text to the EvalGuard firewall API (POST /security/scan) and marks each returned finding as a diagnostic — critical/high as errors, the rest as warnings.

On any EvalGuard scan finding (source EvalGuard Security or EvalGuard Governance), the lightbulb offers "EvalGuard: suggest fix". It asks the EvalGuard platform LLM (POST /copilot/code-fix) to rewrite just the offending line, then applies it as an editable WorkspaceEdit— review it and press undo to revert. No model-provider key ever lives in the extension; the platform holds it. If the model returns nothing better than the current line, you get "no automated fix available" and no edit is made.

Eval runs, results tree & code lens

EvalGuard: Run Evaluation prompts for a dataset ID, a model (gpt-4o, gpt-4, gpt-3.5-turbo, claude-3.5-sonnet, claude-3-opus, gemini-1.5-pro), and a comma-separated metric list, then calls POST /evals. Any returned metric below 70% is surfaced as a warning diagnostic in the active editor.

The EvalGuard activity-bar container hosts the Eval Results tree, which lists your most recent runs (GET /evals?limit=20) with status icons and drills into per-metric percentages. Refresh Resultsreloads it; without a key it shows a "Configure API key to view results" placeholder.

On *.evalguard.{yml,yaml,json} and *.prompt.md files, code lenses appear above detected prompt: / template: / system_prompt: blocks (Run Eval, Security Scan) and above metric-threshold lines like accuracy: 0.9 (trend link). The showCodeLens setting is declared but not yet wired, so lenses currently always show.

Dashboard & Playground

Show Dashboard opens a webview with placeholder metric cards and a link out to evalguard.ai; the status-bar shield also opens it. Open Prompt Playground opens an in-editor scratchpad for a system prompt and user message with Run Eval / Security Scan buttons. Be aware: today these are lightweight UI shells — the dashboard shows static -- figures and the playground buttons are not yet wired to a message handler. Live data and full round-trips are on the roadmap; for real evals use Run Evaluation and the Eval Results tree.

Settings

evalguard.apiKey          ""                          [wired]
    eg_ key; stored in OS-encrypted SecretStorage, not this field.
evalguard.baseUrl         https://evalguard.ai/api/v1  [wired]
    API base URL; must be HTTPS unless localhost / 127.0.0.1.

evalguard.autoScan        false                        [declared, not yet read]
evalguard.defaultMetrics  [accuracy, toxicity,         [declared, not yet read]
                           relevance]
evalguard.showCodeLens    true                         [declared, not yet read]
evalguard.logLevel        info                         [declared, not yet read]
    enum: off | error | warn | info | debug

Only apiKey and baseUrl change runtime behavior in v1.0.1. The other four are reserved in the manifest for upcoming releases — listed here so the docs match the code, not the other way around.

A typical flow

1. Sideload the .vsix (no registry listing yet, in any editor).
2. Run "EvalGuard: Scan Workspace" — offline, no key needed.
3. Open the Problems panel; findings sit on the exact line/column.
4. On a finding, click the lightbulb → "EvalGuard: suggest fix"
   (this one needs a key — run "EvalGuard: Set API Key" once).
5. Review the single-line edit; undo if you don't like it.
6. Optional: "EvalGuard: Run Evaluation" and watch the Eval
   Results tree for scores.

Agentic / CLI-first editors

Agent-mode and CLI-first tools (Claude Code, Codex, the Cursor/Windsurf agent modes) don't load GUI extensions. They get the same governance through the EvalGuard MCP server (@evalguard/mcp-server), which exposes the same scan engines as MCP tools — evalguard_local_code_scan, evalguard_local_repo_scan, and evalguard_local_ai_bom. Point your agent's MCP config at it and governance runs inline in the agent loop.