GitHub App

1. Install

Click Install on the EvalGuard listing. GitHub asks which org / repos to grant access to, then redirects to /api/integrations/github-app/callback. We verify your session and bind the installation to the org you currently have selected in the dashboard.

https://github.com/apps/evalguard/installations/new?state=org:<your-org-uuid>

The stateparameter is anchored to your current org so the install can't be misattributed. Admin role required to complete the handshake.

2. Create evalguard.yaml

Drop this at your repo root. EvalGuard reads it from the PR head ref on every pull_request.opened / synchronize / reopened event.

version: 1                      # numeric, not the string "1"
project: proj_xxxxxxxx          # your EvalGuard project id (required)
api_key_env: EVALGUARD_API_KEY  # optional; this is the default

evals:                          # top-level key is "evals"
  - name: prompt-quality
    type: eval
    trigger: "src/**/*.ts"      # single glob string per gate
    dataset: golden-set
    scorers:
      - factuality
      - relevance
      - toxicity
    threshold: 0.85

  - name: security-scan
    type: security
    trigger: "prompts/**"
    plugins:
      - prompt-injection
      - jailbreak
      - pii-leak
    max_critical: 0             # fail if any critical finding
    max_high: 5                 # fail if more than 5 high findings

3. How matching works

For each gate, EvalGuard takes the list of files in the PR (via GET /repos/:owner/:repo/pulls/:n/files) and matches them against the gate's single trigger:glob using EvalGuard's built-in glob matcher — it supports *, **, and ? (not brace expansion or full minimatch). A gate whose trigger matches no changed file is skipped silently.

gate.trigger = "src/**/*.ts"
PR files = ["src/agent.ts", "README.md", "prompts/system.md"]

→ Gate triggers (src/agent.ts matched).

4. Check Run output

A single Check Run named EvalGuard AI Eval is posted per PR, aggregating every matched gate into one result. The Summarylists each gate's pass/fail status, and the expanded text carries the per-gate markdown tables of scorer / plugin results. The Files changed tab carries line-level annotations on any security finding mapped to a specific prompt file.

Failures default to conclusion: failure, which blocks merge when the branch protection rule requires the EvalGuard AI Eval check. Merge-blocking is controlled solely by that branch protection rule.

5. Permissions

The App requests the minimal scope set GitHub allows:

• Read: contents (to fetch evalguard.yaml + PR files), metadata, pull requests
• Write: checks (Check Run creation + update), pull request comments
• None: secrets, actions, deployments, code-search

6. Suspend / uninstall

Suspending the App temporarily disables checks without losing your config. Uninstalling removes the row from github_app_installations; the webhook handler updates status to uninstalled and the dashboard shows the install CTA again. Re-installing creates a fresh row (GitHub assigns a new installation id on each install).

7. Self-host / GitHub Enterprise

The App is published on github.com/apps/evalguard for the SaaS deployment. For GitHub Enterprise customers we ship a manifest you can self-publish — contact enterprise@evalguard.ai.

API reference

Get installation status

GET /api/v1/integrations/github-app?orgId=<uuid>
→ {
    installed: boolean,
    installationId: string | null,
    accountLogin: string | null,
    accountType: "User" | "Organization" | null,
    repos: [{id, fullName, installedAt, configFound}],
    checkRuns: [],
    installUrl: string | null     // null when installed
  }

OAuth callback

GET /api/integrations/github-app/callback
  ?installation_id=<numeric>
  &state=org:<uuid>
  &setup_action=install

→ 302 to /dashboard/integrations/github-app?connected=1

Webhook

POST /api/webhooks/github-app
Headers: X-GitHub-Event, X-Hub-Signature-256
Events handled: pull_request, check_run, installation