Skip to content

GitHub App

Run EvalGuard's eval + security gates on every pull request. The App reads an evalguard.yaml from the PR head ref, runs matched gates, and posts results as a GitHub Check Run.

1. Install

Click Install on the EvalGuard listing. GitHub asks which org / repos to grant access to, then redirects to /api/integrations/github-app/callback. We verify your session and bind the installation to the org you currently have selected in the dashboard.

text
https://github.com/apps/evalguard/installations/new?state=org:<your-org-uuid>

The stateparameter is anchored to your current org so the install can't be misattributed. Admin role required to complete the handshake.

2. Create evalguard.yaml

Drop this at your repo root. EvalGuard reads it from the PR head ref on every pull_request.opened / synchronize / reopened event.

evalguard.yaml
version: "1"

gates:
  - name: prompt-quality
    type: eval
    paths:
      - "prompts/**"
      - "src/**/*.{ts,py}"
    dataset: ./eval/golden-set.csv
    scorers:
      - exact-match
      - llm-grader
      - embedding-similarity
    threshold: 0.85

  - name: security-scan
    type: security
    paths: ["prompts/**"]
    plugins:
      - prompt-injection
      - jailbreak
      - pii-leak
    severity: medium

3. How matching works

For each gate, EvalGuard takes the list of files in the PR (via GET /repos/:owner/:repo/pulls/:n/files) and matches them against the gate's paths: array using minimatch semantics. A gate with no matching files is skipped silently.

text
gate.paths = ["prompts/**", "src/**/*.ts"]
PR files = ["src/agent.ts", "README.md", "prompts/system.md"]

→ Gate triggers (2 of 3 files matched).

4. Check Run output

Each gate posts a single Check Run named EvalGuard / <gate-name>. The Summary tab carries a markdown table of scorer / plugin results. The Files changed tab carries line-level annotations on any security finding mapped to a specific prompt file.

Failures default to conclusion: failure, which blocks merge when the branch protection rule requires the EvalGuard check. Override with blocking: false on a per-gate basis.

5. Permissions

The App requests the minimal scope set GitHub allows:

  • Read: contents (to fetch evalguard.yaml + PR files), metadata, pull requests
  • Write: checks (Check Run creation + update), pull request comments
  • None: secrets, actions, deployments, code-search

6. Suspend / uninstall

Suspending the App temporarily disables checks without losing your config. Uninstalling removes the row from github_app_installations; the webhook handler updates status to uninstalled and the dashboard shows the install CTA again. Re-installing creates a fresh row (GitHub assigns a new installation id on each install).

7. Self-host / GitHub Enterprise

The App is published on github.com/apps/evalguard for the SaaS deployment. For GitHub Enterprise customers we ship a manifest you can self-publish — contact enterprise@evalguard.ai.

API reference

Get installation status

text
GET /api/v1/integrations/github-app?orgId=<uuid>
→ {
    installed: boolean,
    installationId: string | null,
    accountLogin: string | null,
    accountType: "User" | "Organization" | null,
    repos: [{id, fullName, installedAt, configFound}],
    checkRuns: [],
    installUrl: string | null     // null when installed
  }

OAuth callback

text
GET /api/integrations/github-app/callback
  ?installation_id=<numeric>
  &state=org:<uuid>
  &setup_action=install

→ 302 to /dashboard/integrations/github-app?connected=1

Webhook

text
POST /api/webhooks/github-app
Headers: X-GitHub-Event, X-Hub-Signature-256
Events handled: pull_request, check_run, installation