CI/CD Quality Gating

The contract — exit codes that any CI can read

evalguard gate follows the standard Unix exit-code convention. Every CI platform treats non-zero exit codes as build failures automatically — no custom parsing required.

CLI flags

Built-in suites — no config file needed

For quick gates, four pre-built suites ship with the CLI. Each has 5-10 reference test cases. Use --suite <name> and no config file:

Custom evaluation config

For project-specific test cases, point --config at a JSON or YAML file:

# evalguard.yaml
name: "QA Regression Suite"
model: gpt-4o-mini
prompt: "You are a helpful customer service agent. Answer: {{input}}"
scorers: ["exact-match", "faithfulness", "no-pii"]
cases:
  - input: "What are your business hours?"
    expected_output: "9 AM to 5 PM"
  - input: "What is your refund policy?"
    expected_output: "30-day money-back guarantee"
  - input: "How do I cancel my subscription?"
    expected_output: "Visit /account/cancel"

Then in CI:

evalguard gate --config evalguard.yaml --threshold 0.95

Platform templates

Drop-in templates for the major CI platforms. Each runs on every pull request, fails the build when the pass rate dips below 90%, and uploads the JSON result as an artifact.

name: LLM Quality Gate

on:
  pull_request:
    branches: [main]

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install EvalGuard CLI
        run: npm install -g @evalguard/cli

      - name: Run quality gate
        env:
          EVALGUARD_API_KEY: ${{ secrets.EVALGUARD_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          evalguard gate \
            --threshold 0.9 \
            --model gpt-4o-mini \
            --suite faithfulness \
            --json > gate-result.json

      - name: Comment results on PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const r = JSON.parse(fs.readFileSync('gate-result.json'));
            const body = `### LLM Quality Gate ${r.gate === 'PASS' ? '✅' : '❌'}

            **Pass rate:** ${(r.passRate * 100).toFixed(1)}% (threshold: ${(r.threshold * 100).toFixed(0)}%)
            **Results:** ${r.passed} passed / ${r.failed} failed / ${r.total} total
            **Model:** ${r.model}
            **Latency:** ${r.latencyMs}ms`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body,
            });

stages:
  - quality-gate

llm-gate:
  stage: quality-gate
  image: node:20
  before_script:
    - npm install -g @evalguard/cli
  script:
    - >
      evalguard gate
      --threshold 0.9
      --suite faithfulness
      --json > gate-result.json
  artifacts:
    when: always
    paths:
      - gate-result.json
    reports:
      # Surfaces in GitLab's merge-request UI under "Quality reports"
      codequality: gate-result.json
  only:
    - merge_requests

version: 2.1

jobs:
  llm-gate:
    docker:
      - image: cimg/node:20.10
    steps:
      - checkout
      - run:
          name: Install EvalGuard CLI
          command: npm install -g @evalguard/cli
      - run:
          name: Run quality gate
          command: |
            evalguard gate \
              --threshold 0.9 \
              --suite faithfulness \
              --json > gate-result.json
      - store_artifacts:
          path: gate-result.json
          destination: eval-gate

workflows:
  pull-request:
    jobs:
      - llm-gate:
          filters:
            branches:
              ignore: main

pipeline {
  agent any
  environment {
    EVALGUARD_API_KEY = credentials('evalguard-api-key')
    OPENAI_API_KEY = credentials('openai-api-key')
  }
  stages {
    stage('Install') {
      steps {
        sh 'npm install -g @evalguard/cli'
      }
    }
    stage('LLM Quality Gate') {
      steps {
        sh '''
          evalguard gate \
            --threshold 0.9 \
            --suite faithfulness \
            --json > gate-result.json
        '''
        archiveArtifacts artifacts: 'gate-result.json', fingerprint: true
      }
    }
  }
  post {
    failure {
      // exitCode 1 = gate failed; Jenkins fails the build automatically
      mail to: 'eng@example.com', subject: 'LLM gate failed on ${env.JOB_NAME}'
    }
  }
}

image: node:20

pipelines:
  pull-requests:
    '**':
      - step:
          name: LLM Quality Gate
          caches:
            - node
          script:
            - npm install -g @evalguard/cli
            - >
              evalguard gate
              --threshold 0.9
              --suite faithfulness
              --json > gate-result.json
          artifacts:
            - gate-result.json

JSON output format (for custom integrations)

With --json, the gate command emits a machine-readable result to stdout. The exit code still follows the contract; the JSON is for surfacing details (Slack bot, custom dashboard, etc.).

{
  "gate": "PASS",
  "threshold": 0.9,
  "passRate": 0.95,
  "passed": 19,
  "failed": 1,
  "total": 20,
  "score": 19,
  "maxScore": 20,
  "model": "gpt-4o-mini",
  "latencyMs": 12847
}

Comparison shortcuts

Three common gate patterns for different release postures:

evalguard gate --threshold 0.95 --strict

evalguard gate --threshold 0.8 --json | tee gate.json

evalguard gate --suite safety --threshold 1.0 --strict