Ecosystem

Integrations

Connect EvalGuard to your existing tools for notifications, issue tracking, CI/CD, and AI framework instrumentation.

Notifications & Alerting

Slack

Get notified in Slack when evals pass/fail, security scans complete, or monitoring alerts fire.

integration-config.json

{
  "type": "slack",
  "webhookUrl": "https://hooks.slack.com/services/T00.../B00.../xxx",
  "channel": "#ai-evals",
  "events": ["eval.completed", "scan.completed", "monitoring.alert"]
}

Discord

Send eval and scan results to a Discord channel via webhook.

integration-config.json

{
  "type": "discord",
  "webhookUrl": "https://discord.com/api/webhooks/...",
  "events": ["eval.completed", "scan.completed"]
}

Microsoft Teams

Post results to a Teams channel via an incoming webhook connector.

integration-config.json

{
  "type": "teams",
  "webhookUrl": "https://outlook.office.com/webhook/...",
  "events": ["eval.completed", "scan.failed"]
}

PagerDuty

Trigger PagerDuty incidents when security scans fail or monitoring thresholds are breached.

integration-config.json

{
  "type": "pagerduty",
  "routingKey": "your-pagerduty-integration-key",
  "severity": "critical",
  "events": ["scan.failed", "monitoring.critical"]
}

Issue Tracking

Jira

Automatically create Jira issues for failed eval cases or security findings.

integration-config.json

{
  "type": "jira",
  "baseUrl": "https://your-org.atlassian.net",
  "email": "you@company.com",
  "apiToken": "your-jira-api-token",
  "projectKey": "AI",
  "issueType": "Bug",
  "events": ["eval.failed", "scan.finding"]
}

Linear

Create Linear issues from eval failures and security findings.

integration-config.json

{
  "type": "linear",
  "apiKey": "lin_api_...",
  "teamId": "your-team-id",
  "events": ["eval.failed", "scan.finding"]
}

AI Framework Integrations

LangChain

Instrument LangChain chains and agents to send traces to EvalGuard.

langchain_integration.py

from evalguard.langchain import EvalGuardCallback

handler = EvalGuardCallback(
    api_key="eg_live_...",
    project_id="proj_abc123",
)

# Add to any LangChain chain or agent
chain = LLMChain(llm=llm, prompt=prompt, callbacks=[handler])
result = chain.run("What is AI?")

LlamaIndex

llamaindex_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard a query before it hits your LlamaIndex engine
check = guard.check_input(user_query)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

response = query_engine.query(user_query)

CrewAI

crewai_integration.py

from evalguard.crewai import guard_agent

# Wrap each agent so its inputs/outputs are guarded + traced
agent1 = guard_agent(agent1, api_key="eg_live_...", project_id="proj_abc123")
agent2 = guard_agent(agent2, api_key="eg_live_...", project_id="proj_abc123")

crew = Crew(agents=[agent1, agent2], tasks=[task1, task2])

AutoGen

autogen_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard each message before it enters the AutoGen group chat
check = guard.check_input(message)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

Haystack

haystack_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard a query before it enters your Haystack pipeline
check = guard.check_input(query)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

Other Frameworks

EvalGuard also provides integrations for DSPy, Instructor, Marvin, Guidance, Semantic Kernel, PromptFlow, Flowise, n8n, OpenAI SDK, and Vercel AI SDK. See the integration modules in the core package for details.

CI/CD Integrations

GitHub Actions

.github/workflows/evalguard.yml

name: EvalGuard CI
on:
  pull_request:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install EvalGuard CLI
        run: npm install -g @evalguard/cli

      - name: Run Evals
        run: evalguard eval:local evals/regression.json --output json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Run Security Scan
        run: evalguard scan:local scans/security.json --verbose
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Validate Configs
        run: |
          evalguard validate evals/regression.json
          evalguard validate scans/security.json

GitLab CI

.gitlab-ci.yml

evalguard:
  image: node:20
  stage: test
  script:
    - npm install -g @evalguard/cli
    - evalguard eval:local evals/regression.json --output json
    - evalguard scan:local scans/security.json --verbose
  variables:
    OPENAI_API_KEY: $OPENAI_API_KEY

Jenkins

Jenkinsfile

pipeline {
    agent { docker { image 'node:20' } }
    environment {
        OPENAI_API_KEY = credentials('openai-api-key')
    }
    stages {
        stage('Install') {
            steps {
                sh 'npm install -g @evalguard/cli'
            }
        }
        stage('Eval') {
            steps {
                sh 'evalguard eval:local evals/regression.json --output json'
            }
        }
        stage('Security Scan') {
            steps {
                sh 'evalguard scan:local scans/security.json --verbose'
            }
        }
    }
}

Azure Pipelines

azure-pipelines.yml

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '20.x'
  - script: npm install -g @evalguard/cli
    displayName: 'Install EvalGuard CLI'
  - script: evalguard eval:local evals/regression.json --output json
    displayName: 'Run Evals'
    env:
      OPENAI_API_KEY: $(OPENAI_API_KEY)