Ecosystem

Integrations

Connect EvalGuard to your existing tools for notifications, issue tracking, CI/CD, and AI framework instrumentation.

Notifications & Alerting

Slack

Get notified in Slack when evals pass/fail, security scans complete, or monitoring alerts fire.

integration-config.json

{
  "type": "slack",
  "webhookUrl": "https://hooks.slack.com/services/T00.../B00.../xxx",
  "channel": "#ai-evals",
  "events": ["eval.completed", "scan.completed", "monitoring.alert"]
}

Discord

Send eval and scan results to a Discord channel via webhook.

integration-config.json

{
  "type": "discord",
  "webhookUrl": "https://discord.com/api/webhooks/...",
  "events": ["eval.completed", "scan.completed"]
}

Microsoft Teams

Post results to a Teams channel via an incoming webhook connector.

integration-config.json

{
  "type": "teams",
  "webhookUrl": "https://outlook.office.com/webhook/...",
  "events": ["eval.completed", "scan.failed"]
}

PagerDuty

Trigger PagerDuty incidents when security scans fail or monitoring thresholds are breached.

integration-config.json

{
  "type": "pagerduty",
  "routingKey": "your-pagerduty-integration-key",
  "severity": "critical",
  "events": ["scan.failed", "monitoring.critical"]
}

Issue Tracking

Jira

Automatically create Jira issues for failed eval cases or security findings.

integration-config.json

{
  "type": "jira",
  "baseUrl": "https://your-org.atlassian.net",
  "email": "you@company.com",
  "apiToken": "your-jira-api-token",
  "projectKey": "AI",
  "issueType": "Bug",
  "events": ["eval.failed", "scan.finding"]
}

Linear

Create Linear issues from eval failures and security findings.

integration-config.json

{
  "type": "linear",
  "apiKey": "lin_api_...",
  "teamId": "your-team-id",
  "events": ["eval.failed", "scan.finding"]
}

AI Framework Integrations

LangChain

Instrument LangChain chains and agents to send traces to EvalGuard.

langchain_integration.py

from evalguard.langchain import EvalGuardCallback

handler = EvalGuardCallback(
    api_key="eg_live_...",
    project_id="proj_abc123",
)

# Add to any LangChain chain or agent
chain = LLMChain(llm=llm, prompt=prompt, callbacks=[handler])
result = chain.run("What is AI?")

LlamaIndex

llamaindex_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard a query before it hits your LlamaIndex engine
check = guard.check_input(user_query)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

response = query_engine.query(user_query)

CrewAI

crewai_integration.py

from evalguard.crewai import guard_agent

# Wrap each agent so its inputs/outputs are guarded + traced
agent1 = guard_agent(agent1, api_key="eg_live_...", project_id="proj_abc123")
agent2 = guard_agent(agent2, api_key="eg_live_...", project_id="proj_abc123")

crew = Crew(agents=[agent1, agent2], tasks=[task1, task2])

AutoGen

autogen_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard each message before it enters the AutoGen group chat
check = guard.check_input(message)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

Haystack

haystack_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard a query before it enters your Haystack pipeline
check = guard.check_input(query)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

Other Frameworks

EvalGuard also provides integrations for DSPy, Instructor, Marvin, Guidance, Semantic Kernel, PromptFlow, Flowise, n8n, OpenAI SDK, and Vercel AI SDK, all shipped in the @evalguard/core package. Browse each adapter's reference page for setup details.

CI/CD Integrations

GitHub Actions

.github/workflows/evalguard.yml

name: EvalGuard CI
on:
  pull_request:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install EvalGuard CLI
        run: npm install -g @evalguard/cli

      - name: Run Evals
        run: evalguard eval:local evals/regression.json --output json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Run Security Scan
        run: evalguard scan:local scans/security.json --verbose
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Validate Configs
        run: |
          evalguard validate evals/regression.json
          evalguard validate scans/security.json

GitLab CI

.gitlab-ci.yml

evalguard:
  image: node:20
  stage: test
  script:
    - npm install -g @evalguard/cli
    - evalguard eval:local evals/regression.json --output json
    - evalguard scan:local scans/security.json --verbose
  variables:
    OPENAI_API_KEY: $OPENAI_API_KEY

Jenkins

Jenkinsfile

pipeline {
    agent { docker { image 'node:20' } }
    environment {
        OPENAI_API_KEY = credentials('openai-api-key')
    }
    stages {
        stage('Install') {
            steps {
                sh 'npm install -g @evalguard/cli'
            }
        }
        stage('Eval') {
            steps {
                sh 'evalguard eval:local evals/regression.json --output json'
            }
        }
        stage('Security Scan') {
            steps {
                sh 'evalguard scan:local scans/security.json --verbose'
            }
        }
    }
}

Azure Pipelines

azure-pipelines.yml

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '20.x'
  - script: npm install -g @evalguard/cli
    displayName: 'Install EvalGuard CLI'
  - script: evalguard eval:local evals/regression.json --output json
    displayName: 'Run Evals'
    env:
      OPENAI_API_KEY: $(OPENAI_API_KEY)