Ecosystem

Integrations

Connect EvalGuard to your existing tools for notifications, issue tracking, CI/CD, and AI framework instrumentation.

Notifications & Alerting

Each integration is created with POST /api/v1/integrations (org admin) as { orgId, type, config }. config values must be strings, numbers or booleans; there is no per-event filter. What reads each saved integration today: PagerDuty — monitoring and saved-search alerts; Jira — security-finding issue sync (POST /api/v1/integrations/issue-sync); Slack, Discord, Teams — the test send (POST /api/v1/integrations/test); event-driven Slack/webhook messages are configured as automations or playbook actions, each with its own webhook_url; Linear — stored only, no issue creation yet.

Slack

Save a Slack incoming-webhook URL and send a test message. Event notifications to Slack are set up as automations (see above).

integration-config.json

{
  "orgId": "<your-org-id>",
  "type": "slack",
  "config": { "webhook_url": "https://hooks.slack.com/services/T00.../B00.../xxx" }
}

Discord

Save a Discord webhook URL and send a test message.

integration-config.json

{
  "orgId": "<your-org-id>",
  "type": "discord",
  "config": { "webhook_url": "https://discord.com/api/webhooks/..." }
}

Microsoft Teams

Save a Teams incoming-webhook URL and send a test message.

integration-config.json

{
  "orgId": "<your-org-id>",
  "type": "teams",
  "config": { "webhook_url": "https://outlook.office.com/webhook/..." }
}

PagerDuty

Trigger PagerDuty incidents when monitoring alert rules or saved-search alerts fire.

integration-config.json

{
  "orgId": "<your-org-id>",
  "type": "pagerduty",
  "config": { "routing_key": "your-pagerduty-integration-key" }
}

Issue Tracking

Jira

Create and sync Jira issues (type Bug) for security findings via issue sync.

integration-config.json

{
  "orgId": "<your-org-id>",
  "type": "jira",
  "config": {
    "baseUrl": "https://your-org.atlassian.net",
    "email": "you@company.com",
    "apiToken": "REPLACE_ME_jira_api_token",
    "projectKey": "AI"
  }
}

Linear

Save a Linear API key. Issue creation from findings is not wired to Linear yet.

integration-config.json

{
  "orgId": "<your-org-id>",
  "type": "linear",
  "config": { "api_key": "lin_api_..." }
}

AI Framework Integrations

LangChain

Instrument LangChain chains and agents to send traces to EvalGuard.

langchain_integration.py

from evalguard.langchain import EvalGuardCallback

handler = EvalGuardCallback(
    api_key="eg_live_...",
    project_id="proj_abc123",
)

# Add to any LangChain chain or agent
chain = LLMChain(llm=llm, prompt=prompt, callbacks=[handler])
result = chain.run("What is AI?")

LlamaIndex

llamaindex_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard a query before it hits your LlamaIndex engine
check = guard.check_input(user_query)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

response = query_engine.query(user_query)

CrewAI

crewai_integration.py

from evalguard.crewai import guard_agent

# Wrap each agent so its inputs/outputs are guarded + traced
agent1 = guard_agent(agent1, api_key="eg_live_...", project_id="proj_abc123")
agent2 = guard_agent(agent2, api_key="eg_live_...", project_id="proj_abc123")

crew = Crew(agents=[agent1, agent2], tasks=[task1, task2])

AutoGen

autogen_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard each message before it enters the AutoGen group chat
check = guard.check_input(message)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

Haystack

haystack_integration.py

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(api_key="eg_live_...", project_id="proj_abc123")

# Guard a query before it enters your Haystack pipeline
check = guard.check_input(query)
if not check["allowed"]:
    raise ValueError(f"Blocked by EvalGuard: {check['violations']}")

Other Frameworks

EvalGuard also provides integrations for DSPy, Instructor, Marvin, Guidance, Semantic Kernel, PromptFlow, Flowise, n8n, OpenAI SDK, and Vercel AI SDK, all shipped in the @evalguard/core package. Browse each adapter's reference page for setup details.

CI/CD Integrations

GitHub Actions

.github/workflows/evalguard.yml

name: EvalGuard CI
on:
  pull_request:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install EvalGuard CLI
        run: npm install -g @evalguard/cli

      - name: Run Evals
        run: evalguard eval:local evals/regression.json --output json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Run Security Scan
        run: evalguard scan:local scans/security.json --verbose
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Validate Configs
        run: |
          evalguard validate evals/regression.json
          evalguard validate scans/security.json

GitLab CI

.gitlab-ci.yml

evalguard:
  image: node:20
  stage: test
  script:
    - npm install -g @evalguard/cli
    - evalguard eval:local evals/regression.json --output json
    - evalguard scan:local scans/security.json --verbose
  variables:
    OPENAI_API_KEY: $OPENAI_API_KEY

Jenkins

Jenkinsfile

pipeline {
    agent { docker { image 'node:20' } }
    environment {
        OPENAI_API_KEY = credentials('openai-api-key')
    }
    stages {
        stage('Install') {
            steps {
                sh 'npm install -g @evalguard/cli'
            }
        }
        stage('Eval') {
            steps {
                sh 'evalguard eval:local evals/regression.json --output json'
            }
        }
        stage('Security Scan') {
            steps {
                sh 'evalguard scan:local scans/security.json --verbose'
            }
        }
    }
}

Azure Pipelines

azure-pipelines.yml

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '20.x'
  - script: npm install -g @evalguard/cli
    displayName: 'Install EvalGuard CLI'
  - script: evalguard eval:local evals/regression.json --output json
    displayName: 'Run Evals'
    env:
      OPENAI_API_KEY: $(OPENAI_API_KEY)