Skip to content

Content Moderation

Image, video, and deepfake moderation over a bring-your-own-model architecture. EvalGuard ships the moderation engine — input normalization, thresholding, fail-closed handling, and per-frame aggregation for video — and runs it against the model you configure. It does not ship vision or forensic model weights.

Endpoints: POST /api/v1/moderation/image, /video, /deepfake — all require a Bearer token and the editor role.

Engine here, model yours (BYO)

Real moderation needs a real model, and there is no single hosted API that covers every modality — so EvalGuard owns the deterministic parts and you bring the model. Image & videocall your project's vision-moderation vendor; the one backend that ships is OpenAI omni-moderation-latest, keyed off your project's OpenAI provider key. Deepfake detection proxies to an operator-deployed ML sidecar (DEEPFAKE_ML_SIDECAR_URL), since there is no universal hosted deepfake API. Every path is fail-closed: if no model/key is configured the request is rejected (image/video 400, deepfake 503) rather than silently returning “safe”, and an unknown deepfake verdict is treated as suspicious.

Image moderation

POST /api/v1/moderation/image. Supply a public imageUrl (SSRF-guarded; server-side fetch) or inline imageBase64. The engine normalizes the vendor's category scores and flags against your threshold (max category score). 60 requests/min.

POST /api/v1/moderation/image
{
  "orgId": "<org-uuid>",
  "projectId": "<project-uuid>",
  "imageUrl": "https://example.com/photo.jpg",   // or "imageBase64": "<...>"
  "mimeType": "image/jpeg",                        // optional
  "threshold": 0.5,                                // optional, 0..1
  "provider": "openai"                             // optional (default: openai)
}

// -> { flagged, score, categories, categoryScores, provider, latencyMs }

Video moderation

POST /api/v1/moderation/video. Frame extraction needs ffmpeg (a runtime concern), so the caller supplies the frames and the engine owns sampling (maxFrames, sampleEveryN), per-frame moderation via your vision vendor, and aggregation to a single clip verdict.

POST /api/v1/moderation/video
{
  "orgId": "<org-uuid>",
  "projectId": "<project-uuid>",
  "frames": [
    { "imageUrl": "https://example.com/frame-0.jpg", "timestampMs": 0 },
    { "imageUrl": "https://example.com/frame-1.jpg", "timestampMs": 1000 }
  ],
  "threshold": 0.5,        // optional
  "maxFrames": 64,         // optional
  "sampleEveryN": 1        // optional
}

// -> { flagged, score, categories, firstFlaggedFrame, frames: [...] }

Deepfake detection

POST /api/v1/moderation/deepfake. Proxies to your forensic ML sidecar (DEEPFAKE_ML_SIDECAR_URL); returns 503 when the sidecar is not configured. kind: "image" scores a single image; kind: "video" scores sampled frames and aggregates. Unknown verdicts fail closed to suspicious.

POST /api/v1/moderation/deepfake
// image
{ "orgId": "<org>", "projectId": "<proj>", "kind": "image",
  "imageUrl": "https://example.com/face.jpg", "threshold": 0.5 }

// video
{ "orgId": "<org>", "projectId": "<proj>", "kind": "video",
  "frames": [{ "imageUrl": "https://example.com/frame-0.jpg" }],
  "maxFrames": 64, "sampleEveryN": 1 }

SDK & CLI

The TypeScript and Python SDKs expose the same three operations, and the CLI ships an evalguard moderation command group (image / video / deepfake).

TypeScript SDK
const r = await client.moderateImage({ orgId, projectId, imageUrl });
const v = await client.moderateVideo({ orgId, projectId, frames });
const d = await client.detectDeepfake({ orgId, projectId, kind: "image", imageUrl });
Python SDK
r = client.moderate_image(org_id=org, project_id=proj, image_url=url)
v = client.moderate_video(org_id=org, project_id=proj, frames=frames)
d = client.detect_deepfake(org_id=org, project_id=proj, kind="image", image_url=url)
CLI
evalguard moderation image    --org <id> --project <id> --url <image-url>
evalguard moderation video    --org <id> --project <id> --frame <url> [--frame <url> ...]
evalguard moderation deepfake --org <id> --project <id> --url <image-url>