/api/v1/generators/rag-auto-evalAuto-generate RAG eval suite
Synthesizes a RAG-evaluation test suite from a knowledge corpus using a real LLM call. `projectId` and at least one `{id, content}` document are required; `casesPerDocument` defaults to 3 and must be 1–20, `questionTypes` is a subset of factual/inferential/comparison/multi-hop/unanswerable/edge_case/adversarial, and total document content is capped at 5 MB (413). Documents are chunked, then per question type the generator produces question + reference-answer pairs with per-case scorer assertions and thresholds drawn from context-relevance, answer-relevance, context-faithfulness, factual-correctness, reasoning-quality, completeness, multi-document-retrieval, hallucination-detection, refusal-accuracy, edge-case-handling and adversarial-robustness. Returns `testCases`, `totalGenerated`, `documentCount`, `chunkCount`, `questionTypeBreakdown`, `duration`, `generatorModel` (default `gpt-4o-mini`) and `keySource`. Uses the org's BYOK OpenAI key, falling back to the platform key; 503 `NO_LLM_KEY` if neither is configured. Rate-limited 10/min.
Authentication
Send Authorization: Bearer YOUR_API_KEY on every request. Generate API keys at /dashboard/settings/api-keys.
Request body required
Example
{
"corpus_id": "00000000-0000-0000-0000-000000000000",
"sample_count": 50
}Schema
{
"application/json": {
"schema": {
"type": "object",
"required": [
"corpus_id"
],
"properties": {
"corpus_id": {
"type": "string",
"format": "uuid"
},
"sample_count": {
"type": "integer",
"default": 50
}
}
}
}
}Response
202 example
{
"job_id": "00000000-0000-0000-0000-000000000000"
}All status codes
Code samples
cURL
curl -X POST \
https://evalguard.ai/api/v1/generators/rag-auto-eval \
-H "Authorization: Bearer $EVALGUARD_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "corpus_id": "00000000-0000-0000-0000-000000000000", "sample_count": 50 }'TypeScript
// The TypeScript SDK (@evalguard/sdk) exposes TYPED methods — runEval,
// getEval, runSecurityScan, checkFirewall, … — not a generic request().
// For an arbitrary endpoint, call it directly:
const res = await fetch("https://evalguard.ai/api/v1/generators/rag-auto-eval", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.EVALGUARD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
"corpus_id": "00000000-0000-0000-0000-000000000000",
"sample_count": 50
}),
});
console.log(res.status, await res.json());Python
# The Python SDK (pip install evalguardai) exposes TYPED methods on
# EvalGuardClient — run_eval, get_eval, … — not a generic request().
# For an arbitrary endpoint, call it directly:
import os
import requests
headers = {"Authorization": f"Bearer {os.environ['EVALGUARD_API_KEY']}"}
headers["Content-Type"] = "application/json"
response = requests.request(
"POST",
"https://evalguard.ai/api/v1/generators/rag-auto-eval",
headers=headers,
json={
"corpus_id": "00000000-0000-0000-0000-000000000000",
"sample_count": 50
},
)
print(response.status_code, response.json())Go
package main
import (
"context"
"fmt"
"net/http"
"os"
"strings"
)
func main() {
body := strings.NewReader(`{"corpus_id":"00000000-0000-0000-0000-000000000000","sample_count":50}`)
req, _ := http.NewRequestWithContext(context.Background(), "POST", "https://evalguard.ai/api/v1/generators/rag-auto-eval", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("EVALGUARD_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil { panic(err) }
defer resp.Body.Close()
fmt.Println(resp.Status)
}