Skip to content

context-relevance

How relevant is each individual retrieved passage to the question?

YAML config

yaml
assertions:
  - type: context-relevance

TypeScript usage

typescript
import { runEvaluation } from "@evalguard/core";

const result = await runEvaluation({
  model: "gpt-4o",
  prompt: "{{input}}",
  cases: [
    { input: "..." },
  ],
  scorers: ["context-relevance"],
  callLLM: async (prompt) => callYourModel(prompt),
});

When to use

Use the context-relevance scorer when you need to evaluate if the retrieved context directly addresses the user's query. This scorer is particularly useful when distinguishing between relevant and irrelevant context, unlike the content-accuracy scorer which focuses on the correctness of the information provided.

False-positive patterns

  • A query asking for 'best practices in AI' retrieves a context discussing 'AI ethics' which, while related, does not directly answer the query. To handle this, ensure the context is tightly aligned with the specific aspects of the query.
  • A query about 'Python programming' retrieves a context about 'Java programming'. This would be flagged as relevant incorrectly. To mitigate this, refine the retrieval process to prioritize context that matches the programming language specified in the query.

Threshold guidance

Set the production threshold at 0.7 for relevance; consider raising it to 0.8 if false positives are frequent, or lowering it to 0.6 if too many relevant contexts are being missed.

Worked example

This example demonstrates how to evaluate the relevance of a context to a specific query using the context-relevance scorer.

yaml
- query: 'What are the benefits of using AI in healthcare?'
  context: 'AI can improve patient outcomes by providing personalized treatment plans.'
  assertion: true
- query: 'What are the benefits of using AI in healthcare?'
  context: 'The history of AI development dates back to the 1950s.'
  assertion: false
typescript
import { runEvaluation } from '@evalguard/core';

const result = await runEvaluation({
  model: 'gpt-4o',
  prompt: '{{input}}',
  cases: [{ input: '...', expectedOutput: '...' }],
  scorers: ['context-relevance'],
  callLLM: async (prompt) => callYourModel(prompt),
});

LLM-generated (llm-gpt-4o) · 2026-05-23