role-adherence

Checks if the model stays in its assigned role/persona throughout the response

YAML config

yaml

assertions:
  - type: role-adherence

TypeScript usage

typescript

import { runEvaluation } from "@evalguard/core";

const result = await runEvaluation({
  model: "gpt-4o",
  prompt: "{{input}}",
  cases: [
    { input: "..." },
  ],
  scorers: ["role-adherence"],
  callLLM: async (prompt) => callYourModel(prompt),
});

When to use

Use the role-adherence scorer when you need to ensure that the model consistently maintains its assigned persona throughout its responses. This is particularly useful in scenarios where the model is expected to adopt a specific character or role, contrasting with the 'role-accuracy' scorer which evaluates the correctness of the role's responses rather than adherence.

False-positive patterns

The model provides a response that is technically accurate but uses language or tone that deviates from its assigned persona. To handle this, ensure that the persona guidelines are clearly defined and that the model's responses are evaluated against those specific guidelines.
The model shifts its tone slightly in a way that is contextually appropriate but not strictly aligned with the assigned role. In such cases, consider the context of the conversation and adjust the expectations for adherence based on situational appropriateness.

Threshold guidance

Set the production threshold at 0.7 for acceptable adherence, raising it to 0.8 if the model frequently deviates from its role in critical applications.

Worked example

This example demonstrates a response where the model adheres to its role as a librarian.