task-completion

When to use

Use the task-completion scorer when you need to determine if an agent successfully completed a specific task as assigned. This scorer is particularly useful in scenarios where the output can be clearly defined and checked against the task requirements, unlike the 'task-accuracy' scorer which focuses on the correctness of the output rather than task completion.

False-positive patterns

• An agent is asked to summarize a document but only provides a list of key points. This may be flagged as incomplete, but if the task was to summarize, the output could still be considered valid. To handle this, ensure that the task definition is clear and allows for multiple acceptable formats.
• An agent provides a solution to a problem but does not explicitly state the answer. The scorer may flag this as incomplete, but if the solution is clear and leads to the correct outcome, it should be accepted. To mitigate this, define the expected output format in the task instructions.

Threshold guidance

Set a production threshold of 0.8 for task completion. Raise the threshold if you notice a high rate of false positives, indicating that the task definitions may need refinement.

Worked example

This example evaluates whether the agent successfully completed the task of summarizing a text.

- task: 'Write a summary of the provided text'
  input: 'The quick brown fox jumps over the lazy dog.'
  output: 'A fox jumps over a dog.'
  expected: 'A fox jumps over a dog.'
  result: true

import { runEvaluation } from '@evalguard/core';

const result = await runEvaluation({
  model: 'gpt-4o',
  prompt: '{{input}}',
  cases: [{ input: '...', expectedOutput: '...' }],
  scorers: ['task-completion'],
  callLLM: async (prompt) => callYourModel(prompt),
});

Related scorers