AI Evaluation

How AI Evaluation Works

Before an AI system can be evaluated, it must first be clarified which properties are relevant at all.

The statement:

“The AI should provide good answers.”

is not sufficient for this purpose.

Depending on the intended use, relevant properties could include, for example:

• factual correctness
• completeness
• relevance
• rule compliance
• source grounding
• stability
• robustness to changed inputs
• compliance with a specified format
• functioning generated commands or programs
• appropriate handling of missing information

Such requirements must be turned into criteria that can actually be examined.

First the Question, Then the Metric

A metric is only meaningful if it is known which question it is intended to answer.
For example, a system can comply with the desired output format in 98 percent of cases.
This figure initially says nothing about whether the statements it contains are factually correct.
Likewise, a text can be linguistically very similar to a reference answer and still contain the same factual error.
Evaluation should therefore not begin with an existing metric.

The sequence is more like:

Which property are we interested in?
        ↓
How can this property be observed?
        ↓
What reference or criterion do we need?
        ↓
Which testing procedure is suitable for this?
        ↓
How do we evaluate the result?

Only then does a meaningful measurement emerge.

Not Every Task Has a Single Correct Answer

With traditional software tests, a specific output can often be expected.

For example:

2 + 2 = 4

For many generative AI tasks, by contrast, there is not exactly one permissible character sequence.

For the question:

What is an interface?

several differently worded answers can be factually correct.

An evaluation must therefore distinguish between:

• identical output
• equivalent output
• acceptable output
• incomplete output
• incorrect output

Which differences are permissible depends on the respective task.

References Must Match the Question

Evaluation often requires a basis for comparison.

This can be, for example:

• a documented technical specification
• an actual system state
• a defined set of rules
• a collection of verified reference cases
• measurement data
• expert evaluation by humans
• known correct and incorrect examples

Not every reference has the same evidentiary value.
If the question is whether an OpenBSD command actually works, a second generated text answer is not an equivalent reference to documented behavior or a test on the system in question.
Selecting the reference is therefore itself part of the evaluation design.

Golden Sets and Reference Cases

For recurring evaluations, verified test cases can be assembled.
Such a collection can contain inputs together with known relevant properties or expected results.

For example:

Test case:
  Question: Which option disables function X?

Expected properties:
  - correct option specified
  - applies to version 7.9
  - do not use an option from version 7.8
  - source must be provided

Here, the reference does not necessarily describe a complete model answer.
It describes properties that an acceptable answer must have.
Such reference collections are often referred to as Golden Sets, Golden Datasets, or Evaluation Sets.
They must be maintained.
If the software version changes, for example, a previously correct reference case can also become outdated.

One Run Is an Observation

With a probabilistic system, a single run is initially an observation.
A language model can answer the same test case correctly several times and produce an error on a later run.
An evaluation can therefore include multiple executions of the same or comparable test cases.

The following can then be examined, for example:

• How often is a criterion satisfied?
• Which errors occur repeatedly?
• How much do results differ?
• Are there particular inputs for which errors occur more frequently?
• Does behavior change with different context?
• Does a change degrade other cases that already worked?

This turns:

This test was successful.

into a much more interesting question:

How does the system behave under these conditions?

Automated Checks

Part of an AI Evaluation can be checked deterministically.

Examples include:

• valid JSON
• required fields present
• allowed or forbidden characters
• schema compliance
• successful compilation
• syntactically valid configuration
• required sections present
• compliance with machine-readable rules

Such checks have an advantage:

If the criterion is unambiguously defined, the result can be determined reproducibly.
However, they only test the property for which they were developed.
A syntactically valid command can still be factually incorrect.

Testing Against Real Systems

Some statements can be checked better through actual execution than through further text analysis.

For example, if an AI system generates:

some-command --option

the following can be relevant:

• does the command exist?
• does the option exist?
• does it apply to the version in question?
• does it work in the intended context?
• does it have the claimed effect?

If these questions can be checked on a suitable test system, this produces a different kind of evidence than the statement of another language model.
Real tests also have limits.
A successful test under a particular configuration does not automatically prove behavior under all other conditions.
However, it provides a verifiable observation under documented conditions.

Humans as Evaluators

Not every relevant property can be fully automated.

Humans can, for example, assess:

• whether an explanation is understandable
• whether important relationships are missing
• whether an answer is helpful for the intended user
• whether different technically valid solutions are presented appropriately
• whether a result is practically unsuitable despite formally satisfying the criteria

Human evaluation is also not automatically objective, however.
Different people can reach different evaluations.
Criteria, evaluation guidelines, and comparison examples can therefore also be necessary for human evaluation.

AI Evaluates AI

A language model can be used to evaluate the output of another language model.
Such procedures are often referred to as LLM-as-a-Judge.
They can, for example, be useful for pre-sorting large quantities of results according to specific criteria or examining differences between answers.

However, this creates an additional evaluation layer:

Model A generates result
        ↓
Model B evaluates result

The output of Model B is itself a model result.

It can also:

• evaluate facts incorrectly
• overlook important errors
• prefer plausible statements
• interpret criteria differently
• be influenced by the context provided

An AI evaluation can therefore provide additional evidence.
But it is not automatic proof of truth.

Multiple Testing Procedures Can Complement Each Other

Different evaluation procedures have different strengths.

For example, a technical answer could be checked sequentially through:

structure check
      ↓
rule check
      ↓
source check
      ↓
second expert evaluation
      ↓
test on real system
      ↓
human approval

Not every task requires all of these steps.
What matters is which risks exist and which properties need to be checked at all.
Multiple checks also do not automatically increase quality if they all use the same incorrect assumption.
The independence and evidentiary value of the respective testing procedures are therefore relevant.

Evaluation Requires Documented Conditions

For the results of an evaluation to be interpreted, the conditions must be known.

For a language model, these can include:

• model
• model version
• System Prompt
• user prompt
• provided context
• tools
• external data sources
• generation parameters
• test data
• evaluation procedure
• time of evaluation

If one of these conditions changes, the result can change as well.
Evaluation is therefore not merely the collection of scores.
It describes an observation of a system under specific conditions.

Evaluation Can Itself Become Outdated

An evaluation procedure is not automatically valid forever.
Reference data can become outdated.
Requirements can change.
Models can be updated.
New classes of errors can be discovered.
A test set that covered the relevant problems well yesterday may overlook important new cases tomorrow.
Evaluation must therefore itself be reviewed regularly and developed further when necessary.

Why AI Evaluation Has Practical Effects

• Quality becomes describable: “Looks good” becomes concrete criteria.
• Known errors can be tested again: Problem cases can be retained as test cases and examined again under documented conditions.
• Changes can be compared: New models, prompts, or context strategies can be checked against the same criteria.
• Probabilistic behavior can become visible: Multiple runs can reveal variations and recurring errors.
• Automation is used selectively: Properties that can be checked unambiguously can be controlled automatically.
• Real systems can serve as references: Technical statements do not have to be evaluated exclusively through additional AI outputs.
• Uncertainty remains visible: An evaluation does not have to claim more certainty than the testing procedure actually provides.

Typical Example

An AI system generates a technical article about an operating system.
The first check verifies the structure of the document.
A second check looks for violations of defined editorial rules.
Technical statements are then compared with suitable sources.
Another model independently examines the text for possible contradictions and questionable statements.
Commands and configuration examples are executed, where possible, on a real system of the version in question.
Finally, remaining issues are evaluated by a human.
None of these checks alone proves the complete correctness of the article.
Together, however, they provide different types of evidence from which a substantially more robust evaluation can emerge.

Distinction

• AI Evaluation examines the properties, results, and behavior of an AI system based on defined criteria.
• Verification and Validation are more general Systems Engineering concepts. AI Evaluation can apply Verification and Validation procedures to AI systems and supplement them with evaluation procedures suitable for probabilistic systems.
• Context Engineering concerns which information is made available to an AI system for a task. Evaluation then examines which results arise under these conditions.
• LLM-as-a-Judge is one possible evaluation procedure and is not the same as AI Evaluation as a whole.
• AI Assurance considers more broadly which evidence, processes, and controls are necessary to establish justified trust in the properties and use of an AI system.

Further Reading

Related Concepts

→ Probabilistic
→ Verification and Validation
→ Context Engineering
→ Language Model
→ Observability
→ AI Assurance