Verification and Validation

How Verification and Validation Work

Verification and Validation ask different questions about a system.
In Verification, a result is compared with specified requirements.

These can include, for example:

• functional requirements
• technical specifications
• interface descriptions
• data formats
• security requirements
• performance values
• rules and conventions
• expected states or results

For this purpose, it must first be defined what is to be evaluated and which criteria can be used to determine whether the requirement has been met.
Validation, by contrast, considers the intended use.
It asks whether the system is actually suitable under the relevant conditions to fulfill the intended purpose and the needs of the stakeholders.
A system can therefore be successfully verified and still not be successfully validated.

Built Right or Built the Right Thing?

A commonly used simplified distinction is:

Verification: Have we built the system right?

Validation: Have we built the right system?

This wording is helpful as long as it is not understood as a complete definition.
Verification is about obtaining objective evidence that specified requirements or specifications have been met.
Validation is about obtaining objective evidence that the system fulfills its purpose in its intended use.
Validation therefore considers not only the technical implementation, but also the context in which the system is intended to be used.

Evaluation Is More Than Testing

Tests are an important tool for Verification and Validation, but they are not the only one.

Depending on the system and the question, evidence can be obtained, for example, through

• tests
• measurements
• analyses
• inspections
• reviews
• simulations
• demonstrations
• practical trials

Which method is appropriate depends on the statement to be evaluated.
A coding convention may be checked automatically. The behavior of a machine under real operating conditions, by contrast, may require a practical trial.

Requirements Can Also Be Wrong

Successful Verification initially means only that the specified requirements were evaluated.
It does not automatically prove that those requirements were complete or suitable for the intended purpose.
Requirements themselves can therefore also be validated. This examines whether they adequately describe the needs of the stakeholders and the intended use.
A system can do exactly what was specified and still fail to solve the actual problem.
Validation of the system therefore examines whether the realized system actually fulfills its intended purpose under the relevant operating conditions.
Verification, validation of requirements, and validation of the realized system therefore answer different questions:

“Does the result conform to the specified requirements?”
“Do the requirements describe what is actually needed?”
“Does the realized system fulfill its purpose in its intended use?”

Verification and Validation for Variable Systems

Not every system behaves exactly the same way under the same inputs.
This applies, for example, to systems with probabilistic components such as Language Models.
For such systems, evaluation cannot always consist of comparing a single exactly expected output value.

Instead, other verifiable criteria may be necessary, for example:

• permitted and prohibited properties of a result
• statistical evaluations across multiple runs
• reference cases and comparison data
• rules that are evaluated independently of the model
• limits for acceptable deviations
• human evaluation of certain results

The fundamental distinction, however, remains:

A distinction must be made between whether defined criteria are met and whether the overall system is suitable for its intended purpose.

Why Verification and Validation Have Practical Effects

• Requirements become verifiable: Statements about what a system is expected to do can be linked to evidence.
• Errors become visible earlier: Evaluations do not have to take place only at the end of development.
• Implementation and purpose are considered separately: A correct implementation is not automatically a suitable solution.
• Different evaluation methods can be combined: Tests, analyses, measurements, and practical trials provide different kinds of evidence.
• Changes can be evaluated again: Verification and Validation can be repeated throughout the entire life cycle of a system.
• Uncertainty can be taken into account: Systems without fully deterministic behavior can also be examined using suitable criteria.

Typical Example

A company develops a system that automatically transfers incoming orders to an inventory management system.

One requirement states:

A successfully accepted order must be transferred to the inventory management system within five seconds.

Tests can verify whether this requirement is met.
In real operation, however, it becomes apparent that when a connection is interrupted, some orders are not automatically retransmitted. Employees sometimes do not notice these cases until hours later.
The system can therefore meet the evaluated individual requirement and still be unsuitable for the intended business process.
Validation therefore considers the complete intended use and not only compliance with individual technical requirements.

Distinction

• Verification and Validation are not the same as Quality Assurance. Quality Assurance encompasses a broader range of methods and processes used to plan, support, and evaluate the quality of products and work processes. Verification and Validation can be components of it.
• Systems Analysis also has a different task. It examines properties, relationships, and possible alternatives of a system. Its results can provide requirements and criteria that are later verified or validated.
• AI Evaluation uses methods to systematically evaluate properties and behavior of AI systems. Methods of Verification and Validation can play a role in this, but the characteristics of probabilistic systems require additional evaluation methods.

Further Reading

Related Concepts

→ Systems Analysis
→ Systems Architecture
→ Solution Architecture
→ Systems Integration
→ Fault Tolerance
→ Resilience
→ AI Evaluation
→ AI Assurance