Fault Tolerance

How Fault Tolerance Works

Not all possible faults in a system can be reliably prevented.
A system can therefore be designed so that certain faults do not immediately lead to the loss of its required function.

For this purpose, it must first be considered:

• which faults can occur
• which parts of the system are affected by them
• how a fault can be detected
• which effects are acceptable
• which function must be preserved
• how the system should respond to the fault

Only then can suitable technical measures be determined.

Detecting, Containing, and Handling Faults

Fault Tolerance can be achieved in different ways.
One common approach is redundancy. If one component fails, another can take over its function.
Redundancy alone, however, is not sufficient.
The system must, for example, be able to detect that the component previously in use has actually failed. It must decide when to switch to another component and, if necessary, ensure that the replacement is in a suitable state.

Other methods can include:

• retrying a failed operation
• using an alternative communication path
• isolating faulty components
• falling back on stored or previously known data
• temporarily disabling nonessential functions
• using a degraded operating state

Which response is appropriate depends on the particular system and the fault being considered.

Required Function Does Not Always Mean Normal Operation

A fault-tolerant system does not necessarily have to function exactly as before after a fault.
What matters is which function must remain available under the respective conditions.
For example, a system can offer ten different functions during normal operation. During a partial failure, only the three functions that are absolutely necessary for continued operation remain available.
If this behavior is intended and the required function continues to be provided, degraded operation can also be a form of Fault Tolerance.
For a meaningful evaluation of Fault Tolerance, it must therefore be known which function is expected under which conditions.

Not Every Fault Has to Be Prevented

When developing technical systems, it is natural to want to prevent faults as completely as possible.
However, this is not always possible and not always economically sensible.

Instead, it can be examined:

What happens if this fault occurs anyway?

A network can be interrupted.
A disk can fail.
An external service can be unreachable.
A message can arrive twice.
A process can terminate unexpectedly.
A single faulty state does not necessarily have to cause the entire system to fail.
Fault Tolerance therefore concerns the ability of the system to handle such situations in a controlled manner.

Tolerating Does Not Mean Ignoring

A tolerated fault remains a fault.
If a system automatically switches to a replacement component after one component fails, the service can initially continue to operate.

Nevertheless, it may be necessary to:

• log the fault
• notify responsible personnel
• replace the failed component
• check data sets
• investigate the cause of the fault
• restore lost redundancy

Otherwise, a fault that is initially tolerable can later develop into a critical state.
A system with two redundant components may remain functional after one component fails, but it no longer has its original Fault Tolerance.

Fault, Error, and Failure

Terms such as fault, error, and failure are not always used consistently across different technical disciplines.
A common distinction is made between the cause of an erroneous state, an erroneous state within the system, and the externally visible loss of a required function. In English-language technical literature, the terms Fault, Error, and Failure, among others, are used for these concepts.
Not every observed deviation therefore automatically means that the required function of a system has failed.
A deviation can initially be an observation. Only in relation to requirements, system state, and effects can it be assessed whether a response is necessary and whether the required function of the system is at risk or already impaired.
An automatic response to every deviation can itself have undesirable consequences.

Why Fault Tolerance Has Practical Effects

• Individual faults do not have to cause a total failure: Effects can be limited to part of the system.
• Operation can continue in a degraded state: Nonessential functions can be deprioritized in favor of important functions.
• Faults become part of system design: Not only normal operation, but also known fault cases are considered.
• Redundancy can be used deliberately: Additional components are given a defined purpose.
• Fault states become observable: A tolerated fault can still be detected and handled.
• Requirements become more precise: It must be described which function must remain available even under fault conditions.

Typical Example

A system processes messages from a queue.
After successful processing, a message is marked as completed.
During processing, the network connection is interrupted. The system therefore cannot determine with certainty whether the downstream operation has already been completed successfully.
A simple retry could cause the same operation to be performed twice.
In this case, Fault Tolerance therefore does not simply consist of automatically restarting the operation.
The overall system must be designed so that the fault can be detected and handled in a controlled manner. Depending on the task, this can mean, for example, making operations repeatable, using unique operation identifiers, or checking the actual state before retrying.
The appropriate response depends on the specific system.

Distinction

• Fault Tolerance describes the ability of a system to continue providing its required function despite certain faults.
• It is not the same as fault prevention. Fault prevention attempts to prevent faults from occurring. Fault Tolerance concerns the behavior of the system when certain faults occur anyway.
• Availability is also not the same as Fault Tolerance. High Fault Tolerance can contribute to the availability of a system, but availability is additionally influenced by other factors such as repair times and maintainability.
• Resilience considers a broader context. It concerns not only faults considered in advance, but the ability of a system to deal with disruptions and changes, adapt, and preserve or restore its essential functions.

Further Reading

Related Concepts

→ Resilience
→ System Architecture
→ System Integration
→ Verification and Validation
→ Legacy System