Resilience

How Resilience Works

Technical systems can be exposed to changing conditions during operation.
Hardware can fail, network connections can be interrupted, external services can change, and requirements can evolve.
A resilient system therefore considers more than expected normal operation.
It must also be able to deal with situations in which parts of the system no longer function or behave as originally expected.

This can include, for example:

• detecting disruptions
• limiting impacts
• maintaining essential functions
• switching to alternative components or procedures
• enabling degraded operation
• restoring failed functions
• detecting changed conditions
• making the actual system state observable
• responding to new conditions

Not every resilient system requires all of these capabilities.
Which of them are relevant depends on the particular system and its intended use.

More Than Surviving a Fault

Fault Tolerance concerns continuing to provide a required function despite certain faults.
Resilience considers a broader context.
A system can, for example, tolerate a server failure through a redundant component.
This initially addresses a specific failure case.
However, if the environment changes permanently, a previously reliable external interface behaves differently, or a previously unknown dependency fails, a predefined mechanism may no longer be sufficient.
A resilient system therefore needs more than responses to known failure cases. It also needs ways to make unexpected states visible and respond to them in a controlled manner.

Maintain, Restore, and Adapt

Resilience does not necessarily mean that a system continues to operate completely unchanged during a disruption.
Depending on the situation, a system may require different responses.
It can attempt to maintain its essential function immediately.
It can continue operating with reduced functionality.
It can enter a safe or controlled state.
Or it can initially lose its function and subsequently restore it.
For longer-term changes, adaptation may additionally become necessary.

The decisive question is therefore not only:

“Does the system fail?”

But also:

“What can the system still do during and after a disruption?”

Observability Is Part of It

Unknown or unexpected situations can be addressed more deliberately when relevant impacts or changes of state are observable.
The Observability of a system is therefore important to its Resilience.
It is not always sufficient merely to determine whether individual components are running.

Relevant information can include, for example:

• actual system states
• deviations from expected states
• errors and retries
• unusual data flows
• changed response times
• failed dependencies
• degraded functions
• successful or failed recoveries

Observability does not itself create Resilience.
But it provides information on the basis of which a system or the people responsible for it can respond appropriately.

Not Every Deviation Has to Be Eliminated

An observed deviation from the expected state initially provides information about the system.
It can indicate a fault.
But it can also arise from changed conditions or represent a state that is unproblematic for the system’s essential function.
The appropriate response therefore does not always have to be to restore the previous state immediately.

A deviation can:

• be tolerated
• be observed
• trigger a warning
• trigger an automatic adaptation
• require human evaluation
• make a change to existing rules necessary

A system that automatically suppresses every deviation can itself become inflexible or susceptible to disruption.
Resilience therefore requires not only control, but also an understanding of which deviations are actually relevant.

Adaptation Needs Boundaries

Adaptability does not mean that a system should change its behavior arbitrarily.
An adaptable system also needs boundaries.
Security requirements, data integrity, or other essential properties must not, for example, be abandoned merely because external conditions have changed.

A distinction must therefore be made between:

• immutable requirements
• expected operating ranges
• tolerable deviations
• situations requiring human decision
• states in which continued operation is not responsible

Resilience therefore does not mean unlimited flexibility.
It combines adaptability with controlled boundaries.

Why Resilience Has Practical Effects

• Disruptions are treated as part of operation: More than the ideal normal state is considered.
• Essential functions can be prioritized: Degraded operation can be more useful than complete failure.
• Recovery is considered: What matters is not only the failure, but also the path back to a suitable state.
• Unknown situations become visible: Observability enables responses to events that were not completely anticipated.
• Changes can be taken into account: Systems do not have to assume permanently unchanged conditions.
• Deviations can be evaluated: Not every difference from the expected state has to be eliminated automatically.
• Humans can remain part of the response: Not every decision has to or should be fully automated.

Typical Example

A company uses an external service to process certain data.
During a short outage, new operations are initially buffered locally. Once the service is available again, they are transmitted afterward.
The system thereby tolerates a known failure case.
A few months later, however, the provider changes the behavior of its interface. Requests continue to receive technically successful responses, but in certain cases contain different data than before.
A simple availability test would not detect this change.
A resilient overall system should therefore consider more than the reachability of the interface. It needs ways to observe relevant behavior and results, make deviations visible, and enable an appropriate response.
That response could consist of withholding individual results, using an alternative processing step, or requesting human review.
Only after evaluation must it be decided whether the system itself, its rules, or the integration with the external service should be changed.

Distinction

• Resilience and Fault Tolerance overlap, but are not identical. Fault Tolerance describes the ability to continue providing a required function despite certain faults. Resilience additionally considers a system’s ability to deal with disruptions and changes, maintain or restore essential functions, and, where appropriate, adapt to changed conditions.
• Availability, in turn, describes the extent to which a system is usable at a particular point in time or over a period of time. High availability can be a goal of resilient systems, but is not equivalent to Resilience.
• Robustness is also often defined more narrowly. A robust system can be insensitive to disruptions or changes within certain limits. Resilience additionally includes dealing with impairments and recovery or adaptation afterward.

Further Reading

Related Concepts

→ Fault Tolerance
→ Systems Analysis
→ System Architecture
→ System Integration
→ Verification and Validation
→ Legacy Modernization
→ Digital Sovereignty