Observability

How Observability Works

A technical system has many internal states that are not directly visible from the outside.
An application can, for example, remain reachable while individual internal processes are already failing. A service can return correct responses even though its response times are slowly increasing. Two components can have different views of the same state.
Observability means providing enough suitable information to make such processes examinable.

Typical sources of information include:

• metrics
• logs
• traces
• events
• state information
• time information
• information about involved components and dependencies

Which of these are necessary depends on the particular system.
What matters is not the type or amount of data alone, but which conclusions about the actual system behavior can be drawn from it.

From Visible Behavior to Internal State

In a complex system, its internal state often cannot be observed directly.
Instead, external signals are evaluated.
If, for example, a request takes an unusually long time, initially only this increased response time may be visible.
Additional information can show which components were involved in the request, how long individual processing steps took, and where a delay occurred.
Multiple observable pieces of information thus create a model of what probably happened inside the system.
This model can be incomplete.
Observability therefore does not automatically eliminate uncertainty. It improves the information basis on which a state can be examined and evaluated.

Monitoring and Observability

Monitoring and Observability overlap, but they do not describe exactly the same thing.
Monitoring often works with questions that are already known.

Examples include:

• Is the service reachable?
• How high is CPU utilization?
• How much memory is available?
• Does the response time exceed a threshold?
• Is a particular process active?

Specific measurements and alert conditions can be defined for these questions.
Observability becomes particularly important when the question is not yet fully known.

An alert can, for example, report:

“Response times are too high.”

The subsequent question, however, is:

“Why?”

To answer this question, enough information about the system must be available and capable of being related to one another.
Monitoring can therefore detect a known undesirable state. Observability supports the investigation of what is actually happening in the system.

More Data Does Not Automatically Mean More Knowledge

A system can generate millions of log lines and still have poor observability.
If relevant events cannot be clearly assigned to a request or process, relationships remain hidden.
The same applies to metrics.
Thousands of measurements are of little help if the information needed for a specific investigation is unavailable or its meaning remains unknown.
Observability therefore requires selection and structure.

Helpful information can include:

• unique process identifiers
• consistent timestamps
• traceable state transitions
• structured events
• information about causes of errors
• relationships between involved components
• sufficient context to interpret an event

The goal is not maximum data collection.
The goal is a sufficient information basis for relevant investigations.

Observation Changes a System

Observability is not free.
Generating, transmitting, storing, and evaluating additional information requires resources.

Very extensive logging can, for example:

• consume storage space
• generate network traffic
• slow down applications
• expose sensitive information
• require additional systems and dependencies

Observability should therefore already be considered when designing a system.

The question is not:

“How can we record everything?”

But:

“Which information do we need to examine relevant states and processes sufficiently?”

Observability and Deviations

To compare an actual state with an expected state, the actual state must be sufficiently visible.
An observed deviation from the expected state initially provides information about the system.
A response time is higher than yesterday.
A data value differs from a reference value.
A component uses a different path than expected.
An AI system produces different results for comparable inputs.
Only afterward is the significance of this deviation evaluated.

It can:

• be normal
• be tolerable
• require further observation
• indicate an error
• indicate a change in the environment
• require human investigation

Observability thus provides information for decisions. It does not automatically make the decision itself.

Why Observability Has Practical Effects

• Errors can be investigated: Not only the failure itself, but also possible causes become visible.
• Unknown problems become examinable: Not every question has to be known already when the system is designed.
• Dependencies become visible: Behavior can be traced across multiple components.
• Changes can be observed: The effects of technical changes can be examined during actual operation.
• Resilience is supported: Disruptions and recoveries can be detected and evaluated.
• Verification is supported: Actual behavior can be compared with expected properties.
• People receive a basis for decisions: Observed deviations can be evaluated instead of automatically being treated as errors.

Typical Example

An application processes an order through several services.
Monitoring reports that individual orders are taking significantly longer than usual.
The application itself shows no errors.
Using a shared process identifier, however, the order can be traced across multiple involved components.
This reveals that an external service continues to respond successfully but takes significantly longer for certain requests.
Additional state information shows that this causes internal queues to grow and delays subsequent processes.
The original alert only answered the question that something unusual is happening.
The Observability of the overall system makes it possible to investigate what is actually happening and which components are involved.

Distinction

• Observability is not the same as Monitoring. Monitoring often focuses on known measurements, states, and thresholds. Observability more broadly describes the ability to draw conclusions about internal states and behavior of a system from available information.
• Logging is also only one possible component of Observability. Logs alone do not automatically make a system observable.
• Systems Analysis can use observable information to examine a system. Observability itself, by contrast, describes a property of the system and the information it provides.
• For Resilience, Observability is an important foundation: disruptions or changes can only be addressed deliberately if relevant states and deviations are sufficiently recognizable.

Further Reading

Related Concepts

→ Systems Analysis
→ System Architecture
→ System Integration
→ Verification and Validation
→ Fault Tolerance
→ Resilience