Software Observability from the Lens of Radar and a Black Box

Observability is often a misunderstood and misused term.  It has come to mean nothing and everything at this point.

You can simplify thinking about software observability if you think about it in 2 parts – a Radar and a Black-box

Radar

Radar systems are real-time. They must be able to detect any anomalies, understand related and unrelated signals, handle out-of-order data or loss of signal, and absolutely cannot have a blip. They enable operators to have a live view of their systems and respond immediately.

Definition of Radar

In software observability parlance, metrics & events lend themselves amazingly well to radars (monitoring). They are easy to instrument, store and fast to retrieve and alert. They are must-haves for any business to keep the lights on.

There is ONE purpose of radar systems: To avoid mishaps in the first place.

Understanding Metrics, Events, Logs and Traces - Key Pillars of Observability | Last9

Understanding Metrics, Logs, Events and Traces - the key pillars of observability and their pros and cons for SRE and DevOps teams.

Book a DemoBook a Demo

Black Box

What about black boxes? Before jumping on to 'flight recorders' - I realize everyone imagines them to be black, which they are absolutely not. They are fluorescent in color to ensure they can be traced back easily in case of a mishap!

These black boxes are great at collecting all the data. They can record all signals, a timeline of events, actual user journeys, environments, configurations, and things you never knew existed. Have a large enough storage capacity that spans at least the last three flights!

Definition of Black Box

In software observability parlance, logs & traces are best suited for understanding the sequence of events, user journeys, and code paths. They need lots of storage and are available with a latency of minutes. They are excellent at debugging, performing an RCA, and helping software engineering teams improve the system's design.

The answers from a black box can, at times, mean going back to the drawing board to change the design of the planes itself!

There is one primary purpose of flight recorder systems: To provide all possible details to perform a post-fact or post-incident analysis. RCA.

If you liked this post, you would like my take on Understanding the model of failures and its connection to Software Reliability.