Observability: Logs, Metrics and Traces

When a business application slows down or starts throwing errors, the first question is usually "what is going on?" and the second is "why?" Basic monitoring can answer the first: a dashboard shows that response times have risen or a server is busy. Answering the second, especially in systems made of several services, databases and third-party APIs, needs more. Observability is the ability to understand what a system is doing from the outside, using the signals it produces, so you can investigate problems you did not predict in advance. It rests on three main kinds of data, logs, metrics and traces, used together. This guide explains each, how they fit and how to build observability without drowning in data.

What observability means

The OpenTelemetry project describes observability as letting you understand a system from the outside by asking questions about it without knowing its inner workings, and notes that it helps you troubleshoot novel problems, the "unknown unknowns", and answer the question "why is this happening?" (OpenTelemetry: Observability primer). The same primer explains that to ask those questions, an application must be properly instrumented, meaning its code emits signals such as traces, metrics and logs.

Monitoring tells you when known things go wrong. Observability helps you investigate things you did not anticipate.

Logs: what happened

The OpenTelemetry primer describes a log as a timestamped message emitted by services or other components. Logs are the most familiar signal. Make them useful by:

• Using structured formats, such as JSON, so logs can be searched and filtered
• Including context: request ID, user or account ID where appropriate, service name and environment
• Logging meaningful events, such as errors, failed external calls and important state changes
• Using consistent levels, such as info, warning and error
• Avoiding personal data and secrets in log messages

Unstructured, noisy logs are expensive to store and hard to search when it matters.

Metrics: how much and how often

Metrics are numeric measurements over time, such as request counts, error rates, response times, CPU use and queue length. They are cheap to store and ideal for dashboards and alerts. Useful application metrics include:

• Request rate, error rate and response time for each important endpoint
• Queue length and processing delay for background jobs
• Success and failure counts for external API calls
• Business measures, such as orders placed or sign-ups completed

Track response times by percentile as well as averages, because averages can hide the slow experiences some users have.

Traces: where time is spent

Traces follow a single request as it moves through several services. The primer describes a span as a single unit of work or operation, tracking specific operations a request makes. A trace links these spans together, showing which services were called, in what order and how long each took. Traces are especially valuable when:

• A request passes through several services or functions
• Slow responses come from an unknown dependency
• Errors in one service cause failures elsewhere

Without traces, diagnosing latency in a distributed system often means guessing.

Connect the signals

The real power comes from linking logs, metrics and traces. A typical investigation might start with an alert on rising error rates, move to traces showing which service is failing, then jump to the logs for those specific requests. Use a shared request or trace ID across all three so you can move between them easily.

Start with what users experience

Rather than collecting everything, start with questions that matter to users and the business:

• Can customers log in?
• Are orders completing?
• How long do key pages take to load?
• Are background jobs processing on time?

The OpenTelemetry primer describes a service level indicator as a measurement of a service's behaviour that ideally reflects the user's perspective, such as page load speed, and a service level objective as a way of communicating reliability by attaching indicators to business value. Defining a few of these gives monitoring a clear purpose.

Worked example: an observability plan

The table below is a hypothetical starting plan for a business web application with an API, a background worker and a database. It shows how signals map to questions, not a complete design.

QuestionSignalExample measureAlertAre users able to log in?MetricLogin success rateDrops below agreed levelIs checkout fast enough?Metric and traceSlow-end checkout response timeExceeds agreed target for several minutesWhy is checkout slow?TraceTime spent per service and database callNo alert; used for investigationAre jobs keeping up?MetricQueue age of oldest jobOlder than agreed limitWhat went wrong for this order?LogsStructured log entries with order and request IDNo alert; used for investigationAre external APIs failing?Metric and logsFailure count by providerFailure rate rises sharply

Use open standards where possible

Instrumenting code with open standards, such as OpenTelemetry, reduces lock-in to one monitoring vendor and lets you change tools later without rewriting instrumentation. Many libraries and frameworks offer automatic instrumentation, which provides useful traces and metrics with little code.

Alert on symptoms, not every cause

Alerts should tell people about problems that matter, not every fluctuation. Alert on user-facing symptoms, such as high error rates or slow responses on key paths, and use dashboards and traces to find causes. Too many alerts lead to fatigue and ignored warnings. Our guide to workflow monitoring covers alerting on automation failures in the same spirit.

Add health checks

Health check endpoints tell load balancers and orchestrators whether an instance is ready to serve traffic. They complement observability by allowing automatic responses to unhealthy instances. Our guide to health check endpoint design explains what these endpoints should check.

Control cost and volume

Observability data can grow quickly and become expensive. Manage it by:

• Sampling traces rather than keeping every one
• Setting retention periods by data type
• Dropping noisy, low-value logs
• Watching metric cardinality, since labels with many unique values multiply storage

Protect sensitive data

Logs and traces can accidentally capture personal data, tokens or payment details. Review what is recorded, mask sensitive fields and restrict who can access observability tools.

Build the habit

Observability improves through use. After each incident, ask what signal would have shown the problem sooner or explained it faster, and add it. Review dashboards and alerts regularly, removing those nobody uses.

Frequently Asked Questions

What is observability?

It is the ability to understand what a system is doing from the outside, using signals such as logs, metrics and traces, so you can investigate unexpected problems.

What is the difference between monitoring and observability?

Monitoring tracks known conditions and alerts when they occur; observability lets you ask new questions to understand problems you did not anticipate.

What are logs, metrics and traces?

Logs are timestamped messages, metrics are numeric measurements over time and traces follow a request across services as a set of linked operations.

Where should a team start with observability?

With questions that matter to users, such as whether logins and orders succeed, then instrument key paths and alert on user-facing symptoms.

See what your systems are really doing

We instrument applications, design dashboards and alerts and connect logs, metrics and traces. See our custom app development services, or talk to us about observability.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.