The problem: "checkout is slow" and no way to find out why

A user reports that checkout takes a long time. The service has logs, but each line is free text, there are four instances, and a single request touches three other services. You grep for the user's email, find some lines in one instance, nothing in the others, and cannot tell whether the time went into the database, a payment call or a garbage collection pause.

Observability is the ability to ask new questions about a running system from the data it emits, without shipping new code to answer each one. Three kinds of signals cover most of it: logs, metrics and traces. They answer different questions and are strongest when connected.

Logs: what happened, in detail

Logs are discrete events. Their value depends on structure. A line such as Payment failed for order 42 cannot be filtered reliably; a JSON event with fields can.

{"ts":"2026-10-19T09:14:03Z","level":"WARN","service":"checkout", "traceId":"4bf92f3577b34da6a3ce929d0e0e4736","orderId":"42", "event":"payment_declined","reasonCode":"INSUFFICIENT_FUNDS"}

Rules that pay off:

  • Use JSON (or another structured format) in deployed environments and human-readable output locally.
  • Include a correlation id in every line. In Spring Boot with Micrometer Tracing, the trace id and span id are put in the logging MDC, so you add them to the log pattern or JSON encoder rather than passing ids around by hand.
  • Log events and identifiers, not whole payloads. Never log passwords, tokens, card data or unneeded personal data. Treat logs as data that must be retained, access-controlled and deleted on a schedule.
  • Choose levels with intent: ERROR means someone should look, WARN means unusual but handled, INFO records business-significant events. Logging an exception at every layer as it propagates creates duplicates; log it once where it is handled.

Metrics: how the system behaves over time

Metrics are numbers aggregated over time. They are cheap to store, good for dashboards and alerts, and poor at explaining one specific request. Two checklists help you choose what to measure.

  • RED for request-driven services: Rate (requests per second), Errors (failed requests per second or ratio), Duration (latency distribution).
  • USE for resources such as CPU, memory, connection pools and queues: Utilization, Saturation, Errors.

Spring Boot Actuator with Micrometer already exposes HTTP server metrics, JVM memory and GC, thread pools and the HikariCP connection pool. Add business metrics yourself.

@Componentclass PaymentMetrics {    private final Counter declined;    private final Timer gatewayCall;    PaymentMetrics(MeterRegistry registry) {        declined = registry.counter("payments.declined",            "reason", "insufficient_funds");        gatewayCall = registry.timer("payments.gateway.duration");    }    void declined() { declined.increment(); }    <T> T timed(Supplier<T> call) { return gatewayCall.record(call); }}

Two cautions. First, latency is a distribution: an average hides the slow requests users actually feel, so look at percentiles or histograms. Second, cardinality: every unique combination of tag values creates a time series. A tag such as userId or a raw URL with ids can multiply series until the metrics backend struggles. Tags should have a small, bounded set of values.

Traces: where the time went

A trace follows one request across services. It is made of spans: a span has a name, start and end time, attributes, and a parent. The trace id travels between services in request headers (the W3C Trace Context format is the common standard), which is what lets different processes contribute to one picture.

GET /checkout                          [==================== 840 ms]
  ├─ SELECT cart                        [==                   ]
  ├─ POST /payments (payment-service)   [==============       ]
  │    └─ INSERT payment                [===                  ]
  └─ publish OrderPlaced                [=                    ]

A waterfall like this answers the opening question quickly: the time is inside the call to the payment service, not in the database. OpenTelemetry is the vendor-neutral set of APIs, SDKs and a wire protocol for emitting traces, metrics and logs. In Spring Boot 3 the usual path is Micrometer Observation and Micrometer Tracing with an OpenTelemetry bridge, plus an exporter to the backend you choose. Instrumentation for HTTP clients, servers and JDBC can be automatic; add manual spans around important business steps.

Tracing every request can be costly, so most systems sample. Head sampling decides at the start of a request; tail sampling decides after seeing the whole trace, for example to keep all traces with errors. Pick a policy knowing that rare problems are easy to sample away.

How the three connect

The workflow is usually: an alert from a metric tells you something is wrong; a dashboard narrows which service or endpoint; a trace shows where in the request path the time or error is; logs correlated by trace id give the detail. If your logs lack trace ids, or your metrics cannot be tied to a deploy version, that chain breaks. Add the application version and instance as resource attributes so a change in behaviour after a release is visible.

Designing alerts

Alert on symptoms users feel, not on every internal cause. "Error ratio of checkout above the agreed threshold for several minutes" is a symptom. "CPU above a number" often is not: the service may be fine. Causes belong on dashboards for diagnosis.

  • Every alert should be actionable. If nobody needs to do anything, it is noise and will train people to ignore the pager.
  • Link each alert to a short runbook: what it means, how to check, how to mitigate.
  • Define service level objectives (for example an availability or latency target) and alert on how fast you are consuming the error budget, rather than on single spikes.
  • Alert on absence too: a service that sends no metrics at all is not healthy.

Common mistakes

  • Free-text logs without ids.
  • Logging at DEBUG in production permanently, then paying for storage and exposing data.
  • Metrics with unbounded tags.
  • Averages as the only latency number.
  • Dashboards with fifty panels that nobody reads; keep a top-level view tied to RED and drill-downs per service.
  • Alerts on every exception.
  • Treating observability as a task for after the incident instead of part of the feature.

Trade-offs

All telemetry costs money in CPU, network and storage. Sampling, retention periods and log levels are the levers. A small team with a monolith may do well with structured logs, Actuator metrics and basic health alerts, and add tracing when requests start crossing process boundaries. Do not buy complexity ahead of the questions you need to answer.

Checklist

  • JSON logs with trace id, service and version.
  • RED metrics per endpoint, USE metrics for pools and queues.
  • Percentile or histogram latency, bounded tag values.
  • Trace propagation across every service and message hop.
  • Symptom-based alerts with runbooks.
  • No personal data or secrets in telemetry.

Where to go next