The four signals are not at the same maturity
An incident is not a reason to open the log search box by reflex. If you do not know how far the errors have spread across the service, metrics come first. If one request failed while crossing several services, a trace is faster. If you need to know what state the application was in when the exception fired, you need logs, and if you are looking for the function that actually consumed CPU or memory, a profile is closer to the answer.
So the first signal is chosen by "what shape of failure is this symptom", not by "what does our tooling display well". This article does not redefine what counts as user failure; that belongs to user journey and SLI design. It picks up after a failure has been detected, and covers where to start among logs, metrics, traces, and profiles, and on what condition to move to the next one.
The four signals are not at the same maturity today. In OpenTelemetry, traces are stable across API, SDK, and OTLP. Metrics are stable for the signal, API, data model, and OTLP, while the full SDK and per-language implementations are mixed. Logs are stable for the signal, bridge API, SDK, and OTLP, with per-language appenders and coverage varying.[1]
Profiles differ. They entered public alpha in March 2026, and integration with the collector and the eBPF profiling agent has advanced, but the concepts documentation still carries the alpha marking and OTLP profiles remain in development. The alpha warning says they must not be assumed as a required dependency of a critical production workload.[2] That is not a claim that profiling as a technique is new; it says the specification and its ecosystem interoperability are still maturing.
This difference feeds back into signal choice. Even where a profile answers the question best, if the current runtime and backend do not support the profile type and correlation you need, an existing language profiler or another diagnostic path has to be considered alongside it.
The questions each signal answers do not overlap
OpenTelemetry draws a similar split. Metrics show system-level state, logs show individual events, traces show the distributed request path, and profiles show the code that consumed resources.[1] Metrics aggregate many measurements into interval statistics. Traces connect the lifecycle and context of one request.
The right-hand column of the table below matters most. Digging further into a signal for a question it cannot answer stretches the investigation and makes an incidental correlation easy to mistake for a cause.
Signal | Answers well | Answers poorly |
|---|---|---|
| Metrics | When did it start. How many requests, instances, or users are affected. How did error rate, latency distribution, queue depth, CPU, and memory change | Why did this one request fail. Which function burned CPU. What was the exact business state at the moment of failure |
| Logs | What events occurred. Which exceptions, input classes, state transitions, and retry outcomes were recorded | What share of the whole is affected. What is the causal path across services. Whether an unrecorded event truly never happened |
| Traces | Which services, databases, and external APIs one request crossed. Which span failed or stretched. How async work links back to the original request | Which function consumed CPU or heap. How common the problem is across all traffic. What happened inside uninstrumented state |
| Profiles | Which functions and call stacks consumed CPU, heap, allocation, and lock or I/O wait | How wide the user impact is. Which external services the request crossed. Whether the current environment supports the profile type you need |
Operations connected from observation to improvement — Respond using service indicators and alerts, then connect change records and incident reviews to improvements.
The failure type decides the first signal
The first signal in the table below is not a fixed answer for every environment. It is the default starting point that narrows the search fastest given only the symptom you already have.
Flattening this matrix into a single staircase of metrics → logs → traces → profiles brings the problem back. When one specific user request failed, the trace beats sweeping all metrics. Conversely, when the only input is "the service has felt slow since this morning", looking at the latency distribution and the affected services beats opening one arbitrary trace.
Even within user-visible errors, a single event and a population shift start in different places. Given a concrete identifier such as "payment request ABC failed", it is faster to find which span raised the first error, then move to logs carrying the same trace_id and span_id to see the exception and business state at that moment. But "we have more payment errors than usual" is an aggregate question. Narrow it in metrics first — when the error rate rose, and whether it concentrated on a route, region, deployment version, or instance — and only then pick a representative failing request to trace.
Failure type | Symptom in hand | First signal | Next signal | Condition for moving on |
|---|---|---|---|---|
| User-visible error | A specific request or journey failed | Trace | Logs → Metrics | With no request ID and only an error-rate spike, size the scope in metrics first |
| Latency | The service, or part of it, is slow | Metrics | Traces → Profiles or Logs | If you already hold a slow request's trace ID, start there. If the time sits inside one process, move to a profile |
| Resource saturation | CPU, memory, threads, connections, or queues near their ceiling | Metrics | Profiles → Logs or Traces | Use a profile for the consuming code; use traces and logs for requests that failed as a result |
| Intermittent failure | Only some requests fail, irregularly | Trace or structured logs | Logs ↔ Traces → Metrics | If the failed request left no trace, start from logs carrying a correlation ID |
| External dependency | A remote API, database, queue, or DNS is suspect | Trace | Dependency metrics → Logs | If the trace does not show the remote boundary, check client instrumentation and context propagation first |
| Distributed request failure | One request across several services broke in the middle | Trace | Logs of the failing span → Metrics | Separate the service that logged the last error from the service that caused the first one |
| Code hotspot | Specific code is suspected of being slow or CPU-expensive | Profile | Trace → Metrics | Traces connect it to the user path; metrics size the total impact |
| Memory / CPU anomaly | CPU spike, memory growth, OOM, GC pressure | Metrics | Profiles → Logs → Traces | Confirm profile type support. OOM, restart, and GC events live in logs; affected requests in traces |
| Background job | Queue backlog, processing delay, duplicated, missing, or failed jobs | Metrics | Traces / span links → Logs | With a job ID in hand you may start from logs or a trace. Without producer-consumer context the original request is hard to find |
| Unknown symptom | Nothing beyond "the service feels wrong" | Metrics | Trace or Logs → Profiles if needed | Metrics do not confirm a cause; they are the map for choosing the next question |
Latency and saturation narrow by scope, path, then code
When a service has slowed overall, metrics separate four things. Is every request slow or only one route? Did the mean move, or did the upper latency band thicken? Is it concentrated on one service, instance, or deployment version? Did it start at the same moment as a change in CPU, memory, queue, or connection usage?
Once the scope is narrow, the trace shows which span the slow request's time sat in. If an external API call was long, it becomes a dependency problem; if a database span was long, look at the query and connection state. If the long stretch sits inside a single in-process span, that is where a profile earns its place. On-CPU profiles find the functions that burned CPU, off-CPU profiles find where the code waited on locks or I/O, and heap and allocation profiles find the paths that held or repeatedly allocated memory. Supported profile types differ by runtime and profiler.[2]
A trace shows which segment was slow but does not automatically say which function inside it used CPU. A profile is the reverse: it shows function-level consumption, but without metrics and traces it is hard to judge how large a problem that code was across real user requests.
Saturation behaves the same way. CPU at 95%, a connection pool at 100%, or a spiking queue depth are important clues, but none of them proves a first cause. Workers and connections may have been tied up waiting on a slowed external call, or a retry storm may have pushed CPU and queue up together. So for resource saturation, metrics find the saturated resource and the moment first, and the next signal follows the question. Profiles answer who consumed CPU and memory, traces answer which requests held the resource, logs answer whether OOM, restart, GC, or connection timeouts actually occurred, and metrics answer what share of all requests is affected. Distinguishing whether saturation was the cause or the consequence keeps the same incident from repeating, which adding instances on the strength of one chart does not.
Intermittent and distributed failures need one preserved case
Intermittent failures dilute easily in aggregate metrics. If only one user, tenant, input combination, or subset of instances fails, you need an identifier that lets you compare a failing request against a healthy one. If the failed request left a trace, compare its path and span attributes with a healthy one. If there is no trace, find the event in structured logs through stable fields — request ID, job ID, deployment version, instance, result code. Structured here means consistent field names, types, and meanings, not merely JSON output.[1] Only after finding the difference in one case do you return to metrics and compute the rate under the same conditions, to check whether you found a cause or only an incidental property of a handful of requests.
In a distributed system the service that logged the final error is not necessarily the first cause. An upstream service may have timed out waiting on a downstream API, or a downstream retry may have exhausted queues and connections. Traces connect the path a request took across services, databases, and external APIs, with the time spent in each span. OpenTelemetry uses the W3C trace context traceparent as its default propagation format, carrying trace ID and parent span information across service boundaries.[3] Async work is linked back to the original request through span links or producer and consumer spans. Internal context should not be pushed to external services unconditionally, though. At an untrusted boundary, incoming trace headers are validated or stripped, and outgoing context is reviewed so that it does not expose internal structure. Baggage never carries user credentials, API keys, or personal data.
Background work may start after the HTTP response has finished, or be consumed long after the producer created the job, so a single request's parent-child relationship is not enough. When the whole queue is backing up, look first at enqueue, completion, and failure rates, queue depth, oldest job age, retry and dead-letter growth, consumer count and throughput, and whether it concentrates on a job type, tenant, or version. Tracing one job needs producer and consumer spans, span links, and a job ID. Logs carry state transitions and retry reasons, but only with context that ties back to the trace. OpenTelemetry separates the producer span, which creates async work, from the consumer span, which performs it later.[1]
Moving between signals matters more than collecting them
Even with all four signals present, different names and identifiers turn them into four isolated stores during an incident. What you need is a correlation contract rather than a single screen.
Every OpenTelemetry signal can be tied to a resource, and the same resource identity is the anchor for moving between signals emitted by one service, process, or instance. Metrics and traces connect an aggregate's representative measurement to a trace ID and span ID through exemplars. Logs inject the active trace context into the log record.[1] Profiles also aim at both request context and resource context, but they are alpha today, so real end-to-end support has to be confirmed per combination.[2]
Correlation must not be implemented by putting a per-request trace ID into a metric label. Metrics keep separate aggregation state for every unique attribute combination, so attributes whose value space keeps growing — user ID, raw URL, trace ID — inflate cardinality and memory cost sharply. To reach an individual trace, use a linking structure such as an exemplar instead of another metric dimension.
Nor does every signal have to be collected all the time. The goal of observability is not to buy and store four kinds of data exhaustively; it is to answer recurring operational questions at a reasonable cost and risk. Simple work that runs in one process with few external calls is well served by basic metrics and structured logs. A critical user journey that crosses several services and queues, by contrast, takes a long time to attribute without traces. Processing services dominated by CPU and memory cost gain a lot from profiles, but support status, collection overhead, retention, symbolization, and backend compatibility have to be verified first. Six questions decide whether to add a signal. Which incident question can today's data not answer? How often does it arise and how large is the user harm? Does the new signal actually narrow the search? Can you build the context that links it to existing signals? Can you control volume, cardinality, retention cost, and privacy risk? Do the language, runtime, agent, collector, and backend really support what you need? Planting trace_id, resource, and deployment version consistently between two existing signals often comes before a fourth one.
Move | Context required | What it makes possible |
|---|---|---|
| Metrics → Traces | Exemplar, shared resource, time window | Jump from a representative measurement in a latency or error histogram to the actual request trace |
| Traces → Logs | trace_id, span_id, shared timestamp | Jump from a failing span to exception and state logs from the same execution context |
| Resource move across signals | service.name, service.instance.id, environment, version, region | Query other signals emitted by the same service, instance, or deployment |
| Traces → Profiles | Trace and span context, or the same resource and time window | Connect a slow span to the code stack running at that moment |
| Producer → Background job | Span link, job or message ID, producer and consumer context | Connect the original request to async work executed later |
Switch signals when the next question stalls
In practice you do not open every tab in order. You move when the current signal stops answering the next question. The sequence has five steps.
Classify the shape of the symptom first: one request, a population shift, a resource anomaly, or a code cost. Then cut time and scope in the first signal, narrowing down to service, route, instance, job type, dependency, and deployment version. Next, secure a concrete identifier — trace ID, span ID, job ID, resource identity, or a representative exemplar. Then move to the signal that fits the next question: logs for the detail of an event, traces for the request path, profiles for code-level resource use, metrics for rate and blast radius. Finally, leave the evidence the mitigation needs. Do not record only a presumed cause; record which signal and identifier confirmed it, the scope affected, and the deployment version involved.
What decides the speed of an incident analysis is not the number of signals. Metrics show the scope, traces connect the path, logs explain the event, and profiles reveal where the code spent its resources. That order is not a fixed staircase. The first signal always starts from the shape of the failure in front of you.
