Why is the unit of observation a Run rather than a model call?
An AI agent taking 20 seconds to respond does not mean its model API worked for 20 seconds. The model call may have finished in two seconds, with four spent waiting for retrieval, ten on an external tool timeout and retries, and the rest waiting for human approval.
The opposite also happens. An agent may respond within two seconds after retrieving stale documents, choosing the wrong available tool, or producing a success-shaped answer despite a tool failure. System latency and HTTP status can look normal in all these cases.
Observe one Run from the arrival of a user request until delivery of the final result or termination of the task. Connect retrieval, model calls, tool selection, tool execution, retries, guardrails, external dependencies, costs, and user outcomes within that Run to identify where the problem began.[3]
Storing every raw prompt and completion is not the starting point. Record structure and outcomes first; collect raw content only for a clear need under a protective policy.
Are the OTel GenAI conventions stable now?
As of August 27, 2026, the core OpenTelemetry Semantic Conventions documentation is version 1.44.0.[7] GenAI conventions have moved out of the core documentation into the separate semantic-conventions-genai repository.[6] There, the overall GenAI conventions, model/client spans, agent/framework spans, metrics, and events all have Development status. Language-specific event implementations may not yet exist.
This does not mean avoiding OpenTelemetry. Its conventions provide a useful starting point for aligning operation names such as invoke_agent, plan, retrieval, and execute_tool across runtimes.[1] Two measures are needed: record both Telemetry Schema Version and Instrumentation Version, and normalize experimental attributes into an internal canonical schema before using them in operational dashboards or analysis. Historical and new Runs must remain comparable when conventions change.
Operations connected from observation to improvement — Respond using service indicators and alerts, then connect change records and incident reviews to improvements.
How do Session, Run, Span, and Event differ?
Major agent runtimes use different names for similar structures. OpenAI Agents SDK groups a conversation with group_id, represents a task as a Trace, and records generation, tool, and guardrail work as spans. Google ADK groups a user request within a Session using invocation_id and records model calls, tool calls, and state changes as events. Amazon Bedrock AgentCore places request/response traces under a Session and divides internal processing and external calls into spans. LangSmith calls individual operations Runs, a Run tree a Trace, and a conversation spanning traces a Thread. The table presents a runtime-neutral mapping.
Using the same value for session_id and run_id makes multi-turn problems difficult to analyze. Creating a new Run for every model call scatters one user request across independent operations. Place Runs/Traces under a Session, then retrieval, planning/tool selection, model calls, and tool execution as spans within each Run. Attach HTTP, DB, Queue, and External API spans below tool execution. Keep retry/loop events, guardrail/policy events, human intervention, and user outcomes in that same Run.[2]
Propagate the same trace context or connect with span links when execution crosses asynchronous jobs, queue workers, other agents, or external tools. Otherwise, a trace shows only that the agent invoked a tool, without revealing which external system consumed the time.
Layer | Meaning | Question to answer |
|---|---|---|
| Session | Conversation or task context across multiple turns | Did problems accumulate across requests from the same user? |
| Run·Trace | One user request through its final result | Did the request succeed overall, and how long did it take? |
| Span | An operation with a start and end | Where did retrieval, the model, a tool, or an external API delay or fail? |
| Event | A decision or state change at a point in time | When was a tool selected, why did a retry occur, or when was approval requested? |
What belongs in the Telemetry Model?
The table proposes a runtime-neutral model normalized from current OpenTelemetry GenAI conventions and common runtime structures. It is not OpenTelemetry's finalized Stable Schema. Its seven groups are identity, request, execution spans (model, retrieval, tool, dependency), events, usage_and_cost, outcome, and privacy.
Organizations need not implement every field at once. Capture the following from the outset where possible: session_id, run_id, and trace_id; agent, workflow, prompt, tool, and data-source versions; stage start/end times and status; model-call, tool-call, and retry counts; distinct first and terminal errors; token usage and pricing basis; human intervention and wait duration; technical and user-task success; and Content Capture Mode.
Keep identifiers with continually growing value sets, such as run_id and session_id, in individual traces and logs instead of aggregate metric labels. Unique IDs in metric attributes quickly increase cardinality and storage costs.
Group | Fields | Question answered |
|---|---|---|
| identity | trace_id · run_id · session_id · parent_run_id · workflow_name · workflow_version · agent_name · agent_version · telemetry_schema_version · instrumentation_name · instrumentation_version · environment | Which agent, workflow, and schema versions produced this Run? |
| request | started_at · channel · use_case · turn_number · prompt_template_id · prompt_template_version · input_size_bytes · input_language · input_content_ref | Which entry channel, turn number, and prompt template were used? |
| execution.spans | span_id · parent_span_id · stage(retrieval, plan, model_call, tool_selection, tool_execution, guardrail, human_wait, external_dependency) · sequence_no · attempt_no · loop_iteration · started_at · ended_at · duration_ms · wait_duration_ms · status(ok, error, cancelled, interrupted) · error_type · recovered | Which stages took time or failed, and did they recover? |
| execution.spans.model | provider · requested_model · response_model · time_to_first_chunk_ms · input_tokens · output_tokens · cache_read_tokens · finish_reason | Was the delay provider waiting, generation, or an output limit? |
| execution.spans.retrieval | data_source_id · data_source_version · query_fingerprint · top_k · result_count · document_refs · score_summary | Which data source and version were used, and were results returned? |
| execution.spans.tool | available_toolset_version · selection_source · selected_tool · tool_name · tool_version · tool_call_id · argument_schema_valid · side_effect_class · result_status · argument_content_ref · result_content_ref | Which tools were available, selected, and executed, with what results? |
| execution.spans.dependency | service_name · operation · protocol_status · timeout | Which external dependency was on the critical path? |
| events | event_id · span_id · occurred_at · event_type(tool_selected, retry_scheduled, loop_detected, guardrail_decision, policy_decision, human_approval_requested, human_approved, human_rejected, run_resumed, state_changed, output_delivered) · reason_code · policy_name · policy_version · decision | What decision or state change occurred, and when? |
| usage_and_cost | model_call_count · tool_call_count · retry_count · total_input_tokens · total_output_tokens · estimated_cost · actual_cost · currency · price_book_version · priced_at · cost_is_estimate | What did the Run cost, and is the figure estimated or actual? |
| outcome | terminal_state(completed, partial, failed, cancelled, abandoned, awaiting_human) · technical_success · task_success · final_output_present · user_visible_latency_ms · human_intervention_required · human_wait_duration_ms · user_feedback · user_corrected_result · user_undid_action · business_outcome_ref | Did the user accomplish the intended task? |
| privacy | capture_mode(metadata_only, redacted, external_reference, temporary_debug) · data_classes_detected(pii, secret, customer_data, untrusted_instruction) · redaction_profile_version · content_storage · retention_class · expires_at | What raw content is retained, and for how long? |
Why is retrieval latency alone insufficient?
A Retrieval Span should record which data source was queried and under what conditions, alongside latency. Current OpenTelemetry GenAI conventions support data_source.id, top_k, and retrieved document IDs and scores.[1] Document lists and query text are opt-in, however, and query text may contain sensitive information.
Production defaults can include data-source ID and index/snapshot version, requested top_k, actual document count, de-identified document references and score summaries, filter use and result count, and cache-hit status. Add retrieval latency, empty-result status, derived query length, language and category, and a separate content reference to the raw query only when necessary.
Without copying full documents into spans, you can determine whether results became empty for a source after deployment, the wrong index version was selected, or retrieval succeeded with low relevance scores. Do not directly compare scores from different retrievers or embedding models. Interpret scores alongside data-source and retriever versions because their meaning depends on the retriever and index configuration.
Why separate tool selection from tool execution?
Choosing the wrong tool and choosing the correct tool but failing to execute it require fixes in different places.
For Tool Selection, record the available toolset version or hash, selected tool, and selecting actor: model, plan, rule, or human. Also record the no-tool-selected state, tool_call_id, and whether a guardrail or approval rejected or changed the selection.
For Tool Execution, record tool name and version, the same tool_call_id, argument schema validation, start/end times, status and error_type, and the external service and operation actually invoked. Include absent or partial results, side effects, retries, and duplicate execution.
OpenTelemetry defines tool name, type, call ID, and error type for execute_tool spans.[6] Arguments and results are opt-in and may contain sensitive information. Applications may need to instrument tools missed by automatic instrumentation. Explaining selection does not require retaining the model's raw internal reasoning. Structured records of the available toolset, selected tool, policy decisions, and execution results provide most information needed for operational diagnosis.
Why do retries and loops need separate events?
Current OpenTelemetry model-span conventions recommend one logical operation duration spanning automatic retries for transient problems.[1] This is useful for final latency, but does not automatically expose why each attempt failed.
To diagnose retries, add events or child spans containing attempt_no, retry reason, backoff duration, previous error type, whether the same provider/model/tool was called, whether the previous attempt may have reached the external system, whether recovery followed, and whether this was the terminal failure ending the Run.
Distinguish loops from simple retries. A retry repeats an operation after a transient failure; a loop repeats model and tool calls without enough state progress. Useful signals include model and tool calls per Run, repeated tools with similar arguments, repeated retrieval queries, and repeated state fingerprints. Also track rising tokens and cost without outcome changes, maximum-iteration limits, human intervention, and forced termination. Current OpenTelemetry GenAI metrics propose inference-call and tool-call counts per agent invocation, agent duration, and tool-execution duration.[6]
What should guardrail and policy records contain?
Knowing that a guardrail ran is insufficient. Record where it acted, what it decided, and how the decision affected the final outcome. At minimum, include the check stage (Input, Retrieval, Model Output, Tool Selection, Tool Result), policy/rule name and version, decision (Allow, Block, Modify, Retry, Escalate), violation category, and check latency. Also record escalation to human review, final Run completion, and a de-identified content reference instead of raw content where applicable.
A blocked request may mean a security control worked correctly. Yet if the user cannot understand the reason or complete the task, the user outcome may be failure or partial success. Record Policy Decision separately from User Outcome.
This article covers observation of policy and authorization outcomes. Designing MCP token audiences, token passthrough, tool allowlists, secret boundaries, tenant isolation, and revocation is a separate security-control problem.[4]
Is human intervention latency or normal workflow?
Waiting for approval is neither model nor tool latency. Without separating it, all user-perceived delay is counted as agent performance degradation. Record approval request time, proposed action and tool_call_id, approval/rejection/modification/expiry, role-based reviewer identifier, wait duration, Run resumption time, final outcome after resumption, and whether a human directly corrected the result.
OpenAI Agents SDK's HITL flow likewise interrupts a Run before a tool call, exposes pending approval as an interruption, and resumes the original Run from serialized state after approval or rejection. Record Human Wait as an independent stage in such workflows.
A high Human Intervention Rate is not always bad. Approval for risky side effects may be intentional. Check instead whether waits exceed the user-journey target, unnecessary approval requests recur, or humans frequently correct the results.
Which signals and diagnostic questions belong at each stage?
The table maps fourteen stages, from Session/Run start through Cost, to signals, representative failure examples, and diagnostic questions. Without a signal that answers a row's question, problems at that stage surface only through user reports.
Stage | Signal | Failure Example | Diagnostic Question |
|---|---|---|---|
| Session·Run start | session_id, run_id, Agent·Workflow Version | One request split across independent traces | Are all stages of the user task connected to one Run? |
| Retrieval | Data Source Version, top_k, Result Count, Score Summary, Latency | Irrelevant documents retrieved from a stale index | Which source and version were used, and were results returned? |
| Plan | Plan Span, Agent Version, Child Model Call | Repeated planning without execution | How many plans were generated, and what changed? |
| Model Call | Provider, Requested·Response Model, TTFC, Duration, Token, Finish Reason | Late response start or length-limit interruption | Provider waiting, generation delay, or output limit? |
| Tool Selection | Tool Set Version, Selected Tool, tool_call_id, Selection Source | Data-changing tool chosen instead of search | Which tools were available, selected, or rejected? |
| Tool Execution | Tool Version, Schema Validation, Duration, Status, Dependency Span | Correct tool failed on an external API timeout | Was selection correct but execution unsuccessful? |
| Retry | Attempt, Reason, Backoff, Recovered | Three retries of a provider error increased latency | Which attempt failed, why, and did it recover? |
| Loop | Iteration, Repeated State, Model·Tool Call Count | Repeated searches and tool calls increased costs | Where did repetition without state progress begin? |
| Guardrail·Policy | Policy Version, Stage, Decision, Category, Latency | Valid requests repeatedly blocked or check time spiked | Which policy changed the outcome at which stage? |
| Human Intervention | Request·Decision Time, Wait Duration, Resume Result | Fast agent waited tens of minutes for approval | How much time was execution versus human waiting? |
| External Dependency | Service, Operation, Status, Timeout, Trace Context | Tool appeared successful but its downstream DB was slow | Which external dependency was on the critical path? |
| Failure | First Error, Terminal Error, Recovered, User-visible Failure | Intermediate error recovered but final result was partial | Were the initial and terminal failures the same? |
| User Outcome | Technical Success, Task Success, Feedback, Correction, Undo | HTTP 200 and an answer, but the user redid the task | Did the user accomplish the actual goal? |
| Cost | Usage, Estimated·Actual Cost, Price Book Version | Failed Run used more tokens and tool costs than successful Runs | Where did costs rise, and did they produce a successful outcome? |
Why start slow-agent diagnosis with the Critical Path?
Several tools may execute concurrently within a Run, so summed child-span durations do not equal user wait time. Separate request-to-first-response time, first-chunk-to-final-response time, and actual execution time. Also distinguish queue/backoff/rate-limit waiting, human-approval waiting, the longest critical path through parallel work, and asynchronous work after result delivery.
Current OpenTelemetry GenAI metrics distinguish total model-operation duration, streaming time to first chunk, and subsequent inter-chunk time.[6] Agent/workflow end-to-end duration is proposed separately from individual model-call duration. Do not combine “the answer starts late” and “the answer starts immediately but never finishes” into one average latency.
Why trace incorrect behavior backward from the outcome?
Producing a final string does not mean the task succeeded. Separate at least two states: technical_success means the Run ended without errors and produced a final output; task_success means the user's intended real-world task was completed.
Define Task Success by use case: a resolved inquiry for support agents, an applied and verified change for operations agents, and a created and delivered file for document agents. For purchasing agents, check the correct item and terms; for analytical agents, whether the user adopted the result. If feedback alone is insufficient, connect business events, subsequent corrections, undo actions, repeated questions, human reviews, and evaluation scores to the same Run ID. LangSmith links feedback to Run IDs, while Google ADK supports evaluating actual tool trajectories and responses.
For incorrect results, work backward from Outcome in this order: did actual Task Success fail; did the final output match the evidence; did it correctly reflect tool execution; was the right tool selected; were retrieval results relevant and current; and which model, prompt, agent, and tool versions were used?
Why are token counts and costs different?
Token usage is an important pricing input, not actual cost by itself. Cost depends on provider, model, input versus output, cache reads and writes, prices at usage time, discounts and contracts, and separate tool or retrieval charges.
Within each Run, distinguish provider-reported usage, input/output/cache token breakdowns, per-request usage, and separate tool/retrieval usage. Record price-book version and effective time, estimated cost, actual cost confirmed from billing data, and whether the figure is Estimate or Actual. OpenTelemetry's token-usage metric calls for billable tokens when a system provides both usage and billable-token values.[1] OpenAI Agents SDK separately retains usage for the full Run and individual requests.
Operational metrics should extend beyond cost per Run. Connect costs to successful tasks, results needing no human correction, and completed user journeys to assess whether spending produced actual value.
Why is raw content not collected by default?
Raw content can make debugging easier, but can also carry PII such as names, emails, and phone numbers; credentials, API keys, and session tokens; customer data and trade secrets; internal system instructions; and tool schemas and internal API structures into the telemetry backend. The same applies to tool arguments and results, retrieval queries and documents, and prompt-injection payloads embedded in external documents.
OpenTelemetry recommends opt-in collection rather than recording model instructions, user messages, and model outputs by default.[1] Where sensitive data or storage volume is a production concern, it recommends separate content storage with references in spans. Tool arguments and results and retrieval queries are likewise opt-in sensitive-content fields. The table describes four recommended capture levels.
Do not assume framework defaults match organizational policy. OpenTelemetry describes a no-raw-content default pattern, but OpenAI Agents SDK currently defaults trace_include_sensitive_data to true. Microsoft advises enabling content recording for development/debugging and disabling it in production. Google ADK warns that debug output containing prompts, tool arguments, and session state is sensitive, and provides settings to prevent PII in span attributes. Enabling instrumentation and permitting raw-content capture therefore need separate settings and approval procedures.
Level | Primary use | Capture scope |
|---|---|---|
| Metadata Only | Production default | IDs, versions, size, latency, status, tokens, errors, outcomes |
| Redacted·Derived | Limited root-cause analysis | Language, length, schema validity, PII/secret categories, de-identified fingerprints |
| External Reference | In-depth production analysis | Content references in separately protected storage; separate access and retention |
| Temporary Debug | Temporary reproduction | Explicit approval, limited targets, short TTL, access auditing, deletion after completion |
Can prompt injection inside telemetry be trusted?
Malicious instructions from external documents or tool results can be copied into telemetry alongside retrieval results, prompts, and completions. If another AI assistant later reads those logs for incident analysis, telemetry becomes a new indirect prompt-injection input.
OWASP identifies AI-consumed logs, links, and external references as untrusted inputs.[4] Its general logging guidance also recommends validating event data from outside the trust zone and sanitizing CR, LF, and delimiters to prevent log injection.
Apply these principles to raw telemetry: label it as untrusted data, not executable instructions; escape and encode it appropriately in dashboards and viewers; do not automatically execute external links or active content; separate raw content from operational instructions when passing it to another agent; apply a distinct retention policy to security-analysis payloads; and automatically end temporary capture after debugging. This concerns telemetry as a second data store and input path, rather than the design of MCP authorization.
Why center dashboards on user outcomes rather than models?
A dashboard led only by average model latency and total tokens pushes user impact into the background. Start with Run-level Task Success Rate, Partial/Failed/Abandoned Run Rate, and user-perceived end-to-end latency. Follow with Human Intervention Rate and Human Wait, Retry/Loop Rate, and model/tool calls per Run. Include Guardrail Block/Modify/Escalate Rate, external-dependency error contribution, cost per successful task, and user correction/undo/rerun rates. Then break these down by agent, workflow, prompt, model, tool, data-source version, and deployment environment.
Do not set one universal threshold for all agents. A conversational agent expected to respond within seconds and an agent performing background analysis for minutes need different latency criteria. Choose SLIs from the Critical User Journey the agent supports.[5]
What is a safe implementation order?
Follow the eight steps above before collecting every prompt and tool result. Without defining Run boundaries and propagating context in the first two steps, later instrumentation cannot assemble into one execution path. Establish span coverage without raw content, add retry/loop/human-wait events, and connect user outcomes and cost models. Then establish the separately approved content-capture policy and schema adapter.
- 1. Define Run boundaries
- Define where a request starts and when it completes, partially completes, fails, or stops. Link delayed asynchronous results back to the original Run.
- 2. Propagate Session, Run, and Trace Context
- Carry context through APIs, queues, workers, other agents, retrieval systems, tools, and external services. Without this, detailed instrumentation cannot assemble into one execution path.
- 3. Establish span coverage without raw content
- First separate retrieval, model calls, tool selection, tool execution, guardrails, and external dependencies into spans. Record versions, durations, status, error types, and call IDs.
- 4. Add retry, loop, and human-wait events
- Structure attempts, backoff, repeated states, and approval waits that automatic instrumentation misses.
- 5. Connect user outcomes
- Distinguish final-output generation from actual Task Success. Link feedback, business events, corrections, undo actions, and human reviews to the Run.
- 6. Add usage and cost models
- Separate token and tool usage, price-book version, and estimated versus actual cost. Read cost alongside successful user outcomes.
- 7. Approve Content Capture Policy separately
- Define development/production rules, data classes, sampling, redaction, access, TTL, and deletion. Explicitly override framework defaults.
- 8. Add a schema adapter and regression checks
- Account for Development-status GenAI conventions and runtime telemetry with an internal canonical schema and mapping version. Test for missing or renamed fields after SDK or instrumentation updates.
Is the purpose of observability to store everything?
Good Agent Observability answers three questions: what actually happened in this Run; where did the first delay or wrong turn occur; and did the user achieve the goal, with how much time and cost? Answering these does not require retaining every prompt, completion, or tool argument verbatim.
Model-call durations and token counts alone can show that the system was slow, but not whether retrieval was empty, the wrong tool was chosen, retries looped, a guardrail blocked the task, a human withheld approval, or the user's problem was actually resolved.
AI agent observability connects decisions, actions, waits, costs, and user outcomes within one Run into a single execution path. It is more than looking at the model in greater detail.
