The Missing Span: Designing Observability That Can Tell Causes Apart

1. Observable for Which Question?

Consider two questions about the same checkout application:

Are checkout requests becoming slower?

Are they waiting to acquire database connections, or spending time executing database queries?

A system might answer the first question easily while lacking the information needed to answer the second. Both questions concern the same application, but they require different evidence.

OpenTelemetry’s observability primer describes the ability to ask questions about a system from the outside, including questions that were not anticipated during instrumentation. It also offers a practical standard: an application is properly instrumented when troubleshooting does not require adding instrumentation to answer the question at hand. ^[1]

This is why counting telemetry sources is a poor substitute for evaluating investigative capability. A large volume of metrics, logs, and traces may help engineers detect a problem without providing the relationships or measurements needed to explain it.

Google’s Site Reliability Engineering guidance distinguishes symptoms from causes. An HTTP error is a symptom experienced by a service consumer; a downstream dependency refusing connections may be a contributing cause. Monitoring should expose the symptom promptly, while diagnostic evidence helps engineers investigate the mechanism. ^[2]

A system is observable for a given failure to the extent that its evidence helps engineers discover plausible explanations, distinguish competing causes, and validate the actions taken in response.

A constructed application incident provides a useful test of that principle.

2. The Planted Fault

Our example is hypothetical. We know the fault in advance, but the investigative team does not. This allows us to examine whether the available telemetry would lead to the correct explanation or invite a misleading conclusion.

The application includes a browser frontend, an API gateway, a Java checkout service running on Kubernetes, and a PostgreSQL database.

The checkout service uses plain JDBC access through HikariCP, with the Splunk Distribution of OpenTelemetry Java providing application instrumentation. We intentionally exclude ORM and repository frameworks from this constructed scenario because their instrumentation can produce additional spans.

A configuration deployment reduces the HikariCP maximum from 60 to 20 connections per application replica.

The application uses a five-second connection-acquisition timeout, while the gateway allows ten seconds for the backend request. The checkout service catches the acquisition failure, records an application error, and returns HTTP 500 before the gateway deadline.

Demand exceeds the reduced pool’s capacity only during short bursts. Most requests complete normally, while some wait until acquisition times out.

All numerical values, trace patterns, and diagnostic observations in this article are illustrative predictions—not captured Splunk output, measured performance, or results of a completed laboratory test.

The evidence available to investigators

The table contains two different kinds of evidence. Some measurements establish the symptom, while others distinguish what could have produced it.

A full connection pool, for example, is compatible with undersizing, unusual demand, a connection leak, or long-running transactions. It confirms contention but does not distinguish those causes.

The historical brief maximum of approximately 30 connections per replica is more useful. Earlier bursts already needed more capacity than the new limit allows. With demand and connection use time otherwise unchanged, a 20-connection maximum could produce intermittent acquisition timeouts while most requests continue completing normally.

The investigation still needs to examine the request path, workload, pool behavior, and configuration history before reaching a conclusion.

3. Start with the Customer Experience

The first investigative question is whether the problem affects the service outcome the business cares about.

Splunk Real User Monitoring (RUM) provides information about instrumented browser activity, including network requests, errors, and relevant performance characteristics. In this scenario, its frontend information helps narrow the incident to checkout activity. ^[3]

A browser request returning an error does not explain which internal dependency or application condition caused the failure, so an engineer looks for a link between the affected browser request and its backend trace.

The browser-to-backend link has conditions

Splunk Browser RUM supports correlation with APM traces using backend context supplied through the HTTP Server-Timing response header. The browser agent reads that information and can link a browser span to its corresponding backend trace. ^[3–5]

The header must be produced by supported backend instrumentation and reach the browser. Cross-origin responses may also need to expose Server-Timing through Access-Control-Expose-Headers.

The agent version and framework matter. Splunk documents Java agent support for particular HTTP frameworks, and applications using newer Jakarta Servlet versions should confirm the capabilities of their actual agent release rather than assume older compatibility lists apply unchanged. ^[6]

Failed transactions deserve special attention.

An application-generated HTTP 500 response can still carry the necessary header. A request terminated by a gateway timeout or client disconnect may not receive a backend-generated response at all.

Splunk Browser RUM documents a link.traceId attribute for linking browser activity to an APM trace. Where the necessary browser data is available, engineers can measure the proportion of checkout requests with a usable backend link, segmented by status or outcome. ^[5]

That provides a useful coverage measurement: are failed requests as traceable as successful ones?

In our constructed example, the checkout service returns its error before the gateway timeout. We expect backend correlation to remain possible if the configured instrumentation supplies the header.

The investigation now has a customer-facing symptom and a potential path into the backend.

4. Reading the Trace—and Its Absence

In Splunk APM, the engineer starts with the checkout service and opens Tag Spotlight.

Tag Spotlight breaks down performance by indexed span tags, including request behavior, errors, and supported latency percentiles such as p50, p90, and p99. Environment, endpoint, operation, HTTP method, span kind, and service are among the dimensions indexed by default. [7, 8]

A deployment-version tag is not automatically indexed. An administrator must configure it in advance, subject to the relevant cardinality considerations. Stable version identifiers are generally preferable to ephemeral pod identifiers for this type of indexed aggregation.

If the configuration change is still rolling out, a version breakdown can reveal whether errors are concentrated in the new deployment. If the rollout is complete, the engineer compares the affected period against the earlier baseline.

Neither observation proves the deployment caused the incident. Both help identify where to investigate next.

The missing database-client span

The engineer opens representative failed requests in Trace Analyzer. ^[9]

Under our assumed plain-JDBC configuration and default JDBC statement instrumentation, the predicted pattern is a checkout server span lasting approximately five seconds, ending with an HTTP error, without a corresponding JDBC query span.

The absence matters because the application never obtained a connection and therefore never executed the query.

HikariCP waits for an available connection up to the configured connectionTimeout. If none becomes available, acquisition fails. The subsequent HTTP error is part of our scenario’s defined application behavior. [10, 11]

Default JDBC statement instrumentation records database operations after connection acquisition. Separate datasource instrumentation can represent connection acquisition, but the upstream OpenTelemetry Java agent documents it as disabled by default. Engineers should confirm the setting in the agent release they use. ^[11]

If Spring Data, ORM instrumentation, or manually created repository spans were present, the trace might contain another internal span enclosing the wait. A long repository span without a JDBC query child would not establish that the database server was executing a slow query.

The absence of a JDBC span is meaningful only after confirming that the span should have existed if the request had reached the query.

Where is the exception?

The service catches the HikariCP acquisition failure and returns HTTP 500.

Under OpenTelemetry HTTP server-span conventions, that response sets the server span’s status to Error. The caught exception itself should not be assumed to appear as an exception event because it may never cross an instrumented boundary. ^[12]

Instead, the application logs the acquisition failure.

The trace establishes an unsuccessful request and its timing. The application log supplies the more specific exception and message that can help explain the mechanism.

Successful requests from the same interval offer another comparison. Their JDBC client spans remain near baseline, which argues against widespread slow query execution. It does not exclude every form of database-side delay or long-held connection.

The trace narrows the question; it does not settle it.

What about the service map?

Splunk APM may display PostgreSQL as an inferred service based on database-client spans from instrumented callers. Such a node represents observed client interactions, not direct observation of all database server behavior. ^[13]

In our case, successful requests contribute database-client spans to the map. Requests that never acquire a connection do not execute the corresponding instrumented query.

An inferred database dependency can therefore look ordinary while requests waiting in the application’s connection pool fail before reaching it.

The service map supplies context, but understanding its boundaries is necessary to interpret what it leaves out.

5. The Measurements That Separate the Causes

After identifying the absent query operation, the engineer examines HikariCP metrics.

For supported Java connection-pool libraries, Splunk documents collection of pool metrics through its Java instrumentation. These include connections in use, available capacity, configured limits, pending acquisition requests, timeouts, and connection wait and use times. Exact metric names and units depend on the agent generation and library support. ^[14]

In the constructed scenario, the engineer compares these metrics per replica alongside checkout request rate.

The pool reaches its configured maximum of 20 during short bursts. Pending acquisitions rise, and timeouts occur near the five-second acquisition deadline.

Request demand remains similar to previous peaks. Connection use time is also approximately unchanged.

Those observations point away from a broad demand surge or unusually slow transactions, but the team still needs to compare the alternatives.

Competing explanations

These causes are not necessarily mutually exclusive. Increased demand can expose undersizing; database delays can cause pool contention; a leak can coexist with a configuration change.

The purpose of the matrix is to identify which evidence makes one explanation more credible than another.

The most useful measurement is not necessarily the one showing the largest deviation. A full pool appears under every hypothesis. Normal request rate and normal connection use time are less dramatic, but they help distinguish alternatives.

Configuration history provides the next clue

Our hypothetical deployment record identifies the reduction from 60 to 20 connections per replica.

Earlier bursts briefly required approximately 30 connections per replica. With request demand and connection use time unchanged, that explains why the new maximum could be insufficient during bursts even though the peak-hour average remains below 20.

The combined evidence supports a configuration-related connection acquisition bottleneck.

However, restoring the earlier limit requires checking downstream capacity.

HikariCP’s configured maximum applies to each pool instance. With eight replicas, a maximum of 60 connections per replica permits up to 480 connections across those pools, before considering other applications or autoscaling.

That is a potential connection budget, not a claim that all connections will be open continuously.

The team must compare the projected maximum with the database connection ceiling, other clients, operational headroom, and the effects of greater concurrency.

Restoring a previous configuration is not inherently a safe correction.

6. Following the Evidence into Splunk Logs

The engineer selects a failed checkout trace and looks for the corresponding acquisition-timeout log.

The Splunk Distribution of OpenTelemetry Java adds trace_id and span_id to the logging mapped diagnostic context (MDC) by default. Depending on the configured logging path, supported log events may also be exported through the OpenTelemetry Collector. ^[15]

Those identifiers do not necessarily appear in application log files. When logs are collected from files or through a separate forwarding pipeline, the logging pattern must include the appropriate fields.

Suppose the application error log contains the same trace identifier as the failed request. Through properly configured Related Content and Log Observer Connect, the engineer can move from the APM investigation to the relevant Splunk Platform logs. [16, 17]

The log records a connection-acquisition timeout. The pool metrics show rising pending requests and full configured utilization. The failed request trace shows no executed JDBC query.

Together, these observations support a more specific explanation than a general assertion that the database is slow.

Correlation depends on metadata design

The integration requires compatible identifiers and metadata.

Splunk documents a case in which a log stores a trace identifier as trace.id while the correlation expects trace_id.

A search-time field alias can map the former to the latter without rewriting the stored event. Creating the alias requires appropriate privileges; emitting consistent field names at the source is preferable for long-term interoperability. ^[18]

OpenTelemetry uses service.name for service identification and deployment.environment.name in its current environment conventions. Splunk APM also accommodates the older deployment.environment form. [19, 20]

Kubernetes relationships need sufficient cluster identity to distinguish resources whose names are not globally unique. Splunk Related Content documents the importance of k8s.cluster.name when resolving pod relationships. ^[16]

A trace identifier connects events across a request; consistent resource metadata helps explain where those events occurred.

The Splunk Platform boundary

Log Observer Connect allows supported Splunk Observability Cloud workflows to access logs held in Splunk Enterprise or Splunk Cloud Platform. It does not independently store or index those existing platform logs. ^[17]

Its operational requirements matter to the architecture. Searches use a configured service account, affecting the searchable indexes and access scope.

For self-managed Splunk Enterprise, the search head’s management port, normally TCP 8089, must be reachable as required by the integration.

The documented limitations also include restrictions involving GovCloud logs and Splunk Cloud Platform trials. Deployment-specific support and access requirements should be checked before relying on the workflow.

If the integration cannot query the necessary logs, the investigation has a gap regardless of how useful the APM trace is.

Make the telemetry pipeline observable too

A missing span is only diagnostically useful when the engineer has reasonable confidence that the expected data was not lost.

The Splunk Distribution of the OpenTelemetry Collector exposes internal telemetry for dropped data and export failures. Splunk APM also documents ingestion and system limits, along with tools for investigating throttling. [21, 22]

A span that was never instrumented, removed by upstream sampling, or lost during processing or ingestion will not be available to Trace Analyzer.

Full-fidelity tracing within the documented product architecture does not eliminate the need to examine source instrumentation, upstream sampling, collection behavior, and ingestion limits.

Checking pipeline health helps establish whether the observability system was capable of recording the operation whose absence is being interpreted.

7. A Correction That Tests the Explanation

The investigation now supports a connection-pool configuration problem. How do we validate the correction without confusing recovery with proof?

A simple rollback is useful operationally but diagnostically ambiguous: restarting replicas clears connection pools, so a leak might also appear to recover, and traffic may fall at the same time.

Prefer a controlled staging comparison

For this example, the preferred validation takes place in an isolated staging or load-test environment that reproduces the relevant application configuration and bursty workload.

Two comparable deployment groups receive similar per-replica request rates. Both are restarted at the beginning of the test, removing a difference in process age.

One group retains the reduced maximum of 20 connections. The other uses the previous maximum of 60.

Before the test, the team records its prediction: if the smaller pool is responsible, the reduced-limit group should produce significantly more acquisition delays and timeouts under equivalent demand.

The team compares request rate, error rate, pending acquisitions, wait time, connection use time, and database-side resource behavior.

If the restored-limit group avoids the failures while the reduced-limit group continues experiencing them, the evidence supports the undersized-pool explanation.

If both groups fail, or the restored-limit group degrades with uptime, the team must revisit its assumptions.

The comparison should have predefined limits for database load and test duration, with a responsible operator authorized to stop it. The test must not threaten shared production dependencies.

What if staging is not representative?

Some organizations cannot reproduce production concurrency, downstream latency, or database behavior accurately outside production.

In those cases, the team may need to rely on deployment history, observed production telemetry, and a carefully controlled operational change.

Restoring service safely takes priority. There is no justification for deliberately prolonging a customer-facing incident simply to preserve a comparison group.

If a production canary is permissible, it needs strict exposure limits, monitoring, and rollback conditions. The team must acknowledge that uncontrolled differences reduce the strength of any causal conclusion.

For our constructed case, the expected outcome is that the corrected configuration provides sufficient connection availability without exceeding the database’s safe concurrency budget.

No staging or production test has been run for this manuscript.

8. Recovery Must Mean the Customer Can Check Out

An internal connection-pool metric can confirm that contention has improved. It cannot, by itself, establish that customers are successfully completing orders.

The recovery measurement must correspond to the original business outcome.

Splunk supports service-level objectives (SLOs) using service and endpoint indicators or appropriately constructed custom metrics. Those indicators must be chosen carefully. ^[23]

A technically successful HTTP response is not always a successful business transaction.

For example, a payment decline returned as HTTP 402 or 422 is not automatically classified as an error under OpenTelemetry’s default HTTP server-span conventions. An endpoint-based success metric may report a technically successful request even though no order was created. ^[12]

If the business objective is successful checkout, a custom indicator might count confirmed orders relative to eligible checkout attempts, using supported metric types and consistently defined transaction semantics.

That indicator must account for duplicate attempts, retries, client abandonment, and asynchronous confirmations where those behaviors affect the definition of success.

The team examines immediate checkout behavior following remediation, alongside connection-pool metrics and downstream database health.

Historical SLO compliance requires separate interpretation. Failures remain in a rolling window until they move outside that window; a calendar-based compliance period retains them until the period ends. ^[24]

A service can be recovered now while remaining below its historical objective for the reporting period.

Recovery should therefore be assessed using current customer-facing behavior and operational stability, not simply the disappearance of an alert.

9. Where AI-Assisted Investigation Fits

AI can help engineers explore unfamiliar behavior, especially when relevant evidence spans multiple services and telemetry sources.

Splunk documents an AI Assistant for Observability Cloud supporting natural-language investigative workflows. Separately, its AI troubleshooting agent analyzes eligible alerts and presents suspected causes and supporting evidence. [25, 26]

As documented in November 2026, the troubleshooting agent works with detectors on standard, default metrics; Splunk states that custom metrics are not supported. Its documented remediation-plan scope focuses on Kubernetes-related alerts, and availability depends on product and realm requirements.

These restrictions matter because HikariCP metrics should not be assumed to be eligible inputs for every AI troubleshooting workflow.

The agent’s relative confidence is described in terms of the amount of supporting evidence. That differs from a calibrated probability that a proposed cause is correct.

Several observations originating from the same connection pool can support the conclusion that acquisition is failing, yet do little to distinguish undersizing from a leak.

An assistant may help locate relevant traces, metrics, and logs or suggest explanations worth investigating. The engineer must still determine whether the evidence supports the proposed mechanism.

AI’s useful role here is accelerating the investigation while keeping its conclusions inspectable and testable.

10. Test Your Observability with a Known Fault

An observability architecture should be tested against a question whose answer is known to someone but hidden from the people conducting the investigation.

A practical way to do that is through a controlled diagnostic exercise.

Choose one important transaction, such as placing an order, authenticating a user, or completing a payment. Prefer an isolated staging or load-test environment with representative instrumentation and safe dependencies.

Step 1 — Define success

Write a precise description of the business outcome.

Identify the metric, event, or query that demonstrates success and determine whether it measures the intended outcome rather than a convenient technical proxy.

Step 2 — Map the transaction

Identify the services and dependencies from code, configuration, and architecture records.

Compare that map with the observability platform’s representation. Mark dependencies that are instrumented, inferred, or invisible.

Step 3 — Identify competing explanations

Choose at least three plausible failure mechanisms.

For each hypothesis, specify which evidence would distinguish it from the others. Confirm that the measurements are collected, available at useful resolution, and retained for an appropriate period.

Step 4 — Introduce a controlled fault

The person running the drill chooses one safe fault, records exactly what changed, and documents the evidence expected to reveal it.

Examples include reducing a test connection pool, introducing latency into an isolated dependency, or removing correlation metadata in a disposable application environment.

Tag the test telemetry with a dedicated environment identity, route test alerts away from production on-call workflows, and agree in advance who has authority to stop the exercise.

Do not use uncontrolled fault injection against production systems or shared dependencies.

Step 5 — Diagnose without knowing the answer

Provide a separate investigation team with the telemetry and access it would normally use.

Record the hypotheses considered, observations that supported or rejected them, and evidence gaps encountered.

For each hypothesis, document:

This turns the exercise into a concrete engineering backlog rather than an informal confidence assessment.

Step 6 — Verify recovery

Remove the fault and confirm that the business outcome from Step 1 has recovered.

Use the same measurement that defined success to evaluate recovery, supplemented by relevant technical indicators.

If you don’t have staging

A blind retrospective review of a resolved incident provides an alternative.

Choose a recent incident whose cause is known to one team. Give its surviving telemetry to investigators who were not involved, and ask them to reconstruct the diagnosis without revealing the answer.

The exercise is limited by retention. Splunk APM’s default trace retention is eight days, so older incidents may be reviewable only through retained metrics and Splunk Platform logs, which have their own retention settings. ^[9]

That limitation is itself a useful finding: it reveals which questions the team can still answer after detailed traces are no longer available.

The output of either exercise is a prioritized list of evidence gaps and the specific changes needed to close them.

Conclusion

Our constructed checkout incident was not short of telemetry. Its useful evidence emerged when the investigation examined what the traces omitted, compared connection-pool behavior with historical demand, and tested explanations against measurements that could distinguish them.

The Splunk workflow connected those observations, but its effectiveness depended on instrumentation boundaries, metadata quality, integration, and careful interpretation. Those principles apply regardless of the observability platform.

Choose one critical transaction and run the diagnostic exercise this quarter. The gaps it reveals are the work worth doing.