# Distributed tracing

> See each request's OpenTelemetry trace and span IDs, open the trace in Jaeger, Honeycomb, Datadog or Grafana Tempo, catch servers that break trace propagation, and inject a traceparent into any host's requests.

Source: https://docs.subcloak.com/cloak-proxy/inspect/distributed-tracing/

Added in Cloak Proxy 0.4.0.

Backend teams follow requests through their services in a tracing tool such as Jaeger, Honeycomb, Datadog or Grafana Tempo. Cloak Proxy connects the client side to that view. It reads the trace headers on every captured request and shows the trace and span IDs. One click opens the trace in your backend. It also warns you when a server drops the trace and quietly starts a new one, and it can add a `traceparent` to requests that don't carry one, so every request you debug gets a backend trace.

*[Screenshot: The main window with a Trace column showing short trace IDs for a storefront app's requests, Broken badges on a payment request and a checkout request, and the inspector's Trace tab for the payment request: Trace propagation is broken, with an Open in Honeycomb button, the trace and parent span IDs, and an explanation that the server reported a different trace. The Trace column and the inspector's Trace tab. The payment service started a new trace instead of joining the app's.]*

## What Cloak Proxy reads

Cloak Proxy reads every common way of passing trace context, in requests and in responses. You don't set anything up.

| Format | Headers | Notes |
|---|---|---|
| W3C Trace Context (OpenTelemetry's default) | `traceparent`, `tracestate` | Checked against the specification: version, lowercase hex, IDs that aren't all zeros. The vendors in `tracestate` are listed. |
| B3 (Zipkin) | `X-B3-TraceId`, `X-B3-SpanId`, `X-B3-ParentSpanId`, `X-B3-Sampled`, `X-B3-Flags`, or the single `b3` header | 64-bit and 128-bit trace IDs. |
| Jaeger | `uber-trace-id` | Including URL-encoded values and IDs without leading zeros. |
| Datadog | `x-datadog-trace-id`, `x-datadog-parent-id`, `x-datadog-sampling-priority`, `x-datadog-tags` | Decimal IDs. The upper 64 bits come from `_dd.p.tid`. |
| AWS X-Ray | `X-Amzn-Trace-Id` | `Root=1-…-…`, read as the 32-digit ID OpenTelemetry uses for X-Ray. |
| Google Cloud Trace | `X-Cloud-Trace-Context` | A decimal span ID and the `o=1` sampling option. |
| Sentry | `sentry-trace` | |

On the response side, Cloak Proxy reads the trace the server says it used: a W3C `traceresponse` header, a `traceparent` entry in `Server-Timing`, an echoed trace header, or an `X-Trace-Id` header.

All of these are turned into one trace ID, so a request that sends a 64-bit B3 ID and its child that sends the same trace as a 128-bit W3C ID are recognised as the same trace. When one side only has 64 bits, the lower 64 bits are compared.

## The Trace column

Right-click the traffic list's header and turn on **Trace**, or click the table button in **Tools › Distributed Tracing…**. It also turns on by itself when you start [injecting trace context](#inject-trace-context).

| The cell shows | Meaning |
|---|---|
| The first eight digits of the trace ID, in black | The server confirmed it joined this trace. |
| The first eight digits, in gray | The request carried a trace, or the server started one, but nothing confirmed the server used it. |
| An orange **! Broken** badge | Propagation broke. Hover over it to see why. |
| Nothing | No trace headers either way. |

Requests that got a `traceparent` from Cloak Proxy carry a **Trace** tag next to their URL.

## The Trace tab

Select a request that carries trace headers, or whose response does, and open the inspector's **Trace** tab. A number next to the tab counts the problems.

*[Screenshot: The inspector's Trace tab: The server joined this trace, with an Open in Honeycomb button; the trace ID, parent span and Sampled Yes; the W3C traceparent and tracestate that were sent; the traceresponse the server returned; three other requests in the same trace; and Server-Timing bars for session check, Postgres, Redis and the handler. A request whose server joined the trace, with the other captured requests in it and the server's own timings.]*

| Part | What it shows |
|---|---|
| Verdict | **The server joined this trace**, **Trace context sent** (the response doesn't say), **The server started its own trace** (the request sent none), **No trace context**, or **Trace propagation is broken**. |
| **Open in …** | Opens the trace in your [tracing backend](#open-traces-in-your-backend). With several backends set up, the arrow lists them all. Without any, the button is **Add Trace Link…**. |
| Trace ID, Parent span, Sampled | The IDs as your backend shows them, each with a copy button. **Sampled: No** means the client asked backends not to record the trace, so it may not appear there. Datadog users also see the decimal ID. |
| Problem cards | What broke, and the usual cause. See [Broken propagation](#broken-propagation). |
| **Sent** | Every trace header the request carried, grouped by format, with a check or a cross for whether it's valid. |
| **Reported by the server** | The trace the response named, and the header it came from. |
| **Other requests in this trace** | Other captured requests with the same trace ID, for example the calls one screen of your app made. Click one to select it. **Show All in List** filters the traffic list to the trace. |
| **Server-Timing** | Each `Server-Timing` metric the server sent, such as database or cache time, as a bar. |

## Broken propagation

When trace context gets lost between services, the next service quietly starts a new trace with a different ID. Your trace in the backend stops halfway, and nothing tells you. Cloak Proxy flags three kinds of breakage:

| Problem | What it means | Usual cause |
|---|---|---|
| **The server started a new trace** | The response reports a different trace ID from the one the request sent. | The server's tracer doesn't read the format the client sent (check its propagators, for example `OTEL_PROPAGATORS=tracecontext,baggage,b3`), or a gateway or proxy in between dropped the header. |
| **Malformed trace header** | A request header can't be used: uppercase hex in a `traceparent`, an all-zero ID, a wrong length, version `ff`, a `tracestate` without a `traceparent`. | Code that builds the header by hand, or an old tracer. Receivers ignore invalid headers and start a new trace. |
| **Conflicting trace headers** | Two formats on one request name different traces, for example a `traceparent` and B3 headers with different IDs. | Two tracing libraries in the same client. Different services may follow different headers and split the trace. |

*[Screenshot: The Trace tab for a payment request: Trace propagation is broken. The server reports trace 6e0c63257de34c926f9efcd03927272e in X-Trace-Id, not the 8448eb211c80319c it was sent, with advice about propagators and gateways. The payment service reported a different trace from the B3 one it was sent.]*

> **A response that says nothing isn't a failure:**
> Most servers don't report their trace ID back. Those requests show **Trace context sent**: Cloak Proxy can't tell whether the server joined, so open the trace in your backend to check.

## Open traces in your backend

Choose **Tools › Distributed Tracing…** and open the **Trace Links** tab to set up where **Open in …** goes.

*[Screenshot: The Trace Links tab of the Distributed Tracing window: a Honeycomb link for *.example.com with its URL template, a Jaeger link for every host, and a disabled Grafana Tempo link with a warning to replace its YOUR_ parts, followed by the list of placeholders. Each link is a URL template; the first one whose hosts match the request is the default.]*

1. Click **Add Link** and choose your backend: Jaeger, Zipkin, Grafana Tempo, Honeycomb, Datadog, AWS X-Ray, Google Cloud Trace, Sentry, or **Custom**.
2. Edit the URL template. Starting templates for hosted backends contain parts in capitals, such as `YOUR_TEAM`, which you replace with your own. Until you do, the link shows **Replace the YOUR_… parts**.
3. Optionally, under **For hosts**, list the request hosts the link is for, separated by commas (`api.example.com, *.staging.example.com`). Leave it empty for every host. This lets staging requests open your staging backend and production requests open production.
4. Click the arrow button to open the link with an example trace ID and check it works.

The checkbox turns a link off without deleting it. Links are tried in order, so put the more specific ones first.

| Placeholder | Becomes |
|---|---|
| `{traceId}` | The trace ID, 32 hex digits (16 for a 64-bit B3 or Jaeger ID) |
| `{traceId32}` | Always 32 hex digits, padded with zeros |
| `{traceIdDecimal}` | The lower 64 bits as a decimal, as Datadog shows it |
| `{xrayTraceId}` | The X-Ray form, `1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx` |
| `{spanId}` | The request's span |
| `{startMs}`, `{endMs}` | Five minutes before and after the request, in Unix milliseconds |
| `{startSeconds}`, `{endSeconds}` | The same window, in Unix seconds |
| `{host}` | The request's host |

The time window helps backends such as Honeycomb and Grafana Tempo find the trace quickly. Trace links are saved with your settings.

## Inject trace context

If your app doesn't send trace headers, its requests start a new trace at the first service, or none at all. Cloak Proxy can add a sampled W3C `traceparent` to each request for the hosts you choose, so every request you're debugging has a backend trace you can open.

*[Screenshot: The Inject tab: Add a traceparent to requests for these hosts turned on for api.example.com and *.staging.example.com, Mark the trace as sampled and Keep the app's own trace context turned on, and an example traceparent header. Each matching request gets its own new trace.]*

1. Choose **Tools › Distributed Tracing…** and open the **Inject** tab.
2. Add the hosts, such as `api.example.com` or `*.example.com`, and turn on **Add a traceparent to requests for these hosts**.
3. Make sure [SSL proxying](https://docs.subcloak.com/cloak-proxy/capture/ssl-proxying/) is on for those hosts. Cloak Proxy can only add headers to HTTPS requests it decrypts.
4. Use your app. Each request gets a new trace and a **Trace** tag in the traffic list, and **Open in …** takes you to it.

| Option | What it does |
|---|---|
| **Mark the trace as sampled** | On by default. Sets the sampled flag, asking backends to record the trace. Most drop unsampled traces. |
| **Keep the app's own trace context** | On by default. Requests that already carry a valid trace header in any format are left alone. Turn it off to replace them, which also removes their other trace headers so the backend sees one trace. |

> **Your backend must accept the trace:**
> Injecting only helps if the first service reads W3C Trace Context and records sampled traces. With head-based sampling the server may still decide not to record. And traces you start count against your tracing vendor's quota like any other.

## The overview

The **Overview** tab of **Tools › Distributed Tracing…** sums up trace propagation across the traffic you've captured.

> **On Linux:**
> The Distributed Tracing window opens with <kbd>Ctrl</kbd> <kbd>Alt</kbd> <kbd>Shift</kbd> <kbd>T</kbd>, or by right-clicking a traced request and choosing **Distributed Tracing…**.

*[Screenshot: The Overview tab: tiles for 8 traced requests, 4 confirmed by the server, 2 broken and 3 untraced; a table by host with requests, a traced percentage bar, confirmed and broken counts; and a list of the two requests whose propagation broke, each with a Show button. How trace context travels through the session, host by host.]*

| Part | What it shows |
|---|---|
| **Traced** | Requests that carried trace context, out of all requests. |
| **Confirmed by the server** | Traced requests whose response reported the same trace. |
| **Broken** | Traced requests with a [propagation problem](#broken-propagation). |
| **Untraced** | Requests with no trace context, including those whose server started its own trace. |
| **By host** | The same counts per host, so you can see which APIs get trace context and which don't. The bar turns orange when a host has broken requests. |
| **Broken propagation** | Each broken request with its problem. **Show** selects it in the main window with the Trace tab open. |

The export button saves the overview as a Markdown report.

## Use it from the traffic list

Right-click a request that carries a trace and open the **Trace** submenu:

| Item | What it does |
|---|---|
| **Open in** *backend* | One item for each trace link that covers the request's host. |
| **Show Requests in This Trace** | Filters the traffic list to the requests carrying this trace ID. **Reset** next to the filter brings everything back. |
| **Copy Trace ID** | Copies the trace ID. |
| **Trace Links…** | Opens the Trace Links tab. |

## AI assistants and the command line

The [MCP server](https://docs.subcloak.com/cloak-proxy/automation/mcp-server/) has three tracing tools. See the [MCP tools reference](https://docs.subcloak.com/cloak-proxy/automation/mcp-tools-reference/#distributed-tracing).

| Tool | What an assistant can do with it |
|---|---|
| `get_trace` | Read a request's trace, verdict, problems, Server-Timing and links, and list the other requests in the trace, or find the requests in a trace by its ID (hex, Datadog decimal or X-Ray form). |
| `trace_report` | Summarise propagation per host and list every broken request, as JSON or Markdown. |
| `configure_tracing` | Turn injection on for some hosts and set up trace links. |

Ask, for example: "Which of ShopApp's API calls lose the trace?", or "Give me the Honeycomb link for request 9304's trace, and tell me why its propagation broke."

In CI, `cloakctl run --trace HOST` adds a `traceparent` to each matching request your tests make. `cloakctl trace check` then checks the recording and fails the build when propagation breaks:

```bash title="Terminal"
cloakctl run --port 9091 --ssl 'api.example.com' --trace 'api.example.com' --record run.har -- npm test
cloakctl trace check --har run.har --host api.example.com --fail-on broken
```

| `trace check` option | What it does |
|---|---|
| `--session FILE` / `--har FILE` | The recording to check. |
| `--host HOST` | Only these hosts (repeatable, wildcards allowed). |
| `--format markdown\|json` | The report format. Markdown by default. |
| `--out FILE` | Write the report to a file instead of printing it. |
| `--fail-on broken\|untraced\|never` | Exit with `2` when any request broke propagation (the default), also when any request carried no trace context (`untraced`), or never. |

`cloakctl trace show TRACE_ID --har FILE` lists the requests in one trace. Add `--trace-unsampled` to `cloakctl run` to inject unsampled trace context.

## Limits

- Cloak Proxy sees the requests your app sends and the responses it gets back. Calls between your backend services only show up if they also go through Cloak Proxy, for example with [reverse proxies](https://docs.subcloak.com/cloak-proxy/capture/network-settings/).
- Most servers don't report the trace they used, so a missing confirmation isn't a failure. **The server started a new trace** only appears when the response names a different trace.
- Injection needs the request to be decrypted, and only adds W3C `traceparent`. Services that only read B3, Jaeger or X-Ray headers won't pick it up.
- gRPC's binary `grpc-trace-bin` header and baggage aren't read.

## Related

  - [Inspector](https://docs.subcloak.com/cloak-proxy/inspect/inspector/): Every tab of the request and response inspector.
  - [Traffic list](https://docs.subcloak.com/cloak-proxy/inspect/traffic-list/): Columns, including Trace, and the right-click menu.
  - [Headless runs and CI](https://docs.subcloak.com/cloak-proxy/automation/headless-and-ci/): Record test traffic with cloakctl run.
  - [Trace Value](https://docs.subcloak.com/cloak-proxy/inspect/trace-value/): Follow an ID or token through captured requests (a different kind of trace).