Distributed tracing

Backend teams follow requests through their services in a tracing tool such as Jaeger, Honeycomb, Datadog or Grafana Tempo. Cloak Proxy connects the client side to that view. It reads the trace headers on every captured request and shows the trace and span IDs. One click opens the trace in your backend. It also warns you when a server drops the trace and quietly starts a new one, and it can add a traceparent to requests that don’t carry one, so every request you debug gets a backend trace.

1234

The Trace column and the inspector's Trace tab. The payment service started a new trace instead of joining the app's.

1. The Trace column
2. Propagation broke here
3. The Trace tab
4. Opens the trace in your backend

×

What Cloak Proxy reads

Cloak Proxy reads every common way of passing trace context, in requests and in responses. You don’t set anything up.

Format	Headers	Notes
W3C Trace Context (OpenTelemetry’s default)	traceparent, tracestate	Checked against the specification: version, lowercase hex, IDs that aren’t all zeros. The vendors in tracestate are listed.
B3 (Zipkin)	X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled, X-B3-Flags, or the single b3 header	64-bit and 128-bit trace IDs.
Jaeger	uber-trace-id	Including URL-encoded values and IDs without leading zeros.
Datadog	x-datadog-trace-id, x-datadog-parent-id, x-datadog-sampling-priority, x-datadog-tags	Decimal IDs. The upper 64 bits come from _dd.p.tid.
AWS X-Ray	X-Amzn-Trace-Id	Root=1-…-…, read as the 32-digit ID OpenTelemetry uses for X-Ray.
Google Cloud Trace	X-Cloud-Trace-Context	A decimal span ID and the o=1 sampling option.
Sentry	sentry-trace

On the response side, Cloak Proxy reads the trace the server says it used: a W3C traceresponse header, a traceparent entry in Server-Timing, an echoed trace header, or an X-Trace-Id header.

All of these are turned into one trace ID, so a request that sends a 64-bit B3 ID and its child that sends the same trace as a 128-bit W3C ID are recognised as the same trace. When one side only has 64 bits, the lower 64 bits are compared.

The Trace column

Right-click the traffic list’s header and turn on Trace, or click the table button in Tools › Distributed Tracing…. It also turns on by itself when you start injecting trace context.

The cell shows	Meaning
The first eight digits of the trace ID, in black	The server confirmed it joined this trace.
The first eight digits, in gray	The request carried a trace, or the server started one, but nothing confirmed the server used it.
An orange ! Broken badge	Propagation broke. Hover over it to see why.
Nothing	No trace headers either way.

Requests that got a traceparent from Cloak Proxy carry a Trace tag next to their URL.

The Trace tab

Select a request that carries trace headers, or whose response does, and open the inspector’s Trace tab. A number next to the tab counts the problems.

12345

A request whose server joined the trace, with the other captured requests in it and the server's own timings.

1. The verdict
2. Opens the trace
3. Copy the IDs
4. Other requests in the same trace
5. Server-Timing

×

Part	What it shows
Verdict	The server joined this trace, Trace context sent (the response doesn’t say), The server started its own trace (the request sent none), No trace context, or Trace propagation is broken.
Open in …	Opens the trace in your tracing backend. With several backends set up, the arrow lists them all. Without any, the button is Add Trace Link….
Trace ID, Parent span, Sampled	The IDs as your backend shows them, each with a copy button. Sampled: No means the client asked backends not to record the trace, so it may not appear there. Datadog users also see the decimal ID.
Problem cards	What broke, and the usual cause. See Broken propagation.
Sent	Every trace header the request carried, grouped by format, with a check or a cross for whether it’s valid.
Reported by the server	The trace the response named, and the header it came from.
Other requests in this trace	Other captured requests with the same trace ID, for example the calls one screen of your app made. Click one to select it. Show All in List filters the traffic list to the trace.
Server-Timing	Each Server-Timing metric the server sent, such as database or cache time, as a bar.

Broken propagation

When trace context gets lost between services, the next service quietly starts a new trace with a different ID. Your trace in the backend stops halfway, and nothing tells you. Cloak Proxy flags three kinds of breakage:

Problem	What it means	Usual cause
The server started a new trace	The response reports a different trace ID from the one the request sent.	The server’s tracer doesn’t read the format the client sent (check its propagators, for example OTEL_PROPAGATORS=tracecontext,baggage,b3), or a gateway or proxy in between dropped the header.
Malformed trace header	A request header can’t be used: uppercase hex in a traceparent, an all-zero ID, a wrong length, version ff, a tracestate without a traceparent.	Code that builds the header by hand, or an old tracer. Receivers ignore invalid headers and start a new trace.
Conflicting trace headers	Two formats on one request name different traces, for example a traceparent and B3 headers with different IDs.	Two tracing libraries in the same client. Different services may follow different headers and split the trace.

The payment service reported a different trace from the B3 one it was sent.

×

A response that says nothing isn't a failure

Most servers don’t report their trace ID back. Those requests show Trace context sent: Cloak Proxy can’t tell whether the server joined, so open the trace in your backend to check.

Open traces in your backend

Choose Tools › Distributed Tracing… and open the Trace Links tab to set up where Open in … goes.

Each link is a URL template; the first one whose hosts match the request is the default.

×

1. Click Add Link and choose your backend: Jaeger, Zipkin, Grafana Tempo, Honeycomb, Datadog, AWS X-Ray, Google Cloud Trace, Sentry, or Custom.
2. Edit the URL template. Starting templates for hosted backends contain parts in capitals, such as YOUR_TEAM, which you replace with your own. Until you do, the link shows Replace the YOUR_… parts.
3. Optionally, under For hosts, list the request hosts the link is for, separated by commas (api.example.com, *.staging.example.com). Leave it empty for every host. This lets staging requests open your staging backend and production requests open production.
4. Click the arrow button to open the link with an example trace ID and check it works.

The checkbox turns a link off without deleting it. Links are tried in order, so put the more specific ones first.

Placeholder	Becomes
{traceId}	The trace ID, 32 hex digits (16 for a 64-bit B3 or Jaeger ID)
{traceId32}	Always 32 hex digits, padded with zeros
{traceIdDecimal}	The lower 64 bits as a decimal, as Datadog shows it
{xrayTraceId}	The X-Ray form, 1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx
{spanId}	The request’s span
{startMs}, {endMs}	Five minutes before and after the request, in Unix milliseconds
{startSeconds}, {endSeconds}	The same window, in Unix seconds
{host}	The request’s host

The time window helps backends such as Honeycomb and Grafana Tempo find the trace quickly. Trace links are saved with your settings.

Inject trace context

If your app doesn’t send trace headers, its requests start a new trace at the first service, or none at all. Cloak Proxy can add a sampled W3C traceparent to each request for the hosts you choose, so every request you’re debugging has a backend trace you can open.

Each matching request gets its own new trace.

×

1. Choose Tools › Distributed Tracing… and open the Inject tab.
2. Add the hosts, such as api.example.com or *.example.com, and turn on Add a traceparent to requests for these hosts.
3. Make sure SSL proxying is on for those hosts. Cloak Proxy can only add headers to HTTPS requests it decrypts.
4. Use your app. Each request gets a new trace and a Trace tag in the traffic list, and Open in … takes you to it.

Option	What it does
Mark the trace as sampled	On by default. Sets the sampled flag, asking backends to record the trace. Most drop unsampled traces.
Keep the app’s own trace context	On by default. Requests that already carry a valid trace header in any format are left alone. Turn it off to replace them, which also removes their other trace headers so the backend sees one trace.

Your backend must accept the trace

Injecting only helps if the first service reads W3C Trace Context and records sampled traces. With head-based sampling the server may still decide not to record. And traces you start count against your tracing vendor’s quota like any other.

The overview

The Overview tab of Tools › Distributed Tracing… sums up trace propagation across the traffic you’ve captured.

On Linux

The Distributed Tracing window opens with Ctrl Alt Shift T, or by right-clicking a traced request and choosing Distributed Tracing….

How trace context travels through the session, host by host.

×

Part	What it shows
Traced	Requests that carried trace context, out of all requests.
Confirmed by the server	Traced requests whose response reported the same trace.
Broken	Traced requests with a propagation problem.
Untraced	Requests with no trace context, including those whose server started its own trace.
By host	The same counts per host, so you can see which APIs get trace context and which don’t. The bar turns orange when a host has broken requests.
Broken propagation	Each broken request with its problem. Show selects it in the main window with the Trace tab open.

The export button saves the overview as a Markdown report.

Use it from the traffic list

Right-click a request that carries a trace and open the Trace submenu:

Item	What it does
Open in backend	One item for each trace link that covers the request’s host.
Show Requests in This Trace	Filters the traffic list to the requests carrying this trace ID. Reset next to the filter brings everything back.
Copy Trace ID	Copies the trace ID.
Trace Links…	Opens the Trace Links tab.

AI assistants and the command line

The MCP server has three tracing tools. See the MCP tools reference.

Tool	What an assistant can do with it
get_trace	Read a request’s trace, verdict, problems, Server-Timing and links, and list the other requests in the trace, or find the requests in a trace by its ID (hex, Datadog decimal or X-Ray form).
trace_report	Summarise propagation per host and list every broken request, as JSON or Markdown.
configure_tracing	Turn injection on for some hosts and set up trace links.

Ask, for example: “Which of ShopApp’s API calls lose the trace?”, or “Give me the Honeycomb link for request 9304’s trace, and tell me why its propagation broke.”

In CI, cloakctl run --trace HOST adds a traceparent to each matching request your tests make. cloakctl trace check then checks the recording and fails the build when propagation breaks:

Terminal

cloakctl run --port 9091 --ssl 'api.example.com' --trace 'api.example.com' --record run.har -- npm test
cloakctl trace check --har run.har --host api.example.com --fail-on broken

trace check option	What it does
--session FILE / --har FILE	The recording to check.
--host HOST	Only these hosts (repeatable, wildcards allowed).
--format markdown|json	The report format. Markdown by default.
--out FILE	Write the report to a file instead of printing it.
--fail-on broken|untraced|never	Exit with 2 when any request broke propagation (the default), also when any request carried no trace context (untraced), or never.

cloakctl trace show TRACE_ID --har FILE lists the requests in one trace. Add --trace-unsampled to cloakctl run to inject unsampled trace context.

Limits

• Cloak Proxy sees the requests your app sends and the responses it gets back. Calls between your backend services only show up if they also go through Cloak Proxy, for example with reverse proxies.
• Most servers don’t report the trace they used, so a missing confirmation isn’t a failure. The server started a new trace only appears when the response names a different trace.
• Injection needs the request to be decrypted, and only adds W3C traceparent. Services that only read B3, Jaeger or X-Ray headers won’t pick it up.
• gRPC’s binary grpc-trace-bin header and baggage aren’t read.

Related

InspectorEvery tab of the request and response inspector.

Traffic listColumns, including Trace, and the right-click menu.

Headless runs and CIRecord test traffic with cloakctl run.

Trace ValueFollow an ID or token through captured requests (a different kind of trace).