LLM Inspector

The LLM Inspector reads the model calls in your captured traffic. For each call it shows what was sent and what came back, how many tokens it used, what it cost, how long the first token took, and which tool calls the model asked for and what the app answered. It also shows where the context window went and the first change since the previous turn, which is where a prompt cache stops matching.

It works for any app or agent on your Mac, a simulator or a device, with no SDK and no base-URL change. Prompts and replies stay on your Mac.

1234567

A support assistant's second turn: most of the prompt came from the cache, and the tool result from the previous call is the new turn.

1. Model, provider and API
2. Cost; click for the price list
3. Time to first token
4. Where the context window went
5. How this prompt relates to the previous one
6. A tool call and its result
7. Replay the reply as it arrived

×

Which calls it reads

API	Recognized by	Streamed replies
OpenAI Chat Completions, and compatible servers (Azure OpenAI, OpenRouter, Groq, Mistral, DeepSeek, Together AI, xAI, vLLM, LM Studio…)	POST …/chat/completions	SSE
OpenAI Responses	POST …/v1/responses	SSE (response.* events)
Anthropic Messages	POST …/v1/messages, or …/messages with an anthropic-version header	SSE
Google Gemini	…:generateContent and …:streamGenerateContent	SSE (alt=sse) or a JSON array
Ollama	POST /api/chat and /api/generate with a JSON body naming a model	JSON lines

The provider name comes from the host (OpenAI, Anthropic, Google, Azure OpenAI, OpenRouter…). Calls to localhost show as Local server.

1. Turn on SSL Proxying for the API’s host (for example api.openai.com) or for the app. Calls to a local server over plain HTTP need nothing.
2. Run your app or agent. Command-line tools and SDKs that ignore the macOS proxy can be pointed at Cloak Proxy from a terminal; the Doctor tells you when an app skips the proxy.
3. Click AI in the filter bar to show only model calls. Each one has an indigo badge with its model before the URL, with Live · in front while the reply is streaming.
4. Select a call. The response pane opens on the AI tab.

Streamed replies pass through Cloak Proxy as they arrive, so chat UIs keep streaming while you watch. See Streaming responses.

The AI tab

Numbers at the top

Part	Shows
Model	The model the response named (or the one requested), the provider and the API. Hover to see the model the request asked for when it differs, for example an alias.
Input	Every prompt token: billed at the full price, read from the prompt cache, and written to it. Below: the share read from the cache, no cache hit, or how many tokens were written to the cache. Hover for the split.
Output	Output tokens, with reasoning (thinking) tokens below when the model reported them. Reasoning tokens are billed as output.
Cost	What the call cost, from the price list. Below: how much the cache saved, or no price when the model isn’t in the list. Hover for each part’s price; click to open the list.
First token	Time from sending the request to the first piece of the reply: text, reasoning or a tool call. Below: when the response headers arrived, or not streamed.
Speed	Output tokens per second while generating, from the first token to the last. Below: the whole call’s duration.
Stop	The stop reason the API gave (end_turn, stop, tool_use, length…), orange when the reply was cut short (max_tokens, length, refusal, content_filter). Below: the HTTP status.

In a narrow pane the numbers wrap onto two rows.

When a call fails, a red banner shows the provider’s own error message and type, for example Rate limited (tokens), and how long the provider asked to wait before retrying (Retry-After). A stream that stops in the middle of the reply says The stream ended before the reply was complete.

Context window

A bar splits the prompt’s tokens into:

Part	What’s in it
System prompt	The system prompt or instructions
Tool definitions	Every tool’s name, description and schema
History	Earlier turns of the conversation
Images	Images, at about 1,000 tokens each when the provider doesn’t say
New turn	What this call added: the last user message, or the tool results since the model’s last reply

The parts are sized by how many bytes each takes in the request, then scaled to the prompt token count the provider reported, so they add up to it. When the response has no token counts, the header says estimated (about four bytes per token).

Under the bar, Cloak Proxy compares the prompt with the earlier call to the same host and model that shares the most of it from the start:

• Adds to the prompt of #N: this prompt is the earlier one plus new turns, so a prompt cache can reuse all of the earlier part.
• Differs from the prompt of #N: the line below names the first change (the system prompt, a tool’s definition, or a message) and its character, with the text around it before (−) and after (+). Everything after that point is new to a prompt cache.

A date in the system prompt changed, so the cache stopped matching there. The tool definitions before it still matched.

×

Providers cache in this order: tool definitions, then the system prompt, then messages. A timestamp, a request id or a reordered tool list early in the prompt breaks the cache for everything after it. Keep what changes at the end.

Click #N to select that request.

Prompt

The request as a transcript, one row per part:

Row	Shows
System	The system prompt, folded to three lines. Show more opens it.
Tools	Each tool’s name, description and size. The first four show; Show all lists the rest.
User, Assistant, Tool	Each message’s text (long ones folded), its images, and its reasoning when the request carries it. The first row of the new turn is marked New.

Tool calls in the history show as teal cards with their arguments as formatted JSON; arguments that aren’t valid JSON get a red Invalid JSON badge. Tool results show the content the app sent back, formatted when it’s JSON, red when the app marked it as an error, with Asked for in #N linking to the call whose reply requested it.

Reply

The reply as the app received it, reassembled from the stream: reasoning (folded), text, and tool calls. Each tool call card shows Result in #N, the later request that carried its result back, with the result underneath, or No result captured. Hover the link to see how long the app took before sending the result.

For streamed replies, a slider replays the order the pieces arrived. Drag it, or click the play button to watch the reply arrive at its real pace (long replies are sped up to five seconds). Text that hadn’t arrived yet at the slider’s time is faint, and so are tool calls that hadn’t started. The count shows how many pieces had arrived.

Right-click the tab for Prices… and Copy Reply.

In the traffic list

Where	What
AI chip in the filter bar	Shows only model calls
Badge before the URL	The model, indigo; Live · while streaming
AI columns	Right-click the column header and choose Show AI Columns, or turn on single columns: Model, Tokens (input → output, with the cache share in green), Cost, First Token (orange over 3 seconds), Tok/s and Stop Reason (red rate limited or error for failed calls). Hover a cell for details.

Insights › AI

Tools › Insights… (⌥ ⌘ I), then the AI tab, adds up every model call in the session, or only those the current filter shows (Analyze).

Spend and speed by model. Group by app or host with the buttons above the table.

×

Part	Shows
Tiles	Calls and failures, total cost (and how many models have no price), input tokens and the share read from the cache, output tokens, median and 95th-percentile time to first token, and Retries: calls that repeated an earlier call’s exact prompt within two minutes
Group by	Model, App or Host: calls, failed, input, cached share, output, median first token and cost, most expensive first
Most expensive calls	The 100 costliest calls. Double-click one, or right-click › Show in Traffic List, to select it

Export Report includes an AI section with the same totals.

Prices

Costs use a price list in dollars per million tokens: input, output, cache reads and cache writes. Cloak Proxy has built-in prices for Anthropic, OpenAI and Google models, dated per provider. Your own entries go on top of them.

Costs are estimates

The built-in prices are a snapshot taken when this version of Cloak Proxy was made, not live prices, and some are many months old. Providers change prices and add models often, so treat costs as estimates, and edit the price of any model you rely on, in the price list or in llm-prices.json (below), to match your provider’s current price page or your contract.

Open the list with Prices… in Insights › AI, by clicking Cost in the AI tab, or from the AI tab’s right-click menu.

Edit any price to make it yours.

×

To	Do this
Change a price	Edit it in the table. The row becomes Yours and replaces the built-in entry.
Price another model	Add Model, then rename it and fill in the prices. A gateway’s model ids, a fine-tuned model, or a local model at 0.
Go back to the built-in price	Click the arrow next to Yours
Share or back up your prices	Export… writes the whole list as JSON; Import… adds a file’s entries to yours

A model id matches the longest entry it starts with, ignoring routing prefixes: gpt-4.1-mini-2025-04-14 uses gpt-4.1-mini, openai/gpt-4o-mini uses gpt-4o-mini, and us.anthropic.claude-sonnet-4-6 uses claude-sonnet-4-6. An empty cache price means the input price.

Your entries are saved in llm-prices.json in Cloak Proxy’s folder (~/Library/Application Support/CloakProxy/). The file can also be a simple map, which is easy to write by hand:

llm-prices.json

{
  "my-gateway-model": { "input": 1.5, "output": 6, "cacheRead": 0.15 },
  "llama3.2": { "input": 0, "output": 0 }
}

What the cost leaves out

Costs are list prices for the tokens the provider reported. Batch discounts, long-context surcharges, priority tiers, fast modes, server-side tools (web search) and anything your contract changes aren’t included; add your own prices for those models.

With an AI assistant

Three MCP tools read the same data:

Tool	Returns
llm_calls	Each call’s model, tokens, cost, time to first token, speed, stop reason, tool calls, errors and retries. Filter by app, host, model, errors_only or since_id. With id, one call in full: prompt, reply, tool calls with their results, the context budget, and sinceLastTurn (the first change and the text around it).
llm_summary	Totals by model, app and host, cache hit percentages, median and p95 time to first token, retries and models without a price
llm_prices	The price list. set adds or changes entries, remove takes yours out

Why did request #42 miss the prompt cache, and what did the agent’s last run cost?

See MCP tools reference.

In CI

cloakctl llm summarizes the model calls in a recorded session, and can fail a build that spent too much:

Terminal

cloakctl run --ssl '*' --record run.cloaksession -- npm test
cloakctl llm --session run.cloaksession --calls --max-cost 0.50

Output

8 model calls · $0.011 · 6.3k input (2.5k cached) · 364 output · median first token 660 ms
  claude-sonnet-5-5              3     calls  $0.011
  gemini-2.5-flash               1     calls  $0.0004
#8001  claude-sonnet-5-5  1740 in (0 cached)  54 out  $0.0049  first token 660 ms  tool_use

Option	What it does
--session FILE / --har FILE	Reads a saved session. Without it, asks the running app.
--calls	Also lists every call
--json	Prints the llm_summary result
--max-cost DOLLARS	Exits 2 when the total is above this, or when a model has no price (so the total can’t be trusted)
--prices FILE	Uses a price file instead of your saved prices, so CI doesn’t depend on your Mac
ID	One call in full, as JSON

cloakctl run --report FILE adds an AI section to its Markdown report when the run made model calls.

Privacy

Prompts, replies and keys never leave your Mac: the inspector only reads traffic Cloak Proxy already captured. API keys in headers are shown as captured, so use redaction before you share a session. Prices are a local file; nothing is fetched.

Related

Streaming responsesEvery SSE event of a reply, with its arrival time.

SSL ProxyingDecrypt the model API's host.

InsightsEndpoints, errors, secrets and performance.

Run API tests in CIRecord a test run and check it.