# LLM Inspector

> See every model call your app or agent makes (OpenAI, Anthropic, Gemini, Ollama and compatible APIs) with tokens, cost, time to first token, tool calls, where the context window went and why the prompt cache missed. Nothing leaves your Mac.

Source: https://docs.subcloak.com/cloak-proxy/inspect/llm-inspector/

Added in Cloak Proxy 0.4.0.

The LLM Inspector reads the model calls in your captured traffic. For each call it shows what was sent and what came back, how many tokens it used, what it cost, how long the first token took, and which tool calls the model asked for and what the app answered. It also shows where the context window went and the first change since the previous turn, which is where a prompt cache stops matching.

It works for any app or agent on your Mac, a simulator or a device, with no SDK and no base-URL change. Prompts and replies stay on your Mac.

*[Screenshot: The AI tab of an Anthropic call: model claude-sonnet-5-5, 1,910 input tokens 89% cached, 41 output tokens, cost $0.0012 with $0.0031 saved, first token after 450 ms, 128 tokens per second, stop reason end_turn. Below, a context window bar split into system prompt, tool definitions, history and new turn, a line saying it adds to the prompt of request 8001, then the prompt as a transcript: the system prompt, two tools, the user's question, the assistant's lookup_order tool call, and the tool result marked New. Under Reply, a replay slider and the assistant's answer.. A support assistant's second turn: most of the prompt came from the cache, and the tool result from the previous call is the new turn.]*

## Which calls it reads

| API | Recognized by | Streamed replies |
|---|---|---|
| OpenAI Chat Completions, and compatible servers (Azure OpenAI, OpenRouter, Groq, Mistral, DeepSeek, Together AI, xAI, vLLM, LM Studio…) | `POST …/chat/completions` | SSE |
| OpenAI Responses | `POST …/v1/responses` | SSE (`response.*` events) |
| Anthropic Messages | `POST …/v1/messages`, or `…/messages` with an `anthropic-version` header | SSE |
| Google Gemini | `…:generateContent` and `…:streamGenerateContent` | SSE (`alt=sse`) or a JSON array |
| Ollama | `POST /api/chat` and `/api/generate` with a JSON body naming a model | JSON lines |

The provider name comes from the host (OpenAI, Anthropic, Google, Azure OpenAI, OpenRouter…). Calls to `localhost` show as **Local server**.

1. Turn on [SSL Proxying](https://docs.subcloak.com/cloak-proxy/capture/ssl-proxying/) for the API's host (for example `api.openai.com`) or for the app. Calls to a local server over plain HTTP need nothing.
2. Run your app or agent. Command-line tools and SDKs that ignore the macOS proxy can be pointed at Cloak Proxy from a [terminal](https://docs.subcloak.com/cloak-proxy/devices/terminal-and-runtimes/); the [Doctor](https://docs.subcloak.com/cloak-proxy/capture/doctor/) tells you when an app skips the proxy.
3. Click **AI** in the [filter bar](https://docs.subcloak.com/cloak-proxy/inspect/filtering/) to show only model calls. Each one has an indigo badge with its model before the URL, with **Live ·** in front while the reply is streaming.
4. Select a call. The response pane opens on the **AI** tab.

Streamed replies pass through Cloak Proxy as they arrive, so chat UIs keep streaming while you watch. See [Streaming responses](https://docs.subcloak.com/cloak-proxy/inspect/streams/).

## The AI tab

### Numbers at the top

| Part | Shows |
|---|---|
| **Model** | The model the response named (or the one requested), the provider and the API. Hover to see the model the request asked for when it differs, for example an alias. |
| **Input** | Every prompt token: billed at the full price, read from the prompt cache, and written to it. Below: the share read from the cache, **no cache hit**, or how many tokens were written to the cache. Hover for the split. |
| **Output** | Output tokens, with reasoning (thinking) tokens below when the model reported them. Reasoning tokens are billed as output. |
| **Cost** | What the call cost, from the [price list](#prices). Below: how much the cache saved, or **no price** when the model isn't in the list. Hover for each part's price; click to open the list. |
| **First token** | Time from sending the request to the first piece of the reply: text, reasoning or a tool call. Below: when the response headers arrived, or **not streamed**. |
| **Speed** | Output tokens per second while generating, from the first token to the last. Below: the whole call's duration. |
| **Stop** | The stop reason the API gave (`end_turn`, `stop`, `tool_use`, `length`…), orange when the reply was cut short (`max_tokens`, `length`, `refusal`, `content_filter`). Below: the HTTP status. |

In a narrow pane the numbers wrap onto two rows.

When a call fails, a red banner shows the provider's own error message and type, for example **Rate limited (tokens)**, and how long the provider asked to wait before retrying (`Retry-After`). A stream that stops in the middle of the reply says **The stream ended before the reply was complete**.

### Context window

A bar splits the prompt's tokens into:

| Part | What's in it |
|---|---|
| **System prompt** | The system prompt or instructions |
| **Tool definitions** | Every tool's name, description and schema |
| **History** | Earlier turns of the conversation |
| **Images** | Images, at about 1,000 tokens each when the provider doesn't say |
| **New turn** | What this call added: the last user message, or the tool results since the model's last reply |

The parts are sized by how many bytes each takes in the request, then scaled to the prompt token count the provider reported, so they add up to it. When the response has no token counts, the header says **estimated** (about four bytes per token).

Under the bar, Cloak Proxy compares the prompt with the earlier call to the same host and model that shares the most of it from the start:

- **Adds to the prompt of #N**: this prompt is the earlier one plus new turns, so a prompt cache can reuse all of the earlier part.
- **Differs from the prompt of #N**: the line below names the first change (the system prompt, a tool's definition, or a message) and its character, with the text around it before (−) and after (+). Everything after that point is new to a prompt cache.

*[Screenshot: The context window of a new conversation: Differs from the prompt of request 8002, the first 90% (863 bytes) is the same. First change in system, character 371: before, Today is 2026-10-18; after, Today is 2026-10-19.. A date in the system prompt changed, so the cache stopped matching there. The tool definitions before it still matched.]*

Providers cache in this order: tool definitions, then the system prompt, then messages. A timestamp, a request id or a reordered tool list early in the prompt breaks the cache for everything after it. Keep what changes at the end.

Click **#N** to select that request.

### Prompt

The request as a transcript, one row per part:

| Row | Shows |
|---|---|
| **System** | The system prompt, folded to three lines. **Show more** opens it. |
| **Tools** | Each tool's name, description and size. The first four show; **Show all** lists the rest. |
| **User**, **Assistant**, **Tool** | Each message's text (long ones folded), its images, and its reasoning when the request carries it. The first row of the new turn is marked **New**. |

Tool calls in the history show as teal cards with their arguments as formatted JSON; arguments that aren't valid JSON get a red **Invalid JSON** badge. Tool results show the content the app sent back, formatted when it's JSON, red when the app marked it as an error, with **Asked for in #N** linking to the call whose reply requested it.

### Reply

The reply as the app received it, reassembled from the stream: reasoning (folded), text, and tool calls. Each tool call card shows **Result in #N**, the later request that carried its result back, with the result underneath, or **No result captured**. Hover the link to see how long the app took before sending the result.

For streamed replies, a slider replays the order the pieces arrived. Drag it, or click the play button to watch the reply arrive at its real pace (long replies are sped up to five seconds). Text that hadn't arrived yet at the slider's time is faint, and so are tool calls that hadn't started. The count shows how many pieces had arrived.

Right-click the tab for **Prices…** and **Copy Reply**.

## In the traffic list

| Where | What |
|---|---|
| **AI** chip in the filter bar | Shows only model calls |
| Badge before the URL | The model, indigo; **Live ·** while streaming |
| AI columns | Right-click the column header and choose **Show AI Columns**, or turn on single columns: **Model**, **Tokens** (input → output, with the cache share in green), **Cost**, **First Token** (orange over 3 seconds), **Tok/s** and **Stop Reason** (red **rate limited** or **error** for failed calls). Hover a cell for details. |

## Insights › AI

**Tools › Insights…** (<kbd>⌥</kbd> <kbd>⌘</kbd> <kbd>I</kbd>), then the **AI** tab, adds up every model call in the session, or only those the current filter shows (**Analyze**).

*[Screenshot: Insights, AI tab: 8 calls with 1 failed, cost $0.011 with 1 model without a price, 6,332 input tokens 39% from cache, 364 output tokens, median first token 660 ms with p95 1.120 s, 0 retries. A table grouped by model lists claude-sonnet-5-5, gemini-2.5-flash, gpt-5-mini, gpt-4.1-mini, gpt-4.1 and llama3.2 with calls, failures, input, cached share, output, first token and cost. Below, the most expensive calls.. Spend and speed by model. Group by app or host with the buttons above the table.]*

| Part | Shows |
|---|---|
| Tiles | Calls and failures, total cost (and how many models have no price), input tokens and the share read from the cache, output tokens, median and 95th-percentile time to first token, and **Retries**: calls that repeated an earlier call's exact prompt within two minutes |
| **Group by** | Model, App or Host: calls, failed, input, cached share, output, median first token and cost, most expensive first |
| **Most expensive calls** | The 100 costliest calls. Double-click one, or right-click › **Show in Traffic List**, to select it |

**Export Report** includes an **AI** section with the same totals.

## Prices

Costs use a price list in dollars per million tokens: input, output, cache reads and cache writes. Cloak Proxy has built-in prices for Anthropic, OpenAI and Google models, dated per provider. Your own entries go on top of them.

> **Costs are estimates:**
> The built-in prices are a snapshot taken when this version of Cloak Proxy was made, not live prices, and some are many months old. Providers change prices and add models often, so treat costs as estimates, and edit the price of any model you rely on, in the price list or in `llm-prices.json` (below), to match your provider's current price page or your contract.

Open the list with **Prices…** in Insights › AI, by clicking **Cost** in the AI tab, or from the AI tab's right-click menu.

*[Screenshot: The Model prices sheet: a filter field, Add Model, Import and Export buttons, and a table of models with provider, input, output, cache read and cache write prices and their source.. Edit any price to make it yours.]*

| To | Do this |
|---|---|
| Change a price | Edit it in the table. The row becomes **Yours** and replaces the built-in entry. |
| Price another model | **Add Model**, then rename it and fill in the prices. A gateway's model ids, a fine-tuned model, or a local model at 0. |
| Go back to the built-in price | Click the arrow next to **Yours** |
| Share or back up your prices | **Export…** writes the whole list as JSON; **Import…** adds a file's entries to yours |

A model id matches the longest entry it starts with, ignoring routing prefixes: `gpt-4.1-mini-2025-04-14` uses `gpt-4.1-mini`, `openai/gpt-4o-mini` uses `gpt-4o-mini`, and `us.anthropic.claude-sonnet-4-6` uses `claude-sonnet-4-6`. An empty cache price means the input price.

Your entries are saved in `llm-prices.json` in Cloak Proxy's folder (`~/Library/Application Support/CloakProxy/`). The file can also be a simple map, which is easy to write by hand:

```json title="llm-prices.json"
{
  "my-gateway-model": { "input": 1.5, "output": 6, "cacheRead": 0.15 },
  "llama3.2": { "input": 0, "output": 0 }
}
```

> **What the cost leaves out:**
> Costs are list prices for the tokens the provider reported. Batch discounts, long-context surcharges, priority tiers, fast modes, server-side tools (web search) and anything your contract changes aren't included; add your own prices for those models.

## With an AI assistant

Three MCP tools read the same data:

| Tool | Returns |
|---|---|
| `llm_calls` | Each call's model, tokens, cost, time to first token, speed, stop reason, tool calls, errors and retries. Filter by `app`, `host`, `model`, `errors_only` or `since_id`. With `id`, one call in full: prompt, reply, tool calls with their results, the context budget, and `sinceLastTurn` (the first change and the text around it). |
| `llm_summary` | Totals by model, app and host, cache hit percentages, median and p95 time to first token, retries and models without a price |
| `llm_prices` | The price list. `set` adds or changes entries, `remove` takes yours out |

> Why did request #42 miss the prompt cache, and what did the agent's last run cost?

See [MCP tools reference](https://docs.subcloak.com/cloak-proxy/automation/mcp-tools-reference/#model-calls).

## In CI

`cloakctl llm` summarizes the model calls in a recorded session, and can fail a build that spent too much:

```bash title="Terminal"
cloakctl run --ssl '*' --record run.cloaksession -- npm test
cloakctl llm --session run.cloaksession --calls --max-cost 0.50
```

```text title="Output"
8 model calls · $0.011 · 6.3k input (2.5k cached) · 364 output · median first token 660 ms
  claude-sonnet-5-5              3     calls  $0.011
  gemini-2.5-flash               1     calls  $0.0004
#8001  claude-sonnet-5-5  1740 in (0 cached)  54 out  $0.0049  first token 660 ms  tool_use
```

| Option | What it does |
|---|---|
| `--session FILE` / `--har FILE` | Reads a saved session. Without it, asks the running app. |
| `--calls` | Also lists every call |
| `--json` | Prints the `llm_summary` result |
| `--max-cost DOLLARS` | Exits `2` when the total is above this, or when a model has no price (so the total can't be trusted) |
| `--prices FILE` | Uses a price file instead of your saved prices, so CI doesn't depend on your Mac |
| `ID` | One call in full, as JSON |

`cloakctl run --report FILE` adds an **AI** section to its Markdown report when the run made model calls.

## Privacy

Prompts, replies and keys never leave your Mac: the inspector only reads traffic Cloak Proxy already captured. API keys in headers are shown as captured, so use [redaction](https://docs.subcloak.com/cloak-proxy/sessions/redaction/) before you share a session. Prices are a local file; nothing is fetched.

## Related

  - [Streaming responses](https://docs.subcloak.com/cloak-proxy/inspect/streams/): Every SSE event of a reply, with its arrival time.
  - [SSL Proxying](https://docs.subcloak.com/cloak-proxy/capture/ssl-proxying/): Decrypt the model API's host.
  - [Insights](https://docs.subcloak.com/cloak-proxy/insights/overview/): Endpoints, errors, secrets and performance.
  - [Run API tests in CI](https://docs.subcloak.com/cloak-proxy/guides/run-api-tests-in-ci/): Record a test run and check it.