Tokens A working primer — what a token is, how it's priced, and where the margin lives.

0. Audience + anchor question

For a smart-finance reader orienting to AI inference token economics as pre-call material. The anchor question: what is a token, how is it priced, what's commoditizing vs. defensible, and where does margin live? After reading, you should be able to read a vendor pricing page, an inference earnings disclosure, or an agentic-workload cost analysis and place each number against the resource it stresses and the deflation curve it sits on.

1. What a token is

A token is a sub-word fragment of text — the actual unit that a language model reads in and writes out. Words, punctuation, and spaces are not the model's natural unit; tokens are. In English, one token averages about 0.75 words, or roughly 4 characters¹. Short common words like "the" or "and" usually map to a single token. Longer or rarer words split into multiple pieces — the word "encoding" might be split into "encod" + "ing"².

The splitter is called a tokenizer, and each model family ships its own. That matters: token counts are not directly comparable across providers. Anthropic explicitly flags that "Opus 4.7 uses a new tokenizer compared to previous models … this new tokenizer may use up to 35% more tokens for the same fixed text"¹. So a per-million-token price quote means slightly different amounts of text from one model family to the next.

2. How tokens are counted

"Counting tokens" means: hand a string of text to the tokenizer, get back the list of pieces it produces, and count them. Anthropic, OpenAI, and Google all publish token-counting tools and API endpoints that do exactly this. For example, sending the message "Hello, Claude" with a one-line system prompt to Claude Opus 4.7 is billed as 14 input tokens³ — more than the two visible words, because the system prompt and chat-format scaffolding are tokenized too.

Two separate things get called "counting":

• Counting tokens in a text — what the tokenizer does, above.
• Counting the bill — adding up input tokens × $/input-MTok plus output tokens × $/output-MTok for the period.

3. Input vs. output tokens

Every API call has two sides:

• Input tokens — everything sent to the model: system prompt, user message, prior conversation history, retrieved documents, tool results.
• Output tokens — everything the model generates back.

They are priced separately because they cost different amounts to produce on the underlying hardware. The input pass is called "prefill" and processes the whole prompt in parallel on the GPU — compute-bound. The output is "decode" and generates one token at a time, each step waiting on memory bandwidth to stream the model's weights — memory-bound and serial⁴. The practical consequence: output tokens are roughly 3–10× more expensive than input tokens at every major provider — visible in the rate cards below.

4. Are tokens "data" or "compute"?

Both. A token is a piece of data — a tiny string fragment — but every token, in or out, costs one forward pass through the model. Token throughput is the unit that links data flow to compute consumption: more tokens means more matrix-multiplies on a GPU, which means more electricity, more chip-hours, more revenue for the provider.

What "compute" actually is, for any individual LLM call, lives almost entirely inside the GPU — not the CPU. NVIDIA's own inference-engineering write-up puts it bluntly: during prefill the workload "effectively saturates GPU utilization," and during decode "the speed at which the data … is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens"⁵. The CPU is a side player for inference (it runs the request scheduler, the tokenizer, and — in an agent loop — the bits between LLM calls). The inter-GPU network matters most for training, and for inference only when a model is too big to fit on a single GPU and has to be sharded across several. So when we talk about the price of a token, we are really talking about how it stresses one of four resources inside the GPU, anchored here on an H100 SXM:

• Compute (math units, measured in TFLOPS). H100 SXM peaks at 1,979 TFLOPS at FP16 Tensor Core and 3,958 TFLOPS at FP8 Tensor Core, both with sparsity⁶. This is the matrix-multiply throughput — used heavily during prefill, when the model is processing all the input tokens in parallel.
• Memory bandwidth (HBM throughput, measured in TB/s). H100 SXM = 3.35 TB/s of HBM3 bandwidth⁶. This is how fast model weights stream from on-package high-bandwidth memory to the math units. Used heavily during decode: every output token requires streaming the full weights again, one token at a time.
• Memory capacity (HBM size, GB). H100 SXM = 80 GB; H200 SXM = 141 GB⁷; Blackwell B200 in the DGX B200 system = 1,440 GB total across 8 GPUs, i.e. 180 GB per GPU⁸. Bigger models need more capacity; so do longer conversations, because the running record of what's been computed so far — the KV-cache — also lives in HBM.
• Inter-GPU network (NVLink, measured in GB/s per GPU). 900 GB/s on the Hopper-generation NVLink in H100; 1,800 GB/s on the Blackwell-generation NVLink in B200⁹. Only relevant when a model is split across multiple GPUs and they have to exchange activations on every layer.

Not every accelerator uses HBM. Cerebras's wafer-scale WSE-3 puts 44 GB of on-die SRAM — on-die static RAM, memory built into the same silicon as the compute units, much faster than HBM but much smaller — directly on the wafer¹⁰, delivering ~21 PB/s of on-die bandwidth (roughly 6,300× the H100's 3.35 TB/s HBM3) but only ~150 GB/s of off-wafer I/O¹¹. The architecture trades model size and inter-chip bandwidth for raw streaming speed — a different point on the same throughput-vs-interactivity tradeoff covered in §5.

Same GPU, different bottleneck depending on which side of the request you're on. Prefill (input) is compute-bound — the math units do most of the work; the H100's 1,979 FP16 TFLOPS (sparse; dense ≈ 989) is the relevant ceiling. Decode (output) is memory-bandwidth-bound — for every output token, the entire model's weights have to be streamed from HBM through the math units, and the 3.35 TB/s bandwidth ceiling sets the speed. This is the underlying reason output tokens cost ~3–10× more than input tokens at the rate cards in §5 below: the decoder is the bottleneck.

Now agentic workloads. An agent step is a normal LLM call running on the same GPU — agents don't shift work off the GPU, they multiply both the volume of calls and the memory footprint per call. Two effects on per-token economics: (i) every additional turn in the conversation grows the KV-cache — the model's running scratchpad of "what I've already looked at" — which NVIDIA's inference guide notes grows "linearly with batch size and sequence length" and "can have a large memory footprint"⁵, so longer contexts and multi-step loops consume more of the GPU's 80 / 141 / 180 GB of HBM; (ii) an agent task is many sequential LLM calls, so total GPU-hours per task runs hundreds to thousands of times higher than a single chatbot reply (Stanford's measurement is in §8). At the cluster level the picture also shifts — Microsoft's Fairwater AI campus pairs the dense GPU building with a separate CPU-and-storage building including "millions of CPU cores for operational compute tasks"¹² (independently corroborated by Glenn Lockwood¹³), and AMD/Intel/Meta IR disclosures all flag agentic-driven CPU pull-through for orchestration, retrieval, KV-cache management, and tool-execution. Per-token economics are GPU-led; per-cluster capex pulls CPU back in as a meaningful slice — covered in depth in the agentic AI primer §4.

The pricing model in §5 doesn't try to reflect any of this directly. Providers quote a flat $/MTok within a SKU; the only built-in asymmetry is the input/output split, which is what captures the prefill-vs-decode cost gap. The closest thing to "pricing the resource mix" is Google's long-context surcharge on Gemini 2.5 Pro: prompts above 200K tokens are billed at 2× on input and 1.5× on output (input $1.25→$2.50, output $10.00→$15.00)¹⁴. That surcharge is essentially a tax on KV-cache pressure — long contexts consume disproportionately more HBM per call, and the provider passes that back through the rate card.

5. How tokens are priced

Pricing is quoted in dollars per million tokens ($/MTok), separately for input and output. Below is a snapshot of frontier and commodity tiers from the three largest API providers, May 2026 (all figures verified on each vendor's pricing page on 2026-05-18):

Two structural levers cut these rates substantially:

• Batch processing — submit a queue of requests with a 24-hour service-level guarantee instead of real-time. Anthropic, OpenAI, and Google all offer a flat 50% discount on both input and output for batch¹.
• Prompt caching — when the same prefix (a long system prompt, a document, a conversation history) is reused, the provider serves it from cache. Anthropic charges cache reads at 10% of the standard input rate — a 90% discount on the cached portion¹.

There's also a third pricing axis beyond input/output and frontier-vs-commodity: speed tier within a single model. Anthropic sells a Fast Mode on Opus 4.6 and 4.7 at $30 input / $150 output per MTok — 6× the standard $5/$25 rate¹ — for roughly 2× the measured interactivity. SemiAnalysis, an AI-infrastructure research firm, disclosed that "80% of our AI spend (which peaked at $10M annualized) was on Opus 4.6 fast" in April 2026¹¹ — that's a single-firm data point, not a market-wide claim, but it shows real willingness to pay a steep premium for speed at a given intelligence level. Mechanically, faster service means the provider dedicates more GPU resources to fewer concurrent requests — less batching, more KV-cache headroom per user, lower decode-bandwidth contention — so the same $5/MTok Opus weights, served at higher tokens-per-second-per-user, costs 6× more. For workloads where latency matters (live chat, agentic loops), the effective $/MTok can sit well above the rate-card headline.

6. Why tokens are the unit at all

Tokens are the natural unit because they are what the model literally consumes and produces, and compute scales linearly per token — one forward pass per output token. Alternative units (characters, words, API calls) don't track cost cleanly: a 10-word answer and a 10-paragraph answer cost wildly different amounts; the token count tracks the difference exactly.

7. The deflation curve — bimodal

Token economics has two simultaneous price patterns that look contradictory but aren't:

• Iso-capability deflation (the commodity tier collapses). At a fixed level of model capability, the price falls fast. Andreessen Horowitz's "LLMflation" analysis documents a clean trend: "For an LLM of equivalent performance, the cost is decreasing by 10× every year"¹⁶. The cleanest anchor: GPT-3 was $60/MTok at launch in November 2021; by 2024, a Llama 3.2 3B model matched its MMLU score at $0.06/MTok — a 1,000× drop in three years¹⁶. A March 2023 → August 2024 window for GPT-4-class models works out to about 79% per year¹⁷. The NBER paper "The Emerging Market for Intelligence" (Demirer, Fradkin, Tadelis, Peng, Dec 2025) finds that open-source models are roughly 90% cheaper than closed-source models at the same intelligence level, and documents the same trend across tiers¹⁸.
• Frontier flat. Top-of-stack pricing has barely moved. Anthropic held Opus at $5 input / $25 output from Opus 4.5 → 4.6 → 4.7 — same rates across three consecutive frontier releases¹. OpenAI's flagship has actually crept up (GPT-5.5 output is $30/MTok). The frontier doesn't deflate; the commodity tier underneath it does, because last year's frontier becomes this year's commodity.

SemiAnalysis goes further on the frontier side: in agentic workloads, sticker pricing understates a different dynamic — the realized blended rate on Opus 4.7 agentic tasks lands around $0.99/MTok (90%+ cache hits at $0.50/MTok, 300:1 input-to-output) while Anthropic's introduced premium SKUs (Opus Fast at 6×, Mythos at 5× regular Opus pricing) that buyers pay willingly because productivity gains exceed the cost — implying the market-clearing token price for frontier-quality work has moved up, not down. See the SemiAnalysis counter-thesis.

8. The agentic multiplier

"Agentic" AI systems — autonomous multi-step workflows where the model uses tools, browses, writes and runs code, and chains its own reasoning — consume orders of magnitude more tokens per task than a single chatbot reply. Each step in the agent's loop is itself an LLM call, and each step's output becomes the next step's input, so token counts compound. The Stanford Digital Economy Lab's May 2026 study of agentic coding workloads (Brynjolfsson, Pentland, Pei) found agentic tasks "consuming 1000x more tokens than code reasoning and code chat" — and input tokens dominate the spend, because the agent re-reads the full conversation history on every step¹⁹.

That per-task multiplier shows up in the aggregate. NVIDIA CEO Jensen Huang, at Morgan Stanley TMT (March 2026) and Milken (May 2026), framed the industry-level consequence: "computing scale, which is proportional to token usage, has grown 1,000-fold in just the past two years"²⁰ (reporting on the Morgan Stanley fireside chat). (Morgan Stanley's own write-up of the same fireside chat²¹ frames the same dynamic with a different anchor — "Agentic AI can consume 1 million times more tokens than a standard generative prompt" — but does not carry the specific verbatim "1,000-fold in two years" wording; SE Daily remains the workable citation for that exact quote.)

The two trends in section 7 plus the multiplier here are why total token spending grows even when per-token prices fall: volume is growing faster than price is declining. That is the entire story of token economics in one sentence.