Economics of LLM Inference: Why Your Finance Director Should Know the Difference Between Prefill and Decode

Two approaches for processing a single request

LLM inference is not a monolithic process but two technologically incompatible phases. Prefill (processing the incoming prompt) is the peak computational load. The model loads and analyzes the entire input context, creating a so-called KV cache. This is a highly parallel operation that loads GPU tensor cores to 90-95%.

Once the model begins generating the response token by token, Decode begins. This is a completely different process—sequential and limited by memory bandwidth. GPU utilization during this phase drops dramatically to 20-40%, or even 15-30%.

This is precisely why the economics of a "flat rate per token" are fundamentally flawed. A request with a 2000-token prompt and a 50-token response consumes significantly more GPU resources than a request with a 10-token prompt and a long 2000-token generation, even though the total number of tokens is comparable.

Hardware decides: speed at peak hour

Linking both phases to the same expensive hardware is like using a supercar to deliver mail during peak hour. This is how it looks in the GPU market:

NVIDIA H100 (80GB) — a flagship costing $25-35 thousand per card. Its massive tensor cores and Transformer Engine architecture are ideal for fast Prefill, but they sit idle during Decode, limited by memory bandwidth.

NVIDIA L40S — costs $8-12 thousand. Less powerful, but with excellent memory bandwidth, and it has already captured over 30% of the market specifically in the inference segment. This is an ideal candidate for the Decode pool.

In the cloud, an H100 costs from $2 to $13 per hour, while an L40S is around $1 per hour, or even less.

Next-generation frameworks — vLLM, SGLang, TensorRT-LLM — have already learned to separate Prefill and Decode into different GPU pools. The result: Prefill is sent to a pool of powerful H100s, while infinite Decode goes to a pool of budget L40S. Overall, such disaggregation reduces infrastructure costs by 15-40% and increases throughput by up to 6.4 times.

Models and token economics

The laws of unit economics are also evident in the pricing of APIs from leading vendors. Note the disproportion:

GPT-4o: $2.50 for 1 million input tokens (Prefill) and $10.00 for 1 million output tokens (Decode).

Claude 3.5 Sonnet: $3 for input and $15 for output.

Llama 3 70B (on dedicated hosting): $0.23 for input and $0.40 for output.

DeepSeek R1: self-service at the hardware cost or $0.55 for input and $2.19 for output.

The 3-5 times difference between the price of input and output tokens is due to an objective reality: Decode is the most inefficient stage in terms of hardware usage, where resources idle. Therefore, applications with long dialogues and a small incoming context are a financial nightmare for businesses, while summarizing large documents (Prefill-heavy) turns out to be significantly more economical.

Where optimization yields the maximum effect

The cheapest token is the one that doesn't need to be computed. The technology of "token warehouses" and prefix caching allows pre-filling (Prefill) the KV-cache for repetitive parts of prompts — for example, for huge system instructions or identical product descriptions. This can improve the time to the first token by 75 times and reduce energy consumption by 200 times.

Practice shows: a company operating in SERM and processing up to 300 reviews per day can spend up to 1500 tokens on repetitive system context with each request, leaving only 100 tokens for a useful response. Without caching, this leads to a transaction efficiency of just 6% and astronomical API bills.

Calculating unit economics

The LLM inference market is developing faster than any other technological segment in history: the cost of processing a million tokens is falling 10 times annually. The equivalent performance of GPT-4 now costs $0.40 compared to $20 at the end of 2022.

To ensure that business process automation doesn't turn into a financial disaster, it's necessary:

To count not an abstract "token," but to attribute the cost of GPU to specific phases of work. The cost attribution model within a batch should consider how many milliseconds of GPU time were consumed by Prefill and how much by Decode of a specific request.

To design a hybrid architecture: powerful GPUs for Prefill (H100/A100) and budget-friendly ones (L40S/L4) for the Decode pool.

To implement prefix caching. This will have an effect comparable to purchasing another dozen expensive graphics cards.

Ignoring the differences between Prefill and Decode in 2026 is not technical debt, but direct financial incompetence. The LLM economy has long ceased to be token magic and has become strict practice of managing heterogeneous computing resources.