Inference tier: why RTX Pro 6000D is bought

The line is drawn at a capability tier, not at a border

The Financial Times reported on 13 October that Chinese technology companies bought more than a million of Nvidia’s gaming and workstation chips this year, mostly RTX 5090 and RTX Pro parts, to carry basic AI workloads (FT report). These are not smuggled units. They sit in the tier that export rules never blocked, and the same article names the qualifying parts as the RTX Pro 6000D and RTX Pro 5000.

The line, in other words, is drawn at a capability tier rather than at a national border. Everything below the threshold gets bought as compute.

Evidence strength first

The FT piece sits behind a paywall and we did not read the original. The model mix and the million-unit figure come from secondary relays (Tikr, Startup Fortune). That belongs at the top, because it decides which numbers can carry an argument and which are background.

The supply and demand estimates come from industry people cited in that coverage: roughly 3.5 million domestic AI chips this year, including about one million fabricated overseas, against demand expected to exceed 5 million. On spending, combined capital expenditure at Alibaba, ByteDance and Tencent may more than double this year to about 700 billion yuan. Morgan Stanley puts China’s 2026 compute spend at 193 billion dollars, roughly 21 percent of the comparable US figure. That last one is a bank estimate, not an FT number.

Reports elsewhere describe a detour. Tencent contracted for about 100,000 advanced AI chips through an Oracle data centre in Southeast Asia.

An independent reading comes from IDC, relayed by Reuters: Nvidia’s AI chip shipments in China fell 44 percent year on year in the first half of 2026 to 678,000 units, with its share dropping to 34 percent as Huawei narrows the gap (Reuters).

What those cards really buy is one phase of inference

Inference is not one action. It has two phases. Prefill computes the entire input in one pass: compute-heavy, and cacheable. Decode emits one token at a time: memory-bandwidth-heavy, and largely uncacheable.

A card behaves differently in each phase, and costs differently, which explains an apparent contradiction. Why do parts that are slower than Huawei’s Ascend 910C on most tasks still get bought by the pallet.

The answer is that they do not need to win every task. They need to be adequate for the simpler work, and prefill is exactly that. It also shows that adequate is a relative word: what it is measured against is the workload, not a ranking.

Why hardware choice is really a ratio question

The same split explains why the hardware question gets asked as a capability question. It is a ratio question. How much of your traffic is prefill, and how much of it can be reused, decides whether the sub-threshold tier is enough.

If a large share of requests share a prompt prefix, prefill can be cached and recomputed rarely, so the cheap tier carries the load. If every request arrives with a different prefix, the money goes into recomputing the same phase over and over. A more expensive card then computes that phase faster without making it smaller.

So the order should be workload shape first, hardware second. Reversed, you are guessing at a business from a quotation sheet.

We meter the two phases separately

On our own inference path we meter the two phases separately. The reason is plain. Blended into a single cost per million tokens, savings and waste cancel each other out and you lose sight of where the money goes.

The number we watch is prefix cache hit rate, because it determines whether the same prefix is paid for twice. A high hit rate means the cheap tier suffices. A low one means the bill is dominated by repeated prefill, where a different card changes nothing and prompt structure plus caching policy change everything.

That rule comes with a precondition worth stating. It holds when the workload can be split at all. What cannot be split is a separate case, which the next section covers.

Turn procurement into three numbers

To use this in the next compute budget, break cost into three attributable quantities.

• Prefill compute cost per million tokens.
• Decode bandwidth cost per million tokens.
• Prefix cache hit rate, which decides how many times the first number is paid.

Only then does a comparison mean anything. A cheap tier with a high hit rate can beat an expensive tier with a low one. Reverse the shape and the same logic bites. When decode dominates, pushing the workload onto bandwidth-starved cards gives back in latency what was saved on price.

That is why we move the criterion from unit price to attributable cost. A unit price compares one instant. An attributable cost compares a whole workload.

The mismatch keeps reopening

Drawing the line at a capability tier means the threshold moves whenever new silicon ships. A tier that is out of reach today may sit below the line next year, which keeps reopening the route of serving simple workloads on whatever tier can be bought.

Demand pushes from the other side too. Agent outputs are typically longer than chat replies, so decode grows as a share of the workload, and decode spends memory bandwidth.

Put the two forces together and those three numbers only become more important.

Boundaries

This reasoning covers only the phase that can be moved. Gaming and workstation cards lack ECC and high-bandwidth memory, and they change rack density and cooling requirements. They replace neither training nor latency-sensitive decode.

The supply picture is still moving. At the same event on 12 October, Intel’s chief executive judged that hardware will stay in short supply for another two to three years, with memory the first segment to lose momentum. That is a judgement rather than a recorded fact.

Broadcom’s chief executive went further, saying many investors do not know what they are invested in and that financing may prove a bigger problem than the chip shortage (Yahoo Finance). That is market commentary, not a demand measurement, though it points the same way as what banks are already doing with underwriting.

None of those voices changes the earlier judgement. The million-plus cards bought below the threshold cannot do frontier training. They fill a gap at the entry level. Retail pricing illustrates the same point from the other side, with reports that RTX 5090 street prices have risen above launch prices because the buyers are institutions treating them as fixed assets rather than gamers.

A concrete reading: which part of the bill to open first

If you want to act now, the order can be simple. Take last month’s model service bill and split usage into input and output. The input side corresponds to prefill, the output side to decode.

Then ask two questions. How much of the input volume is repeated prefix. And whether that repeated part is cached, so that it is not billed twice.

Those two answers usually save more money than a hardware negotiation. They cut the amount of computation directly, while a card swap only changes the price of a unit of computation.

Whose decision this changes

For an AI application lead it lands on one bill: split model service cost into prefill and decode, look at hit rate first, and only then discuss hardware.

For infrastructure and transformation teams, renting capacity across regions and mixing tiers is a capacity planning problem. The criterion is the shape of the workload each tier carries, not a preferred part number.

For management, the procurement criterion moves from lowest unit price to whether the spend can be attributed to an output. That is also the criterion the financing side now applies.

Where to start

Before the next model service budget, work out three numbers: prefill cost per million tokens, decode cost per million tokens, and prefix cache hit rate. Without them, tier selection has no basis.

The negotiation then has exactly one starting point, the unit price on the vendor’s quotation.

Why a quotation sheet hides this

Cloud pricing is quoted per token, so the bill looks like a single unit price. Prefill and decode have different cost structures though, one spending compute and the other spending bandwidth. This means the same quotation produces very different real costs under different workload shapes.

Compare vendor A with vendor B on unit price and you compare two sets of assumptions, not your own cost. In other words, a comparison without your workload shape is a comparison of someone else’s business.

That is also why we insist on the three numbers first. Without them, every figure on the negotiating table is unverifiable and you simply accept the other side’s framing.

Getting the three numbers in two weeks

If you want to start now, four steps will do. First, count input and output separately at the gateway, so the bill stops giving you one blended total. Second, switch on prefix caching and record the hit rate. Third, run a controlled comparison on real traffic, once with caching on and once with it off. Fourth, write the result into a one-page capacity plan.

The step people skip is the third. The reason is that the difference only shows up on real traffic. Synthetic data usually has no repeated prefixes, so the measurement comes out more pessimistic than reality.

What you end up with is one number, the cost gap between the two configurations on the same workload. That gap is the starting point for a negotiation, and the only figure you can put in front of a vendor.

What cannot be split

The earlier rule has a precondition: the workload can be split at all. Three workload types fail that precondition. Long-context one-shot question answering has almost no prefix repetition and a long output. Real-time interaction cares more about latency than unit price. Multi-step tool chains that need strict ordering cannot recompute intermediate state.

In other words, buy those workloads however they need to be bought, and do not push them onto a bandwidth-starved tier to save on unit price. The savings return as latency and retries.

That boundary is not an exception, it is the other half of the same question. You do not need one rule for every workload. You need to know which kind you are holding.

References: Financial Times report, 13 October 2026 (paywalled; read via secondary relays this batch), https://www.ft.com/content/9dbd99db-6b0c-43bd-b7be-849f129df098; secondary relays https://www.tikr.com/blog/chinese-tech-firms-bought-1-million-nvidia-gaming-chips-this-year-heres-where-it-shows-up and https://startupfortune.com/ai-firms-are-buying-rtx-5090-gaming-cards-by-the-pallet-pricing-out-gamers/; IDC shipment data via Reuters https://www.reuters.com/business/retail-consumer/nvidias-china-ai-chip-share-falls-huawei-closes-gap-report-finds-2026-10-13/; executive comments via https://finance.yahoo.com/technology/ai/articles/broadcom-ceo-likens-many-ai-095852496.html. Figures labelled as industry or bank estimates are third-party readings, not official statistics.