Engineering, Venturi TS
AI Tech Talk 03 | October 16, 2026
AI Tokenomics: What It Really Costs to Go From One Prompt to an AI Factory
Most conversations about AI infrastructure start with the wrong question: how many GPUs do we need? Nobody can answer that without first knowing what the system has to do. In our third AI Tech Talk, we walked through AI economics the other way around, starting from a single prompt and building up to a full AI factory, then used the Venturi AI Factory Sizer live to show how each decision changes the cost.
The short version: an AI factory is defined by its function, not its size. The same economics apply whether you are serving five clinicians in a small practice or tens of thousands of employees across time zones.
The short version: an AI factory is defined by its function, not its size. The same economics apply whether you are serving five clinicians in a small practice or tens of thousands of employees across time zones.
Where We Left Off: What is a Token
Tokens are the unit of measurement.
A token is the smallest unit AI models work with, roughly four characters of text. A word can be one token or several.
A simple question to an internal assistant, such as "What is our parental leave policy?", might be around 100 tokens as typed. But what actually reaches the model is much larger. Every request carries the conversation history, any documents retrieved to answer it, user and policy context, and guardrails. A 100-token question can easily become thousands or tens of thousands of tokens moving through the inference pipeline.
A token is the smallest unit AI models work with, roughly four characters of text. A word can be one token or several.
A simple question to an internal assistant, such as "What is our parental leave policy?", might be around 100 tokens as typed. But what actually reaches the model is much larger. Every request carries the conversation history, any documents retrieved to answer it, user and policy context, and guardrails. A 100-token question can easily become thousands or tens of thousands of tokens moving through the inference pipeline.
Today's Path: From One Prompt to the Sizer
This talk followed one path, and the rest of this page follows it too: a prompt, the tokens it creates, what happens at scale, what an AI factory is made of, what a token costs to produce, and finally how the Venturi AI Factory Sizer puts all of it together. Each step on its own looks small. Multiplied by real usage, it becomes enterprise economics.
From Prompt to Token Cost
Every AI cost starts the same way. A user sends a prompt: a question, an instruction or a step in a workflow. That prompt, plus its context and history, becomes the input tokens. The model does its work in prefill and decode, and produces output tokens, the generated response.
The economics are the last step. If you consume AI through an API, cost is token volume times the provider's rate. If you own the infrastructure, cost is what it takes to produce those tokens yourself. Newer models accept much larger inputs and produce longer outputs, which changes this math again.
The economics are the last step. If you consume AI through an API, cost is token volume times the provider's rate. If you own the infrastructure, cost is what it takes to produce those tokens yourself. Newer models accept much larger inputs and produce longer outputs, which changes this math again.
The Multiplier is the Story
One prompt plus one answer is a single interaction, a few thousand tokens. Then real usage arrives.
Production workflows add more context and more calls. Departments adopt the tool at the same time, so more users hit it at once. A global company runs it follow-the-sun, so concurrency is different in each region and peaks move around the clock. Every enterprise workflow also brings a body of documents with it, from employee handbooks to engineering documentation and source code. At enterprise scale, the demand is persistent and material.
The cost formula is simple to state: token volume times usage scale times model economics equals total cost. The hard part is that every term in that formula moves.
Production workflows add more context and more calls. Departments adopt the tool at the same time, so more users hit it at once. A global company runs it follow-the-sun, so concurrency is different in each region and peaks move around the clock. Every enterprise workflow also brings a body of documents with it, from employee handbooks to engineering documentation and source code. At enterprise scale, the demand is persistent and material.
The cost formula is simple to state: token volume times usage scale times model economics equals total cost. The hard part is that every term in that formula moves.
You Pay for Intelligence Twice
Microsoft CEO, Satya Nadella, put it sharply this summer: you essentially pay for intelligence twice, once with money and again with something more valuable, the proprietary knowledge you must reveal to make that intelligence useful.
Both costs are real. Frontier model providers benefit from every question you send them, and many organizations are already leaking sensitive information through what amounts to shadow AI: employees pasting company data into public tools because it is the easy option. The financial cost is visible on the monthly bill. The knowledge cost is not.
Both costs are real. Frontier model providers benefit from every question you send them, and many organizations are already leaking sensitive information through what amounts to shadow AI: employees pasting company data into public tools because it is the easy option. The financial cost is visible on the monthly bill. The knowledge cost is not.
The AI Factory Bill of Materials
To produce tokens yourself, you need a working system, not just GPUs. At a high level, and only as a frame of reference rather than a quote:
- Servers: CPU, memory, storage and chassis, roughly $200,000 each, and higher than a year ago because of DRAM supply.
- GPUs: inference capacity and model memory, roughly $30,000 to $60,000 each, typically 2 to 8 per server.
- Networking: moves data and keeps work synchronized, roughly $200,000 for a single 48-port high-speed switch, with 800 Gb and 1.6 Tb switching now common.
- KV cache: retains useful inference state, roughly $500,000 for 200 TB of shared flash.
- Operations: power, cooling, software and support, roughly $1 million over five years at modest rack scale.
Roughly 70% of the Cost is Not GPUs
GPUs are the largest single line, but not the majority. In a typical factory, GPU compute is about 30% of the cost, flash (NAND) about 25%, software and runtime about 15%, and storage, networking, and power and cooling about 10% each.
Flash has caught up fast because of how much inference state modern workloads keep. The takeaway for planning: pricing GPUs alone prices less than a third of the system.
Flash has caught up fast because of how much inference state modern workloads keep. The takeaway for planning: pricing GPUs alone prices less than a third of the system.
How to Measure AI Economics
The slide lists five measures. Here is what each one tells you.
- Dollars per million tokens: the standard unit, split into input, output and blended rates. On frontier models, output tokens carry the higher price, on Anthropic roughly five times the input rate, which is why answer-heavy and agentic workloads get expensive quickly. GPU rental is often quoted per GPU per hour, but that does not tell you what a finished answer costs.
- Speed and latency: time to first token is how long a user waits before the answer starts appearing; latency is how long the full response takes.
- Tokens per second per watt: the efficiency of the factory at full load.
- Goodput: the share of output that is actually useful. A fast wrong answer is not a good answer, and context quality and model choice both drive it.
- GPU utilization: chasing 100% is usually the wrong goal. A business-hours workload can never reach it, and a system run near its ceiling has no headroom for peaks.
We are open about speed: an on-premises factory is not automatically faster than frontier models in the cloud, which are very fast. Some workloads, like a voice agent talking to a person, cannot tolerate delay. Others can accept a slower, cheaper service tier.
Moore's Law and Tokenomics: Two Truths
NVIDIA has said it is aiming for gains far beyond the traditional Moore's Law pace of roughly doubling every two years. That produces two practical rules.
First, the latest architecture at scale usually produces the lowest cost per token. Each new GPU generation tends to deliver about twice the performance for less than 1.5 times the cost.
Second, density pays. Because most of a factory's cost is not GPUs, packing more GPUs into each unit of deployment spreads that fixed infrastructure cost over more output. We call it reaching infrastructure escape velocity.
First, the latest architecture at scale usually produces the lowest cost per token. Each new GPU generation tends to deliver about twice the performance for less than 1.5 times the cost.
Second, density pays. Because most of a factory's cost is not GPUs, packing more GPUs into each unit of deployment spreads that fixed infrastructure cost over more output. We call it reaching infrastructure escape velocity.
Cost Becomes a Production Model
Read the slide left to right. The workload determines the users, context and concurrency. The model determines size, precision and behavior. Those drive the factory: GPUs, servers, networking and cache. The factory determines production, in tokens over time against a service target. And production determines unit economics: cost per million tokens and five-year cost.
Once cost is a production model, it can be planned like any other production line.
Once cost is a production model, it can be planned like any other production line.
Change One Variable. Recalculate the System.
Change any one input and the whole chain moves. Swap the model late in a design and you have a different memory footprint, different throughput, a different bill of materials, a different price and a different total cost of ownership for the finance team. Done by hand, that is days of rework.
Usage patterns matter as much as user counts. A developer running a coding sidecar all day loads the system very differently from someone asking a chatbot one question.
Cache reuse matters too. Context repeated across users is roughly 30% for a chatbot, 40% for code generation, and 80 to 90% for agentic workloads with looping agents. Every point of reuse is work the GPUs do not have to repeat.
Usage patterns matter as much as user counts. A developer running a coding sidecar all day loads the system very differently from someone asking a chatbot one question.
Cache reuse matters too. Context repeated across users is roughly 30% for a chatbot, 40% for code generation, and 80 to 90% for agentic workloads with looping agents. Every point of reuse is work the GPUs do not have to repeat.
Why Venturi Built the AI Factory Sizer
We built the Venturi AI Factory Sizer because the usual starting point, a GPU count, is backwards. The Sizer does four things:
- Starts with the workload: business objectives, users, context and growth.
- Makes assumptions explicit, marked as known, estimated or unknown.
- Tests sensitivity, so you can see what changes demand and cost.
- Connects the conversation, from business requirements to architecture decisions.
Consume, Rent, or Own
There are three ways to get AI today:
- Consume: frontier models through an API. Fast to start, a variable expense, and the widest model choice. It can also balloon; we see local companies spending several hundred thousand dollars a month on tokens.
- Rent: dedicated capacity from a hosting provider. It fixes much of the cost problem, with flexible commitments and simple operations.
- Own: your own AI factory. You get control, the upside of your own utilization, and a strategic asset.
Consuming and renting share one weakness: when you stop paying, you have nothing left. If AI is part of how your business makes money, losing access is a real liability. Only ownership leaves you with an asset and control over your own intelligence.
Read more about Consumption vs. Capability.
Renting the Building Isn't the Problem
An analogy from the talk makes the difference concrete. Renting a building for a restaurant is completely normal. If you keep your recipes, your trained staff and your customer relationships, you can move and the business comes with you.
Now imagine the landlord held your recipes, trained your staff and knew your customers. Moving out would no longer be simple, because you could not take much of your business with you. That is the real risk in renting AI: compute is the building, but intelligence is increasingly the recipes, processes and institutional knowledge inside it.
None of this means ownership always wins. Small or irregular workloads often belong on an API. Steady, high-volume, context-heavy or sensitive workloads are where owning tends to pay off. The workload decides.
Now imagine the landlord held your recipes, trained your staff and knew your customers. Moving out would no longer be simple, because you could not take much of your business with you. That is the real risk in renting AI: compute is the building, but intelligence is increasingly the recipes, processes and institutional knowledge inside it.
None of this means ownership always wins. Small or irregular workloads often belong on an API. Steady, high-volume, context-heavy or sensitive workloads are where owning tends to pay off. The workload decides.
Key Takeaways
- A prompt creates input and output tokens.
- Enterprise usage multiplies token demand.
- The factory bill of materials determines owned production cost.
- Small input changes can alter the whole economic model.
- The Sizer makes assumptions and sensitivity visible.
The Live Demo: 9,600 users, $4 million vs. $18.42 million
In the live demo, we sized a real workload in the Sizer with five questions. About 9,600 named users, with roughly 1,400 active at once, produced a factory of two GPU servers with 16 GPUs in total, handling about 3 billion tokens a day.
The five-year total cost was about $4 million. Running the same workload on frontier models at published prices came to about $18.42 million. Spread over five years of tokens, that works out to well under a dollar per million tokens for the owned factory, about $0.73 by our arithmetic.
The Sizer is honest in both directions. You will also model workloads where owning is not cheaper and the break-even never arrives.
The five-year total cost was about $4 million. Running the same workload on frontier models at published prices came to about $18.42 million. Spread over five years of tokens, that works out to well under a dollar per million tokens for the owned factory, about $0.73 by our arithmetic.
The Sizer is honest in both directions. You will also model workloads where owning is not cheaper and the break-even never arrives.
What the Sizer Does Today, and What's Next
- Workload planning, architecture modeling, storage analysis and growth.
- Full tokenomics, including how much money context reuse saves.
- Industry-specific use cases and executive summaries.
- Import of Anthropic and Fireworks usage exports, to compare your current spend with an equivalent owned factory.
- A 24-hour simulation stack that layers workloads to show peak demand, GPU utilization and spare capacity.
- Engineering detail on GPU memory, weights and KV cache, including the exact turn at which GPU memory runs out.
- A KV cache simulator. In one example, time to first token fell from 1.5 seconds to 191 milliseconds when prior work was served from cache.
- Storage tiers, multi-tenant designs, facility and power planning, and delivery models for teams with established IT or for a fully managed greenfield deployment.
- A micro-factory sizer for very small teams of 10 people or fewer.
Coming soon: a validation and benchmark runbook to prove results on purchased hardware before production, and operational feedback and calibration.
Demo the Venturi AI Factory Sizer
To explore sizing your AI workflows, please book a Sizing Demo. You can import current cloud-spend, size by budget, and download executive summaries with the sizing specs, materials, power, and potential monetization opportunities with a private AI factory. We update the Sizer continually and want your feedback on assumptions, numbers and new use cases.
