Software companies are trained to think of marginal cost as nearly zero. Write the code once, add customers, and let shared infrastructure turn distribution into margin. Generative AI unsettles that habit. Every answer occupies accelerators, moves data through memory, consumes electricity, produces heat, and competes for constrained capacity. The interface may look like software, but the cost structure increasingly resembles an industrial process.
This does not mean AI products are doomed to poor margins. It means their economics will be determined less by the list price of a model API and more by the design of the entire serving system. A company that treats inference as an interchangeable utility may discover that its most popular feature is also its least defensible business.
The factory is hiding behind the endpoint
An API call conceals a long chain of capital: chips, servers, networking, substations, cooling equipment, buildings, grid connections, and operating staff. The International Energy Agency's Energy and AI report makes the physical dependency plain. Data centers are becoming significant new participants in electricity systems, with local effects that can be much sharper than their share of global consumption suggests.
That concentration changes the industry conversation. A model provider can improve computational efficiency while total demand still rises because cheaper inference invites more usage, longer contexts, additional reasoning steps, and autonomous agents that work continuously. Efficiency is economically valuable, but it does not automatically reduce infrastructure requirements. It often converts a scarce luxury into a mass-market input.
OpenAI's description of building large-scale compute infrastructure reads less like conventional software expansion than ecosystem construction: energy, finance, construction, operations, and supply chains must move together. Whatever one thinks about the projected scale, the category shift is already visible. AI strategy now reaches into domains that product organizations once treated as someone else's plumbing.
Utilization is the hidden competitive variable
The simplistic metric is cost per token. The useful metric is cost per successful unit of customer work, measured across the full load profile. That includes unused reservations, retries, failed agent runs, oversized context windows, retrieval, moderation, evaluation, and the human labor needed when automation fails.
Two products using the same model can therefore have radically different economics. One batches predictable document jobs, caches repeated context, routes simple requests to smaller models, and allows flexible completion times. The other promises instant responses, sends every task to its largest model, and gives an agent an open-ended loop. Their nominal model prices may match. Their effective costs will not.
Utilization also creates a tension between product experience and infrastructure efficiency. Users want low latency at peak moments; operators want expensive capacity working steadily. The gap is paid for somewhere. Mature AI products will make that trade visible through product design: asynchronous jobs where immediacy adds little, explicit effort settings, bounded agent budgets, and pricing that reflects computational intensity.
Model choice is becoming workload engineering
The winning architecture will rarely be one model everywhere. It will be a portfolio governed by routing rules and evidence. Small models can classify, extract, check formats, or decide whether a more capable model is needed. Deterministic code can handle arithmetic, validation, permissions, and state transitions. Large models can be reserved for ambiguous reasoning where their flexibility changes the outcome.
This is not merely an optimization pass performed after launch. It shapes the product. If a feature cannot define what success looks like, the team cannot route intelligently or measure cost per successful task. If every response must preserve an enormous conversation transcript, context becomes an accumulating tax. If an agent cannot stop itself, each new capability is also permission to spend.
The same logic applies to ownership. Product managers need to understand workload shape; engineers need visibility into financial and energy consequences; finance teams need metrics tied to actual task completion rather than aggregate token volume. Infrastructure decisions made without product semantics waste capacity. Product promises made without infrastructure knowledge destroy margins.
The moat is operational learning
Access to capable models will continue to broaden. Operational knowledge will remain specific. A company learns which requests tolerate delay, which context can be cached, which tasks merit a premium model, which failures trigger expensive retries, and where human review prevents larger losses. Those observations accumulate into routing policies, evaluation suites, and pricing design.
This suggests a different way to read AI capital spending. Large infrastructure commitments are not proof that every application will succeed. They are evidence that the industry expects computation to become a major production input. Application companies still have to convert that input into valued work with discipline.
The strategic question is no longer simply whether an organization has access to intelligence. It is whether the organization can schedule, constrain, and reuse that intelligence better than its competitors. The electrical room is now part of the software architecture, even when the product team never sees it.
Advertisement
