Software teams are accustomed to treating infrastructure as elastic. A successful feature attracts traffic; capacity expands behind it; unit costs generally improve with scale. AI weakens that comfortable sequence. The most capable services depend on accelerators, power, cooling, networking, and datacenter construction that must be planned long before a product manager knows which use case will win.
This is not merely a cloud bill getting larger. It is a change in the shape of product strategy. When useful intelligence consumes a constrained physical resource, every roadmap becomes a capital-allocation plan in disguise.
Tokens are claims on machinery
Teams often discuss model usage in abstract units: tokens, requests, context windows, or agent runs. Each unit is ultimately a claim on a machine, and increasingly on a surrounding industrial system. The International Energy Agency’s work on energy and AI is valuable precisely because it places computation back inside the power system. Semiconductor and infrastructure analysis from SemiAnalysis similarly emphasizes that AI progress is inseparable from packaging, memory, networking, clusters, and supply chains.
That physicality creates lead times software culture is poorly trained to see. A feature can be prototyped in a week, while the capacity needed to serve it at scale may be governed by equipment orders, grid connections, construction schedules, and multi-year commercial commitments. Product discovery operates in days; infrastructure arrives in quarters or years.
The mismatch encourages a predictable error: subsidize broad usage now and assume efficiency improvements will rescue the economics later. Sometimes they will. Better models, quantization, caching, batching, and specialized hardware can dramatically lower the cost of a task. But efficiency gains do not automatically become savings. They often invite larger contexts, more retries, deeper reasoning, and entirely new workloads. Cheaper intelligence can increase total demand faster than it reduces cost per operation.
Gross margin is too blunt
A conventional software dashboard asks whether a customer or feature clears a target gross margin. An AI business needs an additional question: was this the best use of the constrained capacity available at that moment?
Two requests with identical revenue may deserve different treatment. One might complete a regulated document review that saves hours of specialist labor. The other might generate a disposable variation of marketing copy. If both consume scarce high-end inference during peak demand, equal pricing does not imply equal value. The opportunity cost is hidden unless capacity is managed as a portfolio.
That suggests a more disciplined operating model. Classify workloads by value, latency tolerance, model sensitivity, and deferrability. Route routine work to smaller models. Batch jobs that do not need immediate answers. Reserve expensive reasoning for cases where additional quality changes an economic outcome. Cache stable intermediate results. Give product teams an explicit compute budget, then make them defend the value created per unit of that budget.
This approach is not simply cost cutting. It can improve the product. A system that escalates only ambiguous cases may be faster for ordinary users and more careful when stakes rise. A background research task may benefit from patient, off-peak execution. A coding assistant can use a lightweight model for navigation and reserve a stronger model for architectural decisions or difficult repairs. Good capacity allocation often looks like good interaction design.
Infrastructure commitments narrow strategic freedom
Large commitments also create organizational gravity. Once a company has contracted for capacity or designed a product around a particular performance envelope, it becomes harder to admit that users do not value the resulting intelligence enough. Infrastructure can quietly turn an experiment into an obligation.
Executives should therefore separate three bets that are often bundled together: that model capability will improve, that customers will pay for a specific application, and that the company can serve that demand economically. These propositions influence one another, but none proves the others. A remarkable model demo does not establish willingness to pay. Strong demand does not guarantee attractive unit economics. Falling per-token prices do not ensure that a multi-step agent with retries will be cheap.
The sensible response is not timidity. It is optionality. Use modular architectures that can route across model classes. Measure quality at the task level so substitutions are possible. Negotiate capacity in stages where feasible. Design premium latency and reasoning tiers instead of pretending every request deserves the same treatment. Most importantly, preserve the ability to learn before committing the next block of capital.
AI may still produce software-like margins in many categories. But those margins will be engineered, not inherited. The companies that thrive will connect product telemetry to infrastructure decisions and infrastructure constraints back to product design. They will know which workloads deserve their best machines, which can wait, and which should not exist at all. In the AI economy, restraint is not the opposite of ambition. It is how ambition survives contact with concrete, copper, and power.
Advertisement
