Enterprises Solved the Easy Half of AI Cost Discipline but Still Struggle to Measure Value

Most enterprises can now put a number on what AI costs them, and tokens are the unit that makes it possible. Because billing runs on tokens, teams can look disciplined by staying within allocation even when the quality or usefulness of the output remains unclear. Token spend is increasingly being used to compare models, price projects, and decide which programs deserve continued funding.

Ashok Kumar, Principal Data Scientist at AT&T, leads generative AI, LLM, and agentic AI initiatives from strategy through production, which puts him close to the infrastructure costs behind enterprise AI. He spent more than two decades in the field, including work in classical machine learning, and makes the case for better measures of LLM value. 

"The intelligence produced from the same number of tokens can be completely different," Ashok argues. "Two models given identical budgets are not necessarily doing comparable amounts of work. A cheaper per-token rate can produce a higher cost per completed task when the output needs a second pass, and none of that appears in a consumption dashboard."

Why LLM usage stopped being the automatic default

When LLMs first reached employees at scale, the instruction from leadership was to use them. Teams were encouraged to build solutions around GenAI, and adoption itself was treated as the metric worth moving. Nvidia's Jensen Huang gave the maximalist version of that position publicly, saying he would be “deeply alarmed” if a $500,000 engineer consumed less than $250,000 worth of AI tokens in a year.

Then the invoices landed. Organizations started seeing what the projects cost and how quickly the numbers could climb. "Because these projects are affecting the balance sheet, LLM usage is no longer becoming the automatic default," Ashok explains. Teams started asking if each step required a model call at all. Ashok argues that repetitive jobs and workflow-based tasks are where AI earns its place, particularly when the alternative is significant human resource cost or the work absorbs too much of a team's time. More demanding work calls for a different allocation. "If something is highly complex and requires human intelligence, that's where you should focus your team's bandwidth."

Context as the invisible line item

Once a model does belong in the workflow, the next question is how much context it needs. Much of the resulting spend accumulates without an explicit decision, which is why it can survive rounds of optimization aimed at the wrong layer.

"Data is really the context component," Ashok explains, and the cost consequence follows directly from how much of it gets passed. A chatbot working through a conversation window may resend the entire history every time the model is called, paying repeatedly for information the system has already seen. "That consumes tokens unnecessarily," he notes. "You should only pass the information that is relevant at a given point in time, instead of passing everything available by default and absorbing the difference."

The business value behind the token count

Reducing excess context lowers consumption, but the value produced by that consumption remains unresolved. The industry settled on tokens as a unit early, largely because they were easy to count. "Tokens are primarily used as a unit for measuring consumption," Ashok explains. “From my perspective, we should think about them as a way of measuring AI productivity too.” Billing reinforces the flattened view because companies pay for tokens, not intelligence.

The same misalignment appears within individual organizations. Give two employees identical token allowances, and the question becomes what each gets out of them. "They may both be spending the same amount, but the value they produce can be completely different," Ashok notes. Organizations should also account for the compute underneath the output to extend measurement beyond API usage to the infrastructure supporting the system.

Memory architecture for reduced token spend

Multi-agent systems often need access to more information than a context window can hold. Ashok describes context engineering as a hierarchy of memories. Recent and relevant information stays close to the agent, less frequently needed data sits in a knowledge graph or database, and a metadata layer helps the agent determine what exists and where to find it. "It's similar to the way we design operating systems, with hierarchies of memory and mechanisms for determining which information should be available at a particular moment." The structure varies by use case, but the goal is to keep only the right information within immediate reach and reduce the tokens needed to complete a task.

The broader cost of AI infrastructure

Enterprises are still building, but they're starting to price the whole system, not just the model call, and that's the maturity signal worth tracking. "I wouldn't say they're underestimating it, but they are becoming much more cautious about cost," Ashok says. Data movement, storage, retrieval, and the compute environment underneath all belong in that calculation. Once those costs are included, efficiency per unit of consumption becomes a more useful measure of AI maturity than consumption volume alone.

The outer boundary of that accounting is the one the industry has been slowest to price. Data centers require significant water and energy, the buildout continues, and current rates depend on financial support that cannot continue indefinitely. "I'm not sure the current model is sustainable because of the level of subsidization we're seeing," Ashok cautions, "and you cannot simply continue putting data centers and compute infrastructure everywhere without consequences."