Smaller Models And Shorter Lookback Windows Help Data Teams Get More From AI Infrastructure

Many enterprises can get far more out of their AI infrastructure than they do today. Their models often run at higher precision than the work requires, and their pipelines process years of data when weeks would do. Right-sizing both frees capacity for new AI projects, but few teams check those assumptions when planning for growth.

Janne Kalliola is Founder of Coonti, a consultancy that advises organizations on sustainable IT and responsible AI, and a PhD researcher focused on the energy consumption of software. He has built and led software development organizations at several companies, including Exove, the digital agency he founded, and chairs the board of Code from Finland, an association that promotes software developed in Finland. He also created the association’s carbon neutrality symbol and criteria. His latest research published in ACM Transactions on the Web measured the energy consumption of 100 popular websites across five countries, and he is the author of a guide to procuring green, energy-efficient IT. In an earlier conversation, he made the case that software waste quietly drives up compute demand. He now sees the same pattern in enterprise AI.

“Can you downscale to smaller models without losing quality? That’s something that should be tested,” says Kalliola. Few enterprises run that test, which leaves much of their infrastructure doing extra work.

More model than needed

When a company adds computing power for AI, teams quickly find new ways to use it, from new models to new experiments. The added room fills up, and demand keeps climbing. “If you have more infrastructure, it just creates more demand by providing more opportunities,” says Kalliola. Efficiency work has to keep pace with any expansion.

Kalliola believes better model and data discipline could handle at least half of that demand. Most companies can’t see which models and datasets use the most resources, so that opportunity goes unnoticed.

One place to look is model precision, meaning how many digits a model uses in its calculations. Engineers at a large Finnish retailer recently lowered it, a technique called quantization that makes models smaller so the same servers can run more work. “They reduced the space needed for the model to a quarter or an eighth without major loss, or even minor loss, in the quality of the output. We might be over-engineering this,” notes Kalliola.

AI models are also frequently tuned to a higher level of accuracy than their tasks call for, and that extra margin can take much more computing power to reach. A model that recommends products, for example, may perform just as well at 99.9% accuracy as at 99.999% while using fewer resources. Kalliola suggests testing whether a model still delivers the same results with a tenth of its weights, or even a hundredth. Smaller models built for specific tasks can often match larger ones while using far less energy.

Trimming the footprint

Forecasting tools often use all the historical data they can reach, even when older records add little. “If you try to predict consumer behavior, how long do you need to look at the history? Is it one year, three months, or two or three days?” asks Kalliola.

To answer that, a team can run the same prediction three times, using three days, three months, and three years of data, then see which result comes closest to what happened. If the shorter timeframe is just as accurate, keeping the older records in active use adds cost without improving the prediction. Duplication has a similar effect. When several pipelines each keep their own copy of the same records, the company pays to store and handle them more than once. Comparisons like these show data teams which records should stay readily available and which can move to archive.

Switching to a smaller model, or a compressed version of an existing one, lowers the memory and computing power each model needs. That makes it possible to run several models on the same infrastructure. “It would also give you the answers faster, or you could use the same hardware for a number of different tasks,” says Kalliola. Enterprises commonly give each AI application its own environment, so the footprint grows with every new use case. Sharing raises utilization, and quicker responses help everyone who depends on those tools. Planning for that sharing before deployment is easier than consolidating later.

Paying for the analysis

The cost of running AI tends to build up gradually, while a review of models and data comes with a large upfront bill. “You need to pay a person who probably charges you quite a lot, and whether you will get the money back depends on the company size,” says Kalliola.

Adding hardware also feels like progress in a way that tuning existing systems doesn’t. “It’s way easier to buy new stuff than to figure out how to use the stuff you’ve already invested in, because that work is just way less sexy,” he adds.

Large enterprises with heavy AI workloads are the most likely to see a return, since small gains add up across many models and datasets. Smaller organizations may find the cost harder to justify, so the review often doesn’t happen.

The better planning question is how much infrastructure each AI project needs to deliver its value. Answering it means building model and data checks into every capacity plan. Leaders have to back that work, since the savings arrive after the bill. “There should be momentum coming from the company, a sense that it wants to change. Leaders should also be prepared to invest some money in this, because it gives them substantial savings in the future,” says Kalliola.