Energy Companies Once Competed on Assets and Now Compete on Data Quality

Every large energy trader is rebuilding its data plumbing at the same time, and the push didn't come from the AI teams. It came from regulators who got tired of watching crises unfold without the numbers to understand them. The surprise is what the compliance spend turned into: the clean, centralized, auditable data that algorithmic trading and AI were going to demand anyway.

Arnab Ray is a Product Owner at Shell, leading the strategic delivery of counterparty lifecycle management applications and enterprise data products that enable secure, compliant, and efficient trading operations, and he spoke in a personal capacity about patterns he sees across the industry. He's spent 14 years in enterprise digital delivery across energy trading, data analytics, and IT operations, and the question that started his research was why every energy company seemed to be making the same data changes at once.

"The foundation for AI is good data," Ray says. "For AI to deliver stronger outcomes, we need high-quality data that is accurate, complete, consistent, and trusted. Better data doesn't just improve AI, it builds confidence in every decision AI makes."

Two crises wrote the rules

The answer he found was regulatory, and it runs on a crisis cycle. Europe's REMIT framework for wholesale energy market integrity arrived in 2011, after the 2008 financial crisis exposed how little visibility regulators had into energy trading. A decade of relative calm ended with the 2021-22 gas shortage. "The regulators saw a huge gap in trading, the prices and the volume being traded," Ray says. "They did not have proper data to see what the demand in the market was, what was being supplied by each company, what the actual price was." The revised REMIT that took effect in May 2024 answers that gap by demanding better data, not just more of it, with reporting obligations that now reach storage and hydrogen.

The American version, in his reading, traces to the California electricity crisis of 2000 and 2001, and points inward at market abuse rather than at cross-border trade. Different geography, same conclusion. Regulators want data good enough to reconstruct a crisis.

Seventeen requests for one customer

What that demand exposed inside energy companies was fragmentation. A large trader might run 20 different trading applications across regions and lines of business, each with its own version of the same customer.

Ray felt the cost personally. “A few years ago, we faced a significant challenge. A single counterparty was engaging with multiple lines of business across three regions. Because counterparty data was not centralized, there was no enterprise-wide visibility into those interactions. As a result, different regional and business teams independently requested KYC documents before executing trades, with some counterparties being contacted as many as 17 times." The fix is master counterparty data, one verified record per customer mapped across every system and region, feeding a single source of truth. Sanctions turned that plumbing into a trading requirement. "When Russian companies were gradually sanctioned, how would we know that this particular counterparty I'm trading with is compliant to be traded? That data helps us to know."

Standard tools, borrowed trust

Regulators also standardized the industry by accident. Because every company must report the same fields, market-standard trading platforms built around those fields have displaced homegrown systems, and shared data structures do the trust-building. "My line of business can be different, my ways of working can be different, but whatever tool you use, you will be compliant with the data the regulator wants," Ray says. "That builds the trust."

Transparency compounds from there. "When I share my data transparently, other energy companies share their market data transparently. When the data is available, we know where we are heading," he says. "Earlier we did not have good data, so there were more failed projects. Now we make business investment decisions with more data behind them." Algorithmic trading, already the norm, runs entirely on that trusted, traceable data.

Fix it at the source

His sharpest advice for industries following the same path is about where quality gets made. "Getting bad quality data and making it correct doesn't work that much. I have seen it in my day-to-day work," Ray says. "Go two or three steps back to where the data is being generated, correct the issue at the source, and then intake that data." The payoff is structural: fewer manipulation steps, better insights from the same data, and lower infrastructure cost, since processing bad data demands heavier tooling than ingesting clean data ever will.

He frames the stakes in the industry's own history. "Oil powered the industrial revolution. Electricity powered the digital revolution," Ray says. "The next revolution is the data revolution."

The views and opinions expressed are those of Arnab Ray and do not represent the official policy or position of any organization.