How Certified Data Is Eliminating Duplicate Pipelines Across Enterprises

Across enterprise data teams, the same scenario keeps playing out: two business units pull near-identical datasets from the same warehouse, build adjacent pipelines, and ship dashboards that show roughly the same numbers to different stakeholders. The pitch for modern, unified data platforms is supposed to solve this problem by collapsing the stack into a single environment, where shared consumption and centralized storage eliminate the duplicate work underneath the surface. The pattern is harder to dislodge than the architecture suggests, with consolidated tooling rarely fixing what is fundamentally a governance and ownership problem. Engineering leaders working through these migrations are learning that the platform decision is the easier half of the equation.

Gangadhar Thumu, Lead Data Engineer at AT&T, is working through exactly these kinds of migrations inside one of the largest enterprise data environments in the country. With more than 14 years of IT experience and a stack of Azure and Fabric certifications, Thumu approaches the modern data ecosystem through a strictly pragmatic lens. Culture is part of the equation, but the plumbing has to work before anything else can. When evaluating infrastructure choices, Thumu relies on a straightforward heuristic to cut through the marketing layer that surrounds most platform discussions.

"It's all about the data duplication, it's about the processing power, and more importantly, the same work being done by multiple teams. That's a big challenge," says Thumu. The first decision in Thumu’s framework is choosing between a warehouse and a Lakehouse. Structured, tabular data aligns well with traditional warehouse architectures, while environments handling APIs, JSON payloads, and unstructured data require the flexibility of a Lakehouse.

Connectivity is the next consideration, since a unified platform only delivers on its consolidation promise when it can actually reach the source systems that feed it. Legacy orchestration tools tend to break down as the source list expands, with coverage caps that force teams to bolt on secondary connectors for the systems the main tool cannot reach. A single platform able to handle a project drawing from ten sources alongside another drawing from five is what makes the consolidation pitch work in practice.

Invisible infrastructure as the goal

While business users often see little change during backend migrations, the burden shifts heavily to engineering teams responsible for migrating pipelines, datasets, and integrations. Large-scale migrations involving Synapse, ADF, Fabric, Snowflake, Power BI, Power Automate, and OneLake can take months. However, automated readiness and assessment tools are reducing this effort significantly. In one case, Thumu’s team analyzed 75–80 pipelines and leveraged built-in readiness tooling to streamline migration. “It shows everything is intact. To move to Fabric, the assessment portal provides the blueprint and effectively turns technical validation into a one-click confirmation, even though the deployment orchestration still requires a structured rollout,” he explains.

Solving problems for one team often creates duplication elsewhere. Thumu describes a recurring scenario: his team automates backend analysis and distributes results via email, eliminating manual effort. Meanwhile, other departments request the same data through custom Power BI dashboards. The result is parallel pipelines, duplicated processing, and increased infrastructure costs without shared visibility across teams. “With disparate tooling, data duplication becomes inevitable,” Thumu notes. Invisible infrastructure as the goal

This is not simply a communication issue; it’s an architectural one. Thumu advocates for governance-driven solutions centered on certified and endorsed data models. These mechanisms enable teams to reuse trusted datasets instead of rebuilding pipelines, significantly reducing redundant processing across the enterprise. As platforms mature, infrastructure is increasingly abstracted. Dynamic scaling models, such as F64 or F128 capacities, replace granular infrastructure decisions, shifting focus toward cost optimization rather than system configuration. “When the infrastructure layer becomes invisible, the platform simplifies. The only thing we track is cost—before and after Fabric,” Thumu explains.

The next governance frontier

The reliability of baseline infrastructure is finally stabilizing across enterprises that have invested in serious data modernization work. With core data movement operating without daily failures, engineering teams are freed up to focus on what to build with the data rather than fighting to keep it flowing. The era of daily pipeline and ETL failures is largely behind the industry, with foundational technology delivering reliable, near-instant data on a consistent basis. "A couple years back, we used to have pipeline failures, ETL failures, many daily. I haven't seen any pipeline failures in the past few years," Thumu says.

The attention is now turning toward speed of AI adoption, with the same governance discipline that solved the duplication problem applied to the question of how fast and how safely AI can be brought into the enterprise. Many companies are balancing aggressive adoption with stricter guidelines around token usage, applying the same operational mindset to AI workloads that data engineering teams have been applying to core pipelines for years. The combination of reliable infrastructure underneath and disciplined governance on top is reshaping what enterprise data engineers can realistically deliver. "A project that used to take three months can now be completed in 10 days with more quality," Thumu concludes.

The views and opinions expressed are those of Gangadhar Thumu and do not represent the official policy or position of any organization.