Physical AI Forces Data Teams To Rebuild Architecture Around The Real World

Enterprise data teams are hitting a wall with physical AI, deploying the same Python scripts, orchestration frameworks, and cloud architectures that power the digital SaaS world and watching them break in production. The problem starts with how data originates. In traditional enterprise software, business events are first captured inside digital systems, where databases become the primary system of record. In physical environments, data is merely an observation of something that has already happened in the real world. A dusty camera lens, a robot tripping over a rock, or a layout changing without a system update can turn a data bottleneck into an infrastructure problem, forcing teams to adapt their architecture to physical reality rather than simply optimize the code.

An Phan, Senior Data Infrastructure Engineer at Hippo Harvest, a robotics-driven agriculture startup, builds systems that have to survive intermittent internet and unpredictable physical environments. He brings more than a decade of experience bridging data infrastructure and physical systems to the work, including a tenure as a data engineer at AITOMATIC, where he built production data infrastructure supporting AI systems across manufacturing, maritime, and other industrial domains. Phan recently delivered a talk on cloud-edge data backbones at the IEEE Cloud Summit, with his core argument being that physical AI forces teams to completely rethink how data is collected, stored, and reconstructed.

"The data is not the end. You have to go back more and more to reality," says Phan. Physical AI is still the Wild West, with no Stack Overflow thread waiting when standard tech stacks behave unexpectedly in the field. Phan treats the absence of documented solutions as the operating condition, with localized domain expertise becoming the most practical asset teams can deploy when something breaks. The disconnect explains why rethinking data management for the AI era keeps surfacing underneath so many failed AI platform readiness pilots, with physical anomalies introducing data that traditional digital infrastructure was never built to process.

Designing for the historical replay

Engineers working in physical AI often have to rewind reality months after the fact, rebuilding the timeline of why something went wrong or why a robot failed from whatever was logged in the moment. The infrastructure underneath that work has to support historical replay, which Phan says is less about analytical complexity than about discipline at the point of collection. "The way to solve it is very straightforward: capture enough context to reconstruct it," he notes. "You need to capture different timestamps, including the time of the event, the time on the device, the time it uploaded to the server, and the time of processing. You also have to capture the location. When you capture enough contextual information, you can reconstruct the history behind every data point."

Preserving that level of context creates a downstream problem, with the volume and velocity of the data forcing infrastructure teams to deliver on timelines that traditional batch processing was never designed to handle. The model development cycle has accelerated to a point where any historical replay request can become an emergency, and the underlying systems have to be ready to run before the request even arrives. "When you experience something new, you have to figure something new out," says Phan. "A scientist might suddenly need to rerun the entire data pipeline on two years of historical data for a new model. On the system side, the next question becomes how soon we can deliver it. You can't wait a month, they need the results tomorrow or in six hours."

The pressure to preserve that level of context runs into a different problem at the modeling layer, where the industry conversation around generative physical AI and the headlines coming out of CES often suggest that modeling the entire physical world is within reach. Phan reads the operational reality differently, with the boundary of each system defined by the boundary of its environment. A physical AI system doesn't need a model of the universe. It needs a model of the environment in which it operates. "In the digital world today, embedding models mostly just capture text and images," he explains. "In physical AI, the challenge is figuring out how much of reality we can actually digitize and embed. There are a lot of things to capture, so you must scope the problem first."

Engineering the cloud-edge data backbone

Architecture for physical AI has to account for intermittent internet connectivity, a constraint that shows up across agriculture, aerospace, and maritime deployments where remote operating environments cannot pause when the connection drops. Phan frames the cloud-versus-site decision not as choosing one over the other, but as determining how responsibilities should be divided between centralized cloud infrastructure and on-site systems, with the answer driven by operational requirements rather than technology trends. "Mission-critical operational control must remain on-site whenever interruption would affect the physical system," he explains. "It depends entirely on the operational requirements. You have to define for each activity: does it need centralized infrastructure, or can it operate autonomously on-site? That business question will drive the technical decision."

Keeping certain workloads on-site forces a second discipline around two-way synchronization, with historical telemetry flowing up to the cloud and new models pushed back down through dedicated robotics IDE and developer tooling. Engineering that pipeline introduces real AI growth constraints, with infrastructure teams under pressure to minimize the edge hardware footprint anywhere they can. The trade-off shapes the operating model: cloud as the single pane of glass for fleet-wide management, local systems for anything that cannot fail. "From the cloud, you should be able to see everything. Operators should be able to monitor every deployment centrally without requiring manual access to each site," Phan says. "From a high-level management point of view, you must be able to check each site and see what's happening."

Scope holds the architecture together

Replay workloads are where traditional data architectures start to break under physical AI, with the demand to rerun years of historical data through a modeling pipeline in a matter of hours sitting outside what traditional relational databases are optimized for. The result is a structural pattern where AI strategy gets stuck behind architecture the rest of the business has already moved past, with the ingestion layer becoming the bottleneck that nothing downstream can fix. The move teams are making is to keep the messy ingestion layer in object storage and let relational databases handle the clean, structured serving layer, which lines up with the broader case for a new data architecture built specifically for physical AI workloads. Unlike traditional enterprise software, where databases often serve as the system of record, physical AI treats the physical world as the source of truth. The role of data infrastructure is to preserve enough context to faithfully reconstruct that reality as models, algorithms, and our understanding of the physical system evolve.

The clearest evidence appears in how teams now manage versioned datasets. Rather than treating relational databases as the system of record for every stage of the pipeline, many physical AI architectures preserve raw observations in object storage while using modern table formats such as Apache Iceberg to organize them into versioned, queryable datasets. Relational databases remain valuable for serving clean, structured application data, but the ingestion layer increasingly lives in object storage, where historical replay and large-scale reconstruction become much more practical. "For versioning, we use Apache Iceberg on top of object storage. At the site, we use MinIO as the local object store. Iceberg provides snapshots and version history, while MinIO stores the underlying data objects. Together, they provide a versioned data foundation that feels much more like Git for data than managing records inside a traditional transactional database."

Taken together, these architectural decisions all serve the same purpose: preserving enough context to faithfully reconstruct reality as the physical world, models, and scientific understanding evolve. The constraint that holds all of it together is scope, with practitioners modeling the specific environment the system actually operates in rather than the world it sits inside. "Even today, humanoid robots are very far away from actual humans because we can only scope the problem to specific tasks, like maintenance in manufacturing," Phan concludes. "The reality we want to capture is just around that specific task."