Physical AI Forces Data Teams To Rebuild Architecture Around The Real World

Enterprise data teams are hitting a wall with physical AI, deploying the same Python scripts, orchestration frameworks, and cloud architectures that power the digital SaaS world and watching them break in production. The problem starts with how data originates. In traditional enterprise software, business events are first captured inside digital systems, where databases become the primary system of record. In physical environments, data is merely an observation of something that has already happened in the real world. A dusty camera lens, a robot tripping over a rock, or a layout changing without a system update can turn a data bottleneck into an infrastructure problem, forcing teams to adapt their architecture to physical reality rather than simply optimize the code.

An Phan, Senior Data Infrastructure Engineer at Hippo Harvest, a robotics-driven agriculture startup that recently announced a $30 million Series C to support its next phase of growth, builds systems that have to survive intermittent internet and unpredictable physical environments. He brings more than a decade of experience bridging data infrastructure and physical systems to the work, including a tenure as a data engineer at AITOMATIC, where he built production data infrastructure supporting AI systems across manufacturing, maritime, and other industrial domains. Phan recently delivered a talk on cloud-edge data backbones at the IEEE Cloud Summit, with his core argument being that physical AI forces teams to completely rethink how data is collected, stored, and reconstructed.

"The data is not the end. You have to go back more and more to reality," says Phan. Physical AI is still the Wild West, with no Stack Overflow thread waiting when standard tech stacks behave unexpectedly in the field. Phan treats the absence of documented solutions as the operating condition, with localized domain expertise becoming the most practical asset teams can deploy when something breaks. The disconnect explains why rethinking data management for the AI era keeps surfacing underneath so many failed AI platform readiness pilots, with physical anomalies introducing data that traditional digital infrastructure was never built to process.

Designing for the historical replay

Engineers working in physical AI often have to rewind reality months after the fact, rebuilding the timeline of why something went wrong or why a robot failed from whatever was logged in the moment. The infrastructure underneath that work has to support historical replay, which Phan says is less about analytical complexity than about discipline at the point of collection. "The solution is actually straightforward: capture enough context to reconstruct what happened," Phan says. "That means preserving different timestamps, such as when the event occurred, when the device recorded it, when it reached the server, and when it was processed. You also need the location. Once you capture enough context, you can reconstruct the history behind every data point."

Preserving that level of context creates a downstream problem, with the volume and velocity of the data forcing infrastructure teams to deliver on timelines that traditional batch processing was never designed to handle. As physical AI deployments expand across larger facilities and eventually multiple sites, replay also becomes a fleet-wide capability rather than a local debugging tool. Engineers increasingly need to compare behavior across different environments, hardware generations, and operating conditions. The model development cycle has accelerated to a point where any historical replay request can become an emergency, and the underlying systems have to be ready to run before the request even arrives. "When you experience something new, you have to figure something new out," says Phan. "A scientist might suddenly need to rerun months or even years of historical data, compare different deployments, or reconstruct what happened under new operating conditions. From the infrastructure side, the question immediately becomes how quickly we can deliver those answers. Waiting weeks isn't an option."

The pressure to preserve that level of context runs into a different problem at the modeling layer, where the industry conversation around generative physical AI and the headlines coming out of CES often suggest that modeling the entire physical world is within reach. Phan reads the operational reality differently, with the boundary of each system defined by the boundary of its environment. A physical AI system doesn't need a model of the universe. It needs a model of the environment in which it operates. "In the digital world today, embedding models mostly just capture text and images," he explains. "In physical AI, the challenge is deciding how much of the real world we can realistically capture and represent. There is simply too much reality to model, so you have to scope the problem first."

Engineering the cloud-edge data backbone

Architecture for physical AI has to account for intermittent internet connectivity, a constraint that shows up across agriculture, aerospace, and maritime deployments where remote operating environments cannot pause when the connection drops. Phan frames the cloud-versus-site decision not as choosing one over the other, but as determining how responsibilities should be divided between centralized cloud infrastructure and on-site systems, with the answer driven by operational requirements rather than technology trends. "Mission-critical operational control must remain on-site whenever an interruption could affect the physical system," he explains. "It depends entirely on the operational requirements. For each activity, you have to decide whether it needs centralized infrastructure or can operate autonomously on-site. That operational requirement should drive the technical architecture."

Keeping certain workloads on-site forces a second discipline around two-way synchronization, with historical telemetry flowing up to the cloud and new models pushed back down through dedicated robotics IDE and developer tooling. As organizations scale from individual deployments to fleets of physical AI systems, the cloud also becomes the coordination layer for centralized observability, model rollout, and version management across every site. Engineering that pipeline introduces real AI growth constraints, with infrastructure teams under pressure to minimize the edge hardware footprint anywhere they can. The trade-off shapes the operating model: cloud as the single pane of glass for fleet-wide management, local systems for anything that cannot fail. "From the cloud, you should be able to see every deployment. Operators should be able to monitor the entire fleet centrally without requiring manual access to each site," Phan says. "That centralized visibility becomes increasingly important as physical AI systems scale."

Scope holds the architecture together

Replay workloads are where traditional data architectures start to break under physical AI, with the demand to rerun years of historical data through a modeling pipeline in a matter of hours sitting outside what traditional relational databases are optimized for. The result is a structural pattern where AI strategy gets stuck behind architecture the rest of the business has already moved past, with the ingestion layer becoming the bottleneck that nothing downstream can fix. The move teams are making is to keep the messy ingestion layer in object storage and let relational databases handle the clean, structured serving layer, which lines up with the broader case for a new data architecture built specifically for physical AI workloads. Unlike traditional enterprise software, where databases often serve as the system of record, physical AI treats the physical world as the source of truth. The role of data infrastructure is to preserve enough context to faithfully reconstruct that reality as models, algorithms, and our understanding of the physical system evolve.

The clearest evidence appears in how teams now manage versioned datasets. Rather than treating relational databases as the system of record for every stage of the pipeline, many physical AI architectures preserve raw observations in object storage while using modern table formats such as Apache Iceberg to organize them into versioned, queryable datasets. Relational databases remain valuable for serving clean, structured application data, but the ingestion layer increasingly lives in object storage, where historical replay and large-scale reconstruction become much more practical. "For versioning, we use Apache Iceberg on top of object storage. At the site, we use MinIO as the local object store. Iceberg provides snapshots and version history, while MinIO stores the underlying data objects. Together, they provide a versioned data foundation that feels much more like Git for data than managing records inside a traditional transactional database."

Taken together, these architectural decisions all serve the same purpose: preserving enough context to faithfully reconstruct reality as the physical world, models, and scientific understanding evolve. As physical AI systems expand across larger facilities and geographically distributed deployments, that same architecture must also preserve consistency across the entire fleet while allowing each site to adapt to its own physical environment.

The constraint that holds all of it together is scope, with practitioners modeling the specific environment the system actually operates in rather than the world it sits inside. "Even today, every successful physical AI system succeeds because it focuses on a well-defined environment and a specific task," Phan concludes. "The reality we want to capture is only the part of the physical world that matters for that task."