Last month I waited at a crosswalk behind a sidewalk delivery robot. It rolled to the curb, paused, crept forward, then stopped again. Two pedestrians stepped back. A third walked around it and into the street. The robot had no screen and said nothing. Its only interface was its behavior, and that behavior told everyone nearby it had no idea what to do next. Within seconds, nobody else did either.
That moment captures the design problem at the center of physical AI. As artificial intelligence moves out of chat windows and into homes, streets, hospitals, and warehouses, UX design has to expand from designing how people operate systems to designing how intelligent systems behave around people.
What Is Physical AI? Artificial Intelligence That Acts in the Real World
Physical AI refers to AI systems that perceive the physical world through sensors, reason about what they sense, and take physical action in real time. It is the branch of artificial intelligence focused on physical machines that interact with the physical environment, enabling machines to handle complex tasks in complex environments. Autonomous mobile robots, self-driving cars, robot arms, drones, and robotic surgical devices are all examples. Where most AI models produce words or images, physical AI systems produce movement.
NVIDIA CEO Jensen Huang pushed the term into the mainstream during his Computex 2024 keynote in Taipei, where he described a future in which robots learn skills inside simulated digital twins before they ever touch a factory floor.1 Since then, “physical AI” has become shorthand for a broad category of autonomous machines that share one trait: they occupy the same space we do.
Physical AI vs. Digital AI vs. Embodied AI
The easiest way to understand physical AI is to compare it with the AI most people already use.
Digital AI, including generative AI and most AI agents, lives on screens and runs in a data center. It typically trains on static datasets of text, images, and code. When it makes a mistake, the cost is usually a wrong answer that a person can ignore or correct.
Physical AI addresses real-world physical devices, robotics, and machinery that incorporate AI into their perception, operations, movement, and function. These devices tend to operate in continuous time with tight feedback loops. A warehouse robot can’t stop to think for ten seconds while a forklift approaches. It relies on continuous streams of multimodal sensor data, and small delays in feedback can turn into dropped packages, collisions, or injuries. Traditional AI in software can fail gracefully. An AI system that is manipulating objects or moving through a crowded hallway has far less room for error.
Embodied AI, a subset of Physical AI, is a closely related term from robotics research. It emphasizes that embodied intelligence develops through a body interacting with its environment, the way a toddler learns about gravity by dropping things. The two terms overlap heavily. Physical AI tends to describe the commercial category, while embodied AI tends to describe the research question of how physical intelligence emerges.
Either way, physical AI bridges the digital and physical worlds. It takes the reasoning we’ve seen in software and gives it wheels, arms, and a presence in the room.
How Does Physical AI Work?
Physical AI works through three layers running in a continuous loop.
Perception. Sensors capture the environment. Cameras, lidar, and radar map distance and shape, while inertial measurement units track the machine’s own motion and orientation. The system turns this raw sensory input into structured representations that help it recognize objects, estimate spatial relationships, and track the people nearby.
Reasoning. Machine learning models, increasingly including foundation models and world models, predict what will happen next and choose what to do. This layer handles cognitive reasoning and decision making while respecting safety constraints and the laws of physics. A robot can’t plan a path through a wall, and it shouldn’t plan one through a person’s personal space either.
Action. Control systems translate decisions into physical movement: steering a vehicle, closing a gripper, or slowing down near a doorway. The robot observes the result, and the loop starts again.
That loop runs many times per second. Real time decision making and real time feedback make physical AI useful, and they also make it hard to build.
Training Physical AI: Real World Data, Synthetic Data, and Reinforcement Learning
Training physical AI is harder than training a chatbot for one simple reason: the internet contains very little robot experience. Language models learned from trillions of words people had already written. Physical AI requires real sensor readings and actuator commands, paired with the outcomes they produced.
Real world data collection. Teams often gather demonstrations by having people teleoperate robots through a task, then use imitation learning to teach the robot to copy them. It is slow and expensive. One 2026 industry cost analysis puts simple pick-and-place demonstrations at $6 to $11 each and complex two-handed tasks with soft materials at up to $157 each.2 At those rates, a 10,000-demonstration dataset costs tens of thousands of dollars for simple tasks and well into six figures for harder ones. Models that generalize also need diverse data across hundreds of tasks. The Open X-Embodiment project pooled more than one million real robot trajectories from 22 robot types and 21 institutions, covering 527 skills.3 Training physical AI also requires human oversight and monitoring, since a robot learning in real environments can damage itself or its surroundings.
Synthetic data and simulation. To reduce that cost, developers train in a simulated environment first. Physics simulators like NVIDIA Isaac Sim create digital twins of warehouses, streets, and homes, a virtual space where robots can practice thousands of scenarios in parallel.4 A key technique is domain randomization: varying lighting, textures, colors, and object positions so the model can’t memorize one version of reality. In a 2017 study, researchers trained an object detector using only simulated images with random, unrealistic textures, then used it on a real robot with about 1.5 centimeters of accuracy.5 Randomized synthetic data makes physical AI models more resilient when they meet messy real world environments.
Reinforcement learning. Reinforcement learning lets robots learn through trial and error, earning rewards for success and penalties for failure. It plays a central role in training physical AI for walking, balance, and difficult tasks like dexterous grasping. The catch is volume. Reinforcement learning requires extensive trial-and-error interactions, which is why most of it happens in simulation, where a robot can fall a million times without breaking anything. The learned policies are then transferred to hardware and fine-tuned there.
Physical AI development depends on all three. Simulation provides scale, real data provides truth, and robot policies get refined in the gap between them. That gap is also where many impressive demos fall apart, once a robot has to execute actions in real homes and factories, under shifting weather conditions, around people who don’t behave the way the simulation expected.
What Autonomous Vehicles Teach Us About Physical AI Development
Autonomous vehicles are the most visible case study in real world physical AI, and they show why engineering capability alone isn’t enough.
Self-driving systems must process sensor data in real time, avoid obstacles, and predict what pedestrians, cyclists, and other drivers will do. When they fail, the consequences are ethical and regulatory as well as technical. In 2018, an Uber test vehicle struck and killed a pedestrian in Tempe, Arizona. The National Transportation Safety Board found that the automated driving system kept changing how it classified the pedestrian and didn’t determine a collision was imminent until 1.2 seconds before impact. The board also cited an inadequate safety culture at the company.6
That report is technical, but its lesson is about behavior and trust. People around autonomous systems need to understand what the machine sees, what it intends, and when a human needs to step in. Punchcut has spent years on these questions through its mobility and automotive design work, and we’ve written about the design fundamentals that help people trust autonomous vehicles.
The same lesson applies across physical AI. Robotic surgical systems translate a surgeon’s hand movements into stabilized, scaled instrument motion, filtering out tremor so the machine does precisely what the human intends.7 Drones inspect bridges and power lines so people don’t have to climb them. Robots take on work in hazardous environments. In factories, physical AI supports predictive maintenance by tracking heat and vibration to catch a failing machine before it breaks down, and it adapts to disruptions in real time in ways that make supply chains more resilient. It also creates demand for new roles, like the technicians who keep autonomous robots running. In every case, the technology succeeds only when people understand and trust how it behaves.
Six Design Principles for Physical AI
For physical AI, behavior is the user interface. Movement, proximity, timing, orientation, sound, gesture, attention, and the degree of autonomy a system claims are now design materials, just as type and color have always been. People read these signals whether or not anyone designed them. A robot that speeds toward a person to save two seconds is announcing who owns the hallway.
Trust changes, too. In software, trust mostly concerns accuracy and privacy. Next to robots interacting with them in real space, people ask more basic questions, and they ask them fast. Can I predict what it will do? Does it see me? Will it stop? Can I take over?
At Punchcut, we design physical AI across robotics, environments, and autonomous mobility. Six principles guide that work.
1. Signal Intent
People need to know what a machine will do before it does it. Cars solved a version of this with turn signals, and drivers still read each other’s body language: a drift toward the lane line, a slowdown at an intersection. Robots need their own equivalents, like a pause before moving or a deceleration on approach. Intent revealed after the fact is only a surprise.
2. Respect Human Space
Proximity, speed, orientation, and boundaries carry meaning. How close is too close depends on the room, the task, and who is standing there. A machine that treats a hospital corridor like a warehouse aisle comes across as rude. Personal space should be a deliberate design decision rather than a path-planning default.
3. Calibrate Autonomy
Autonomy is a range: observe, suggest, ask, act, or stand down. The right point on that range shifts with the stakes, the system’s confidence, and who is present. Products that get this wrong make people uneasy, either by acting when they should ask or by asking about things nobody cares about. This principle builds on Shared Autonomy, one of the core principles behind Punchcut’s human-centered AI practice, which holds that people and intelligent systems should share control in ways that keep humans in charge of what matters most.
4. Preserve Human Agency
People need an obvious way to interrupt, redirect, or take back control, and they need to know how before they need it. Knowing you can stop the machine is what makes it comfortable to let it run.
5. Align Every Channel
Movement, sound, light, voice, and gesture each say something. When they conflict, people believe the movement. A cheerful voice from a robot that is crowding you reads as oblivious. Designers need to compose these channels together, as they would a visual system.
6. Earn Trust Slowly
First impressions matter, but consistency is what lasts: the same behavior in the same situation, honest uncertainty instead of false confidence, and a graceful recovery when something goes wrong. One good recovery can build more trust than a hundred smooth interactions. One violated boundary can end it.
How Social Should Physical AI Be?
People read intent into anything that responds to them. We do it with pets and vending machines, and we’ll certainly do it with a machine that turns to look at us. Whether we mean to or not, we’re designing a robot’s perceived character: confident or hesitant, attentive or indifferent, polite or pushy.
The tempting move is to make these systems seem human. That would be a mistake. Faces and simulated warmth promise an understanding the system doesn’t have, and people sense the gap quickly. A robot needs enough social intelligence to behave appropriately around people: yield in a doorway, wait for someone to finish talking, know the difference between a hallway and a bedroom. Call it manners, and treat manners as a design problem. Seamless integration into daily life depends far more on those manners than on a friendly face.
The definition of a user changes, too. Physical AI encounters coworkers, children, guests, pets, other machines, and bystanders who never agreed to anything. Most of them will never open the app or read the settings. The unit of design is now a person, a system, and an environment full of others who never opted in.
The Next Dimension of UX Design
Engineers get better every year at answering what a machine can do. The harder question is how it should behave around people, and that question belongs to UX design. The goal is machine behavior that people find understandable, appropriate, and accountable, and careful behavior design gets there faster than simulated humanity ever could.
For decades, designers have shaped the space between people and screens. Physical AI asks us to shape the relationship between intelligent machines, the people near them, and the environments they share. That is a bigger responsibility and a real opportunity to expand what UX design can do.
Physical AI won’t win on capability alone. It will succeed when machines operate in ways people can understand, predict, and live with. Designing that behavior is work UX designers can’t hand off.
A version of this article originally appeared in UXmatters.
References
- DigiTimes, “Nvidia’s Jensen Huang discloses next-gen Rubin platform and physical AI vision at NTU keynote,” June 3, 2024. https://www.digitimes.com/news/a20240603VL200/jensen-huang-nvidia-computex-2024-ai-gpu-cuda-digital-twin-robotics.html
- Silicon Valley Robotics Center, “Robot Data Collection Cost Breakdown: What You’re Really Paying For,” April 7, 2026. https://www.roboticscenter.ai/research/robot-data-collection-cost-breakdown
- Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic Learning Datasets and RT-X Models.” https://robotics-transformer-x.github.io/
- NVIDIA Developer, “Isaac Sim: Robotics Simulation and Synthetic Data Generation.” https://developer.nvidia.com/isaac/sim
- Tobin et al., “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World,” IROS 2017. https://arxiv.org/abs/1703.06907
- National Transportation Safety Board, Highway Accident Report NTSB/HAR-19/03, Tempe, Arizona, March 18, 2018. https://www.ntsb.gov/investigations/accidentreports/reports/har1903.pdf (summary: https://smartcitiesdive.com/news/ntsb-blames-driver-inadequate-safety-culture-for-uber-av-crash/567673)
- SAGES, “DaVinci Surgical System: A SAGES Technology and Value Assessment.” https://www.sages.org/publications/tavac/tavac-analysis-davinci-surgical-system/
Author: Ken Olewiler
Ken Olewiler is the CEO & Co-founder of Punchcut.
For over 20 years, Ken has driven Punchcut’s vision and strategy — from its inception as the first mobile design consultancy to its position today as an AI design accelerator for intelligent product innovation and business growth
Reviewed By: Akshat Srivastava
Akshat Srivastava is the Director of Design Engineering at Punchcut.
Akshat leads Punchcut’s growing AI Prototyping & Development practice, uniting UX strategy, technical R&D, and AI-native design systems to help clients ship intelligent products at scale.
