What the AI decides is what moves the machine

A transport robot in a factory decides its next route from camera images and sets off. A collaborative robot decides the trajectory of its arm, an autonomous mobile robot inside a facility picks its patrol path, an inspection drone decides its flight path. Sensors perceive the situation, a learned model or a language model decides the next motion, and the result drives motors, arms, and conveyors, and opens and closes doors. That loop is what a Physical AI system consists of.

At the end of the loop, the decision becomes physical motion. An incorrect output, or an output planted from outside, shows up as motion unless something along the way stops it. What the design has to add follows from that single fact.

Safety and security overlap, but they are not the same

Safety means reducing the risk of harm to people and equipment, including harm from faults, malfunction, and misuse. Security means guarding against unauthorized access, alteration, and interference. There is overlap, in that an attack can undermine safety, but meeting one does not mean the other is met.

SP 800-82 Rev. 3, the US NIST guidance on control-system security, states that in OT (operational technology) risk management safety is a primary concern, and that a security measure that impairs safety is unacceptable.

The same document takes as a premise a design that transitions to a known safe state when something unexpected happens (fail-safe), and notes that an event such as a loss of communications can trigger that transition. On that basis it recommends considering the thresholds for how far operation should continue in a degraded state. It also sets out the approach of placing a safety mechanism independent of the basic control system, as with a safety instrumented system.

Translated to Physical AI, the principle is a single one. Do not turn the AI’s recognition results or natural-language output directly into actuator authority. The AI proposes, an independent verification layer and the safety mechanisms authorize, and the control system executes. That separation is what lets you reject a command that violates constraints defined in advance, before it is executed. Separating the layers does not mean every error will be found, however. The risk that remains is confirmed through verification under the conditions that apply to the system.

Five things to decide early, and who decides them

If you settle what has to be decided but not who decides it, implementation goes ahead with the blanks unfilled.

What to decide What it covers Primary decision-makers What to write down
Trust boundary for inputs Which sensors, communications, and external instructions you trust, and where you verify them System architect, security lead Boundary diagram, data-flow diagram
Action authority and action validation The range of actions the AI may propose, the conditions for authorization (limits on speed, area, and force; the state of the machine), and who or what authorizes them Control engineer, safety lead Permitted-action table
Model and configuration updates Who updates them, through which verification, how they are distributed, and how they are rolled back Development lead, operations lead Change-management procedure
Transition on a fault The safe state for each use, the conditions for transitioning, and the conditions for returning Safety lead, site manager Fault-condition table
Monitoring What you monitor, the threshold at which a person is notified, and the conditions for an automatic transition Operations lead List of monitored items

These five items are our design proposal, not a requirement laid down by SP 800-82.

Of these, the transition on a fault is decided according to the use and the failure state. An immediate stop is not always the safest response to every fault. From options such as stopping, slowing down, and holding the current state (maintaining grip or brake), the safe state and the conditions for transitioning are set through risk assessment and verification carried out in advance.

For an arm holding a load, or a transport robot partway up a slope, immediately removing power or releasing can be more dangerous. SP 800-82 recommends considering thresholds so that this judgment is made at the design stage rather than after operation has begun.

Updates to models and configuration fall under the same change management as software updates. The same document states that in OT, thorough testing is needed before implementation and that revalidation may be required as part of the update process. Swapping in a trained model should not sit outside that procedure.

Protecting the traffic is not the same as checking what the command says

In a configuration that uses ROS 2 for the robot’s internal communication, with a middleware implementation (RMW) that supports DDS and its security plugins, ROS 2’s security features are available for protecting that communication. ROS 2 Security provides authentication of participants, access control through signed permission files, and encryption of communication, by way of the DDS security plugins.

None of it takes effect on its own, however. According to the official documentation (Jazzy), security is off by default and has to be enabled explicitly through an environment variable, and under the default behavior a participant whose configuration is faulty still launches, without security. To keep such a participant from launching, you specify the enforcing mode separately. Whether the RMW and the version you use support the security plugins also has to be confirmed for each configuration.

What these protect is who can write to a topic, and whether anything was altered in transit. Whether that command is appropriate in the current situation remains a separate question. When a properly authenticated node commands a speed outside the permitted range, or movement into an area where a person is standing, neither authentication nor encryption stops it.

The semantic validity of a command, meaning its consistency with limits, permitted areas, and the state of the machine, is checked in a verification layer that is independent of both the AI model and the communication layer. The ROS 2 threat model design document (a draft) likewise lists the integrity of commands to actuators as an asset to protect, and states that actors without permission must not be able to control the actuators.

By role: the AI model proposes, the command verification layer checks validity, the existing safety mechanisms (safety PLC, emergency stop, independent safety systems) are the last line of defense, and the control system executes. Adding authentication does not achieve functional safety, and the AI’s judgment alone cannot guarantee safety either.

Where to start

The first thing to draw is the flow from sensors to actuators. Identifying on the diagram where the AI’s output turns into a command to an actuator tells you where the verification layer belongs.

Next, fill in the permitted-action table one row at a time, in four columns: action, conditions, limits, and who authorizes it. A row you cannot fill is a hole in the design. For the transition on a fault, decide per use whether it stops, slows down, or holds its state, through risk assessment and prior verification for each failure state.

If your configuration uses ROS 2, confirm the enabling settings, the enforcing mode, and support in the RMW and version you use at this stage. Finally, decide who runs change management and monitoring for models and configuration.

Once this is filled in, you can decide, with reasons you can show, whether a system that worked in PoC should go to production, which actions pass through automatically and which keep human approval, and whether additional safety mechanisms are needed. The structure that rejects constraint-violating commands before execution, and the conditions its verification actually covered, can be presented from the same material, whether in a deployment decision or in an explanation to someone outside the company. It also becomes the basis for revalidating against the same criteria with every update.

Talk to us

A structure that stops the AI’s proposal short of execution is easier to handle when it is part of deciding the PoC configuration than when it is inserted later. In our custom AI development, robustness testing, adversarial resilience assessment, and audit log design are built into the development process for AI implementations, including edge and on-premises environments. To discuss the design of a system with a perception-decision-control loop, or a PoC, use the contact form.

Reference materials

The design items, the division of roles, and the suggested first steps in this article are U-Rec, Inc.’s general design proposals, informed by the sources above. They do not indicate conformance to a particular standard or the safety of a particular product.