Deploying Machine Learning Policies for Robots

Shipping AI agents into production without governance is like deploying software without security, logs, or controls. It might work at first. But sooner or later, something breaks - silently. As AI agents move from experiments to real decision-makers, governance becomes infrastructure. This framework breaks AI Governance into the core functions every production-grade agent system needs: - Policy Rules Turn business and regulatory expectations into enforceable agent behavior - defining what agents can do, must avoid, and how they respond in restricted scenarios. - Access Control Limits agents to approved tools, datasets, and systems using identity verification, RBAC, and permission boundaries — preventing accidental or malicious misuse. - Audit Logs Create a full activity trail of agent decisions: what data was accessed, which tools were called, and why actions were taken — making every outcome traceable. - Risk Scoring Evaluates agent actions before execution, assigns risk levels, detects sensitive operations, and blocks unsafe decisions through thresholds and safety scoring. - Data Privacy Protects confidential information using PII detection, encryption, consent management, and retention policies — ensuring agents don’t leak regulated data. - Model Monitoring Tracks real-world agent performance: accuracy, drift, hallucinations, latency, and cost - keeping systems reliable after deployment. - Human Approvals Adds human-in-the-loop controls for high-impact actions, enabling escalation, overrides, and sign-offs when automation alone isn’t enough. - Incident Response Detects failures early and enables rapid containment through alerts, rollbacks, kill switches, and post-incident reporting to prevent repeat issues. The takeaway: AI agents don’t just need intelligence. They need guardrails. Without governance, agents become unpredictable. With governance, they become enterprise-ready. This is how organizations move from experimental AI to trustworthy, compliant, production systems. Save this if you’re building agentic systems. Share it with your platform or ML teams.

If your AI agent fails in production, it’s rarely the model. It’s ungoverned reasoning. My production shortlist: 17 reasoning and planning algorithms. Not a prompt issue. Not a model issue. Not a framework issue. It’s a systems problem inside the agent. Here is the reality: ✸ The agent reasons well, then gets stuck with no backtracking ✸ The plan looks clean, then becomes outdated after one new observation ✸ The agent explores, but nothing is scoring paths so it wanders ✸ Costs spike because depth was never bounded ✸ Latency creeps in because replanning has no trigger rules Most builders pick a reasoning method like a feature. In production, reasoning and planning are a control policy. → Pick how the agent explores → Define how it evaluates progress → Set boundaries for depth, cost, and time → Decide when it replans → Make failure behavior explicit System shift: ✖ Do not ask: “Which algorithm is best?” → Ask: “Which policy fits my failure mode and budget?” Here is the checklist I use to select and deploy reasoning and planning methods: 𝟭 Classify the task uncertainty Low, medium, high uncertainty changes everything. 𝟮 Choose a planning posture Reactive, plan-first, or search-first. 𝟯 Separate plan from execution state Plans expire. State persists. 𝟰 Define replanning triggers Observation change, constraint breach, confidence drop. 𝟱 Bound depth and breadth Hard caps beat hope. 𝟲 Install path scoring No evaluation means no direction. 𝟳 Add backtracking conditions When is reversal allowed and required? 𝟴 Decide sampling strategy Single path, multi-path, or vote. 𝟵 Control iteration loops Reflection and refinement need exit rules. 𝟭𝟬 Support parallelism carefully Concurrency without dependency control creates chaos. 𝟭𝟭 Use hierarchy when goals are nested Goal → subgoal → action. 𝟭𝟮 Add graceful degradation When budgets tighten, reduce depth not correctness. 𝟭𝟯 Log traces for audit If you cannot inspect it, you cannot improve it. 𝟭𝟰 Measure cost per solved task Track tokens, steps, retries, latency. 𝟭𝟱 Validate against real edge cases Not curated examples. 𝟭𝟲 Freeze the policy per environment Dev, staging, prod need different bounds. 𝟭𝟕 Treat the method as swappable infrastructure Policies evolve, architecture stays stable. AI agents do not fail because they lack intelligence. They fail because reasoning and planning are left ungoverned. ✍️What is the most unstable part of your agent today? --- Join 46,000+ engineers building production-grade AI agents. Start with the 30-minute Zero → Hero training: https://lnkd.in/g-knYN9T

It is really hard to get robot policies that are both precise (small margins for error) and general (robust to env variations). We just released ViSk, where skin sensing is used to train fine-grained policies with ~1 hour of data. I have attached a single-take video on this post. The key technical idea in ViSk is that simply treating skin-based touch data as tokens for a transformer is enough to get multi-modal (vision+touch) policies. An empirical insight is that skin-sensing significantly improves generalization to positions, size, shape, and type of objects. This means that you do not need to collect 1000s of demos. All of our tasks needed <200 demos to train. This work was led by Venkatesh Pattabiraman and Raunaq Bhirangi with Yifeng Cao and Siddhant Haldar. More details (paper, code and videos) are here: https://lnkd.in/e8QANsjs

Robot models get better only when humans feed them more demos. This one improves by learning from its own mistakes. pi*0.6 is a new VLA from Physical Intelligence, that can refine its skills through real-world RL, not just teleop data. The team calls the method Recap, and from what I can see, the gains are not small. A quick summary: ✅ Learns from its own rollouts using a value function trained across all data ✅ Humans only step in when the robot is about to drift too far ✅ Every correction updates the model and improves future rollouts ✅ Works across real tasks like espresso prep, laundry, and box assembly ✅ Throughput more than doubles on hard tasks, with far fewer failure cases What stands out is the structure: a general policy, a shared value function, and a loop where the robot collects data, improves the critic, then improves itself again. No huge fleets of teleoperators. No massive manual resets. If VLAs can reliably self-improve in the real world, the bottleneck shifts. Data becomes cheaper. Deployment becomes the real test bench. Full paper, videos, and method details here: https://lnkd.in/dgCeZdjT

Robotics data is expensive and slow to collect. A lot of videos are available online, but not readily usable by robotics because of lack of action labels. AMPLIFY solves this problem by learning Actionless Motion Priors that unlock better sample efficiency, generalization, and scaling for robot learning. Our key insight is to factor the problem into two stages: The "what": Predict the visual dynamics required to accomplish a task The "how": Map predicted motions to low-level actions This decoupling enables remarkable generalizability: our policy can perform tasks where we have NO action data, only videos. We outperform SOTA BC baselines on this by 27x 🤯 AMPLIFY is composed of three stages: 1. Motion Tokenization: We track dense keypoint grids through videos and compress their trajectories into discrete motion tokens. 2. Forward Dynamics: Given an image and task description (e.g., "open the box"), we autoregressively predict a sequence of motion tokens representing how keypoints should move over the next second or so. This model can train on ANY text-labeled video data - robot demonstrations, human videos, YouTube videos. 3. Inverse Dynamics: We decode predicted motion tokens into robot actions. This module learns the robot-specific mapping from desired motions to actions. This part can train on ANY robot interaction data - not just expert demonstrations (think off-task data, play data, or even random actions). So, does it actually work? Few-shot learning: Given just 2 action-annotated demos per task, AMPLIFY nearly doubles SOTA few-shot performance on LIBERO. This is possible because our Actionless Motion Priors provide a strong inductive bias that dramatically reduces the amount of robot data needed to train a policy. Cross-embodiment learning: We train the forward dynamics model on both human and robot videos, but the inverse model sees only robot actions. Result: 1.4× average improvement on real-world tasks. Our system successfully transfers motion information from human demonstrations to robot execution. And now my favorite result: AMPLIFY enables zero-shot task generalization. We train on LIBERO-90 tasks and evaluate on tasks where we’ve seen no actions, only pixels. While our best baseline achieves ~2% success, AMPLIFY reaches a 60% average success rate, outperforming SOTA behavior cloning baselines by 27x. This is a new way to train VLAs for robotics which dont always start with large scale teleoperation. Instead of collecting millions of robot demonstrations, we just need to teach robots how to read the language of motion. Then, every video becomes training data. led by Jeremy Collins & Loránd Cheng in collaboration with Kunal Aneja, Albert Wilcox, Benjamin Joffe at College of Computing at Georgia Tech Check out our paper and project page for more details: 📄 Paper: https://lnkd.in/eZif-mB7 🌐 Website: https://lnkd.in/ezXhzWGQ

𝐑𝐞𝐢𝐧𝐟𝐨𝐫𝐜𝐞𝐦𝐞𝐧𝐭 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐈𝐬 𝐍𝐨𝐭 𝐄𝐯𝐨𝐥𝐯𝐢𝐧𝐠 𝐑𝐨𝐛𝐨𝐭𝐢𝐜𝐬. 𝐈𝐭 𝐈𝐬 𝐑𝐞𝐰𝐢𝐫𝐢𝐧𝐠 𝐈𝐭. Reinforcement learning has crossed the line from academic promise into measurable industrial and real world dominance. Robots are no longer executing hand coded instructions. They are learning through consequence, adapting through uncertainty, and improving through reward. This is the moment where automation becomes intelligence. In high fidelity simulation environments, modern RL policies now achieve performance levels that were considered unattainable just a few years ago. In a recent dual arm robotic assembly system, the policy reached a 99.8 percent success rate across 35,000 training episodes. Mean cycle times stabilized at under five seconds while maintaining precision insertion under randomized joint noise. This is not marginal improvement. This is near perfect reliability in a task that historically caused massive failure rates under traditional control. When transferred into the physical world, those same learned behaviors did not collapse. They improved. This is what true autonomy looks like. Not scripted motion. Adaptive force, perception, and decision making in real time. Virtual reality is now accelerating that loop even further. In distributed supervisory control systems that combine immersive VR interfaces with deep reinforcement learning, operators issue high level goals while autonomous policies execute low level motion. In recent trials, this hybrid architecture reduced task completion time by over 50 percent and eliminated collisions entirely. Operator workload dropped significantly while system usability scores exceeded 84 out of 100. Human intent and machine intelligence are no longer competing. They are converging. At scale, reinforcement learning is now coordinating swarms of autonomous systems using graph based policies that distribute decision making across hundreds of agents. Efficiency gains exceeding 90 percent in cooperative tasks such as navigation, sensing, and area coverage are now being reported. At the edge, quantized RL models running on compact hardware are executing real time inference under extreme size, weight, and power constraints. Autonomy is moving out of the lab and into everything. The deeper truth is this: We are no longer programming robots. We are training them. Simulation builds the mind. Real world deployment proves it. Virtual reality sharpens it. Multi agent learning scales it. Reinforcement learning is becoming the nervous system of the next generation of machines. And the results are no longer theoretical. They are measurable, repeatable, and already reshaping what autonomy means. #changetheworld

Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states? Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖 Quickly and dynamically moving around and in-between obstacles (motion planning) is a crucial skill for robots to manipulate the world around us. Traditional methods (sampling, optimization or search) can be slow and/or require strong assumptions to deploy in the real world. Instead of solving each new motion planning problem from scratch, we distill knowledge across millions of problems into a generalist neural network policy.  Our Approach: 1) large-scale procedural scene generation 2) multi-modal sequence modeling 3) test-time optimization for safe deployment Data Generation involves: 1) Sampling programmatic assets (shelves, microwaves, cubbys, etc.) 2) Adding in realistic objects from Objaverse 3) Generating data at scale using a motion planner expert (AIT*) - 1M demos! We distill all of this data into a single, generalist policy Neural policies can hallucinate just like ChatGPT - this might not be safe to deploy! Our solution: Using the robot SDF, optimize for paths that have the least intersection of the robot with the scene. This technique improves deployment time success rate by 30-50%! Across 64 real-world motion planning problems, Neural MP drastically outperforms prior work, beating out SOTA sampling-based planners by 23%, trajectory optimizers by 17% and learning-based planners by 79%, achieving an overall success rate of 95.83% Neural MP extends directly to unstructured, in-the-wild scenes! From defrosting meat in the freezer and doing the dishes to tidying the cabinet and drying the plates, Neural MP does it all! Neural MP generalizes gracefully to OOD scenarios as well. The sword in the first video is double the size of any in-hand object in the training set! Meanwhile the model has never seen anything like the bookcase during training time, but it's still able to safely and accurately place books inside it. Since, we train a closed-loop policy, Neural MP can perform dynamic obstacle avoidance as well! First, Jim tries to attack the robot with a sword, but it has excellent dodging skills. Then, he adds obstacles dynamically while the robot moves and it’s still able to safely reach its goal. This work is the culmination of a year-long effort at Carnegie Mellon University with co-lead Jiahui(Jim) Yang as well as Russell Mendonca, Youssef Khaky, Russ Salakhutdinov, and Deepak Pathak The model and hardware deployment code is open-sourced and on Huggingface!  Run Neural MP on your robot today, check out the following: Web: https://lnkd.in/emGhSV8k Paper: https://lnkd.in/eGUmaXKh Code: https://lnkd.in/e6QehB7R News: https://lnkd.in/enFWRvft

Most robotics companies don’t fail because their hardware is weak. They fail because their AI does not survive reality. The moment conditions drift — lighting, clutter, wear, timing, human interaction — perception breaks, reasoning degrades, and recovery logic explodes in complexity. What looks like an AI problem quickly becomes a business problem: stalled deployments, high integration cost, and products that never scale. I’ve been writing a growing document on this exact gap: How to Build Physical AI https://lnkd.in/gZXk9TJX This is an AI-first blueprint, written for teams building real robots, not demos. The premise is simple: Physical AI must reason over time, geometry, and uncertainty — not just predict from pixels. In the document, we outline how modern AI stacks fall short and what replaces them: • World models grounded in physics, not frame-by-frame perception, so policies generalize across sites and conditions • Long-horizon reasoning that remains stable as the environment unfolds in real time • Edge-native AI architectures that close the loop between perception, decision, and action without cloud dependence • Learning systems that compound, instead of resetting every time you introduce a new SKU, layout, or task variant For CEOs, this directly impacts rollout velocity, gross margins, and customer confidence. For CTOs, it determines whether your AI roadmap compounds or collapses under edge cases. Physical AI is not about adding more models. It’s about building an intelligence layer that understands the physical world well enough to act in it — reliably, repeatedly, and at scale. This document will continue evolving as the field matures. If you’re building AI that has to run on real machines, in real environments, I’d love your perspective. #AI #PhysicalAI #Robotics #Autonomy #EngineeringLeadership Dragomir Anguelov Aditya Jami Steve Cousins Oliver Cameron

These days, it's easy to train neural networks. But deploying autonomous robots in the real world takes more than that. Watch as EarthSense, Inc.'s TerraPreta robot completes an autonomous lane turn, fusing AI with odometry, navigation algorithms, and heuristics. Anyone can pip install Ultralytics and train up a basic YOLO segmenter. Anyone can make a cool video. But when you have the robot in front of you, tasked with deploying to a new field, you run into endless problems. ⚠️ Where should you place your cameras? ⚠️ What do you actually want your neural network to see? ⚠️ How do you run inference on all of your cameras on edge? ⚠️ How can you convert 2D detections into a 3D map? ⚠️ How many pixels of error until your robot crashes into the corn? ⚠️ Are you sure you need AI for this? An AI solution needs a dataset solution. A labeling solution. A training solution. An architecture solution. An evaluation solution. An edge deployment solution. And going through it all again if you run into problems. AI by itself will only get you so far. Eventually you'll need control theory, old school computer vision, and handcrafted heuristics. ✅ Fusing camera with LiDAR and IMU ✅ Odometry ✅ Mapping algorithms ✅ Domain Knowledge AI powers our autonomy, not alone, but with help from many other areas of research. How does your organization provide support for your AI in deployment? #AI #PrecisionAg #FieldRobotics #ComputerVision #AgTech #Autonomy

Run Reinforcement Learning Policy with ROS 2 and Isaac Sim If you want to try reinforcement learning in action inside a ROS 2 + Isaac Sim setup, this tutorial from NVIDIA Isaac Sim documentation is a great place to start. It walks you through how to connect a trained locomotion policy with the H1 humanoid robot in simulation, letting you send commands through ROS 2 topics and watch the policy drive the robot in real time. What you’ll learn: ✅ Set up ROS 2 nodes to publish observations and receive actions. ✅ Run a reinforcement learning policy inside a simulated environment. ✅ Configure the robot’s IMU to provide motion data for control feedback. ✅ Control the simulated robot live using ROS 2 twist commands. If you’re working with robot learning, sim-to-real, or just curious how RL fits into ROS 2 pipelines, this tutorial is definitely worth a look. 🔗 Check the tutorial to start your ROS 2 + Isaac Sim journey! Have you already tried integrating Isaac Sim into your ROS 2 projects? Let's connect and share robotics insights 🔽 #ROS2 #Robotics #AI #Nvidia