Applications of Robotics

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

A Spider-Man stunt… executed by a robot. That says more about robotics than it seems. Disney Imagineers built a system that flies over 25 meters in the air → adjusting its motion in real time. → flips → rotation → speed control → balance All handled mid-flight. What stands out to me is not the spectacle… it’s the decision-making happening in the air. This isn’t scripted motion. It’s real-time adaptation to physics. That’s the shift. From robots that repeat actions to systems that respond to the environment And once that threshold is crossed, the implications extend far beyond entertainment: → high-risk environments → dynamic industrial tasks → real-world human assistance This is where things start to matter. Because the ability to adapt in real time is what turns machines into systems we can rely on. So here’s the real question: Where will real-time adaptive robotics create the most value next? #ArtificialIntelligence #Robotics #Innovation #FutureOfWork #Technology

computer vision + robotics 🔥 🔥 🔥 Over the last few days, I trained my Reachy Mini robot to track and follow a human face. I fine tuned RF-DETR Nano on a custom face detection dataset. The system maps pixel coordinates of a detected face to yaw and pitch commands for head control. I plan to release the code soon. The main issue is inertia. The robot head has significant mass. At higher angular velocities, inertia causes overshoot. During the next control step, the controller overcompensates. This behavior leads to oscillations. ⮑  RF-DETR: https://lnkd.in/dVQRpvWU #computervision #opensource #objectdetection #robotics

The 'monocopter' is a type of #micro #aerial #vehicle (MAV) largely inspired from the flight of botanical samaras (Acer palmatum). A large section of its fuselage forms the single wing where all its useful aerodynamic forces are generated, making it achieve a highly efficient mode of flight. However, compared to a multi-rotor of similar weight, monocopters can be large and cumbersome for transport, mainly due to their large and rigid wing structure. Overall, the vehicle weighs 69 grams, achieves a maximum lateral speed of about 2.37 ms−1, an average power draw of 9.78W and a flight time of 16 min with its semi-rigid wing. In this work, a monocopter with a foldable, semi-rigid wing is proposed and its resulting flight performance is studied. The wing is non-rigid when not in flight and relies on centrifugal forces to become straightened during flight. The wing construction uses a special technique for its lightweight and semi-rigid design, and together with a purpose-designed autopilot board, the entire craft can be folded into a compact pocketable form factor, decreasing its footprint by 69%. The proposed craft accomplishes a controllable flight in 5 degrees of freedom by using only one thrust unit. It achieves altitude control by regulating the force generated from the thrust unit throughout multiple rotations. Lateral control is achieved by pulsing the thrust unit at specific instances during each cycle of rotation. A closed-loop feedback control is achieved using a motion-captured camera system, where a hybrid Proportional Stabilizer Controller and Proportional-Integral Position Controller are applied. #research #paper: https://lnkd.in/gbtUTExx #authors: Shane Kyi Hla Win, Luke Soe Thura Win, Danial Sufiyan, Shaohui Foong #robotics #engineering #quadcopter #drones #innovation #technology

What happens when a robot loses a leg mid-mission? Most robots would fail immediately. But watch this one figure out how to walk again in just a few tries. The researcher deliberately damages the robot. Cuts off a leg. Adds weights. Attaches wheels to limbs. Each time, the robot experiments with different gaits until it finds one that works. This is omni-bodied intelligence. The software doesn't panic when the hardware changes. It adapts. Here's why this matters: we talk about robots in homes and factories, but we rarely talk about what happens after six months of use. Parts break. Joints wear out. Sensors fail. If robots can't handle imperfection, they'll never leave the lab. This approach treats adaptability as a core feature, not an edge case. That's the difference between a demo and a tool you can actually rely on. Video credits: SkildAI --- Interested in starting your robotics career? Check out our free robotics career guide to get you started: https://lnkd.in/gpPVTPKE

Will robots replace humans at Amazon?   It’s a question I’m often asked, and one I discussed with Jimmy McLoughlin OBE on Jimmy's Jobs of the Future Podcast.   There’s this perception that robots will take jobs away, but the reality is much more nuanced - and in many ways far more promising for Amazon and beyond.   When we first introduced robots into warehouses, some outside the company raised questions about job losses. But over the years, we’ve found that robotics has unlocked new opportunities. Not only did they double our capacity and boost productivity at Amazon, but they have also increased employment. In fact, our warehouses (or what we call fulfilment centres) with robotics employ 50% more people than traditional sites.   The reason?   Robotics has created demand for new roles that didn’t exist before. Yes, we still need people for picking, packing, and shipping, but now we also need robotic engineers, technicians, and specialists who can operate, maintain, and improve these systems.   As technology advances, the future of fulfilment isn’t about replacing people - it’s about expanding possibilities and creating more varied and specialised roles than ever before while delivering more for customers.   #Amazon #Innovation #Robotics

Production changes everything. What worked in a demo starts breaking at scale. That’s where real AI systems are tested. Here are the concepts that actually matter 👇 - Prototype vs production A demo works in controlled conditions, while production systems deal with scale, failures, and messy edge cases. - Training vs inference Training happens occasionally to build the model, while inference runs continuously to serve real users. - Batch vs real-time inference Batch is cost-efficient for large workloads, while real-time is critical when user experience depends on instant responses. - Accuracy vs reliability Accuracy looks good on test data, while reliability shows consistent performance under real-world conditions. - Guardrails vs validation Guardrails prevent unsafe outputs, while validation ensures correctness. Both are needed for safe and dependable systems. - Offline vs online evaluation Offline testing uses past data, while online evaluation measures real user impact. One doesn’t guarantee the other. - Data drift vs model drift Data drift changes inputs, while model drift shows performance degradation. Detecting this early avoids silent failures. - Monitoring vs observability Monitoring tracks known issues, while observability helps you understand unknown failures and system behavior. - Model hosting vs model serving Hosting deploys the model, while serving handles scaling, routing, and real-time requests. This is where complexity grows. - RAG vs fine-tuning RAG brings in fresh external knowledge, while fine-tuning embeds knowledge into the model. One adapts, the other is fixed. - Latency vs throughput Latency is response speed, while throughput is volume. Systems often fail because latency becomes too high. - Prompting vs fine-tuning Prompting shapes behavior through instructions, while fine-tuning changes model weights. Many real systems rely more on prompting. Understanding these trade-offs is what makes AI systems actually work. Which of these has been the toughest in your production setup?

Humanoids just got a human sense of touch — and it’s a game-changer. We’ve all seen robots walk, grasp, and even dance. But third-generation humanoids like Optimus, Figure, and XPENG are now moving from “motion” to “feeling.” The breakthrough? Ultra-thin tactile electronic skin — flexible fabrics thinner than 0.2 mm that can be tailored like a custom suit over the robot’s body. These aren’t simple pressure sensors. They detect gram-level touches, sense textures, feel objects slipping before they drop, and even map warmth and pressure in real time. Watch the demo 👇 A robot gets patted on the shoulder, hugged, and responds with live pressure mapping. The same fabric tech works as a smart mat that instantly visualizes every touch on a laptop screen. Why it matters: • Industrial dexterity jumps to a new level (no more dropped boxes) • Robots become safe enough for homes, hospitals, and eldercare • High-density sensor arrays (dozens per cm²) are now the new standard Market projection: The global flexible sensor industry is headed toward ~$4 billion by 2030. This is the final piece that turns robots from tools into true collaborative partners. The future isn’t just smarter robots — it’s robots that feel.

🦾 Great milestone for open-source robotics: pi0 & pi0.5 by Physical Intelligence are now on Hugging Face, fully ported to PyTorch in LeRobot and validated side-by-side with OpenPI for everyone to experiment with, fine-tune & deploy in their robots! π₀.₅ is a Vision-Language-Action model which represents a significant evolution from π₀ to address a big challenge in robotics: open-world generalization. While robots can perform impressive tasks in controlled environments, π₀.₅ is designed to generalize to entirely new environments and situations that were never seen during training. Generalization must occur at multiple levels: - Physical Level: Understanding how to pick up a spoon (by the handle) or plate (by the edge), even with unseen objects in cluttered environments - Semantic Level: Understanding task semantics, where to put clothes and shoes (laundry hamper, not on the bed), and what tools are appropriate for cleaning spills - Environmental Level: Adapting to "messy" real-world environments like homes, grocery stores, offices, and hospitals The breakthrough innovation in π₀.₅ is co-training on heterogeneous data sources. The model learns from: - Multimodal Web Data: Image captioning, visual question answering, object detection - Verbal Instructions: Humans coaching robots through complex tasks step-by-step - Subtask Commands: High-level semantic behavior labels (e.g., "pick up the pillow" for an unmade bed) - Cross-Embodiment Robot Data: Data from various robot platforms with different capabilities - Multi-Environment Data: Static robots deployed across many different homes - Mobile Manipulation Data: ~400 hours of mobile robot demonstrations This diverse training mixture creates a "curriculum" that enables generalization across physical, visual, and semantic levels simultaneously. Huge thanks to the Physical Intelligence team & contributors Model: https://lnkd.in/eAEr7Yk6 LeRobot: https://lnkd.in/ehzQ3Mqy

Could AI Robots Help Fill the Labor Gap? As a futurist field, embodied AI—also known as humanoids—is captivating. Labor shortages spurred by long-term demographic shifts, coupled with advances in generative AI, are accelerating the commercialization of robots designed to emulate human behavior. The global economy faces labor shortages due to demographic trends that may hinder growth for years. Concurrently, advancements in large language models and generative AI are poised to drive transformative innovations across various industries, from healthcare to manufacturing. These trends are likely to fuel the development of humanoids—advanced robots equipped with limbs and AI-powered "brains." The adoption of these humanoid robots might outpace that of autonomous vehicles, presenting significant opportunities for investors in companies developing these robots and their components, and industries integrating them into their workforce. Its worth noting that Adam Jonas, Head of Global Autos and Shared Mobility research at Morgan Stanley, notes the adaptability of humanoids: "Consider the vast array of tasks humans perform using just our hands or tools, and the numerous machines tailored for human dexterity. As the growth of the working-age population in advanced economies continues to decline, humanoids could become essential for industries struggling to attract sufficient labor to maintain productivity." Morgan Stanley analysts project that by 2040, the U.S. alone could have 8 million working humanoid robots, impacting wages by $357 billion. By 2050, this number could rise to 63 million, potentially affecting 75% of occupations, 40% of employees, and approximately $3 trillion in payroll. "The commercialization of humanoid robots will encounter significant challenges, particularly in gaining social and political acceptance, given their potential to disrupt a large portion of the global workforce," says Jonas. He highlights that up to 70% of construction jobs and 67% in farming, fishing, and forestry could be impacted. "While they may not be the ideal solution, they are an increasingly necessary one for a world facing significant longevity challenges." #HumanoidRobots #AILaborSolutions #FutureOfWork #LaborShortage #GenerativeAI #RoboticsInnovation #AIInvestment #EconomicGrowth #TechTrends #WorkforceTransformation #futures