The robotics platform is adding new foundation models, a robot learning framework, and tools for AI workflow orchestration and robot perception.
The NVIDIA Isaac robotics platform is tapping into the latest generative AI and advanced simulation technologies to accelerate AI-enabled robotics.
At GTC today, NVIDIA announced Isaac Manipulator and Isaac Perceptor - a collection of foundation models, robotics tools and GPU-accelerated libraries.
On stage before a crowd of 10,000-plus, NVIDIA founder and CEO
'Building foundation models for general humanoid robots is one of the most exciting problems to solve in AI today,' said Huang. 'The enabling technologies are coming together for leading roboticists around the world to take giant leaps toward artificial general robotics.'
NVIDIA also announced a new computer for humanoid robots based on the NVIDIA Thor system-on-a-chip, and new tools for the NVIDIA Isaac robotics platform, including
Introducing Isaac Manipulator for Robotic Arms
NVIDIA Isaac Manipulator offers a collection of state-of-the-art motion generation and modular AI capabilities for robotic arms, with a robust collection of foundation models and GPU-accelerated libraries.
Robotics developers can use combinations of software components customized for specific tasks to perceive and interact with surroundings, enabling the building of scalable and repeatable workflows for dynamic manipulation tasks by accelerating AI model training and task programming.
'Incorporating new tools for foundation model generation into the Isaac platform accelerates the development of smarter, more flexible robots that can be generalized to do many tasks,' said
Leading robotics companies Yaskawa, Solomon, PickNik Robotics, READY Robotics, Franka Robotics, and Universal Robots, a Teradyne company, are partnering with NVIDIA to bring Isaac Manipulator to their customers.
'By bringing NVIDIA AI tools and capabilities to Yaskawa's automation solutions, we're pushing the boundaries of where robots can be deployed across industries,' said
NVIDIA is introducing foundation models to augment existing robot manipulation systems. These will help develop robots to sense, adapt and reprogram for varied environments and applications in smart manufacturing, handling pick-and-place tasks, machine tending and assembly with the following:
FoundationPose is a pioneering foundation model for 6D pose estimation and tracking of previously unseen objects.
cuMotion taps into the parallel processing of NVIDIA GPUs for solving robot motion planning problems at industrial scale by running many trajectory optimizations at the same time to provide the best solution.
FoundationGrasp is a transformer based model that can make dense grasp predictions for unknown 3D objects.
SyntheticaDETR is an object detection model for indoor environments that allows faster detection, rendering and training with new objects.
Introducing Isaac Perceptor for Autonomous Mobile Robots Visual AI
Manufacturing and fulfillment operations are adopting autonomous mobile robots (AMRs) to improve efficiency and worker safety as well as to reduce error rates and costs.
Isaac Perceptor provides multi-camera, 360-degree vision capabilities, offering early industry partners such as
The NVIDIA Nova Orin DevKit - created in collaboration with Segway Robotics and Leopard Imaging - allows companies to quickly develop, evaluate and deploy Isaac Perceptor.
'
Project GR00T for Humanoid Robotics Development Takes a Bow
Demonstrated at GTC, GR00T-powered humanoid robots can take multimodal instructions - text, video and demonstrations - as well as their previous interactions to produce the desired action for the robot. GR00T was shown on four humanoid robots from different companies, including Agility Robotics, Apptronik, Fourier Intelligence and Unitree Robotics.
Humanoid robots are complex systems that require heterogeneous computing to meet the needs of high frequency low level controls, sensor fusion and perception, task planning and human-robot interaction. NVIDIA unveiled a new Jetson Thor-based computer for humanoid robots, built on the NVIDIA Thor SoC.
Jetson Thor includes a next-generation GPU based on the NVIDIA Blackwell Architecture with a transformer engine delivering 800 teraflops of 8-bit floating point AI performance to run multimodal generative AI models like GR00T. With an integrated functional safety processor, a high-performance CPU cluster and 100GB of ethernet bandwidth, it significantly simplifies design and integration efforts.
Project GR00T uses Isaac tools that are available to robotics developers for building and testing foundation models. These include
Accelerating Robot Learning With
Robots that require advanced locomotion skills, whether with walking or grasping, need to use deep reinforcement learning in a simulated environment and be trained repeatedly in a virtual environment to learn skills. However, this utility becomes more useful when the model transfers to the real robot deployment, which has been demonstrated with Project GR00T.
As the successor to
Enabling Cloud-Native Robotics Workflow Scheduling With NVIDIA OSMO
NVIDIA OSMO scales workloads across distributed environments. For robotics workloads with complex multi-stage and multi-container workflows, the platform provides a location-agnostic deployment option and dataset management and traceability features for deployed models.
'Boston Dynamics employs a range of machine learning, reinforcement learning and AI technologies to power our robots,' said
OSMO supports GR00T, for example, by concurrently running models on NVIDIA DGX for training and NVIDIA OVX servers for live reinforcement learning in simulation. This workload involves generating and training models iteratively in a loop. OSMO's ability to manage and schedule workloads across distributed environments allows for the seamless coordination of DGX and OVX systems, enabling efficient and iterative model development. Once the model is ready for testing and validation, OSMO can uniquely orchestrate software-in-the-loop workflows on OVX (x86-64) as well as hardware-in-the-loop workflows with NVIDIA Jetson (aarch64) compute resources.
Supporting the ROS Ecosystem of Developers
NVIDIA joined the
'The increasing capability of autonomous robots is driving a rise in demand for more powerful but still energy-efficient onboard computing,' said
NVIDIA Isaac Perceptor with
(C) 2024 Electronic News Publishing, source