How Frontier AI Models and NVIDIA Omniverse Accelerate Simulation Development

Turning a simulation idea into a practical application requires developers to assemble assets, connect physics and rendering, and verify that the scene behaves as intended. Frontier AI models, including GPT-6 Astra, can help developers explore scenarios, investigate failures, and build applications with the NVIDIA Omniverse Library.

Developers guide the AI agent with natural-language instructions, review the results, and direct further changes. The Omniverse Library provides GPU-accelerated physics, rendering, and sensor-simulation capabilities.

Explore the projects below to see frontier AI models in action, along with examples from NVIDIA teams and developers across the ecosystem.

Build a Humanoid Robot Simulator for Warehouse Environments

Explore gamified, physics-based control of humanoid robots from first- and third-person perspectives.

Before automating warehouse tasks, developers need an interactive simulation environment to explore task behavior and evaluate how work is completed. Frank DeLise, an Omniverse product manager at NVIDIA, used Astra to integrate warehouses and humanoid robots into an interactive simulator.

DeLise instructed Astra to connect the NVIDIA Omniverse Library for physics, scene updates, rendering, and user-interface functionality. He also used Astra with SimReady Foundation to create a physics scene for the simulation. Astra then generated animations and application code to integrate these features.

Learn how to prepare and validate SimReady robot assets with Frontier AI models and NVIDIA Omniverse libraries.

Connect Autonomous Driving Test Workflows

Changes to a scene, sensors, or driving model can significantly affect self-driving car simulations. Doyub Kim, manager of NVIDIA’s Simulation Technology team, commissioned Astra to build Zero to Alpamayo, a reusable simulation environment based on San Francisco’s Market Street.

Kim used Astra to plan the workflow, create assets and traffic, and connect Omniverse RTX Sensor Simulation with Alpamayo driving. The workflow checked each integration step by step.

The resulting prototype provided a testing ground for comparing models and tracking how changes in scenes and sensors affected downstream driving behavior. In another Cosmos3-Nano experiment, weather and lighting varied in recorded simulation videos, allowing Kim to compare the driving model’s response to the same scenario under different conditions.

Create and Improve Digital Twins with Sensor Comparison

Comparing recorded and simulated camera and LiDAR outputs gives developers feedback based on key performance indicators, helping them adjust a scene.

To test robots and self-driving cars, developers need to understand how closely simulated sensors match real sensors. Ashley Reid, who works on RTX sensor validation at NVIDIA, asked agents to compare camera and raw LiDAR output from ovrtx with recorded data.

The agents created two digital twins from scratch and improved two existing ones. Over approximately three days, Reid guided the agents through an iterative workflow that measured differences, created or modified OpenUSD scenes, and reviewed the results.

Updates addressed missing objects, geometry, and materials. Changes were accepted according to camera and LiDAR metrics, giving developers a way to use measured discrepancies to guide scene creation and improvement.

Start by rendering the OpenUSD scene with the ovrtx minimal Python example, then define the sensor measurements to compare with recorded data.

Robo Olympics: Test Robot Skills in Simulation

Teaching a robot a new behavior requires testing whether it works under physical constraints. Tae Kim, head of engineering and products for NVIDIA Omniverse, guided Astra in building Robo Olympics, an experimental project that uses sports videos and natural-language instructions to test a simulated Unitree G1 humanoid performing sports movements.

Under Kim’s direction, Astra built the controller and refined it through physical testing. The project used the Newton physics engine to simulate behavior, the open-source NVIDIA Warp framework for fast calculations, and ovrtx to render the scene and virtual camera images.

In one experiment, the robot cleared one hurdle in 64 out of 100 simulation trials. The test provided feedback that Kim could use to improve the robot’s timing and control.

Explore architectural simulation using Astra and NVIDIA Omniverse libraries. Watch the example.

Test Robot Disassembly with CAD and Simulation

Before a robot can disassemble a product, developers need to know whether its tools can reach and remove the required part. Jens Jebens, OpenUSD senior product manager at NVIDIA, asked Astra to model and configure a car suspension in PTC Onshape and NVIDIA Isaac Sim.

Using computer-aided design and simulation, Jebens explored revisions to the robot’s tools. The agent measured the available space and designed a wrench that the robot could use to reach the suspension bolts.

Jebens reported successful removal of suspension components in the simulation. This connects design and tooling decisions to disassembly results and provides a starting point for training robot policies.

Explore the Onshape Importer Guide.

Bring the International Space Station into a Browser

Transforming a 3D model into an application requires developers to connect assets, live data, and interfaces. Nick Johns, NVIDIA’s director of engineering, integrated NASA assets into Astra, including an OpenUSD model of the International Space Station with telemetry.

Johns built the application with a single prompt and then used follow-up prompts to move the scene to the daytime side of Earth so the planet would be visible.

The workflow used Blender to prepare assets and the NVIDIA Omniverse Library for rendering with ovrtx, scene runtime with ovstage, and streaming with ovstream. The application brings 3D models and operational data into a browser while Johns guides development through prompts and modifications.

Try the Omniverse Real-Time Viewer Skill.

Turn a Captured Room into a Testing Environment

Digitally reconstructed rooms need editable objects and accurate physical behavior before developers can test interactions. Chirag Majithia from NVIDIA’s Isaac Engineering Applications team tasked Astra with converting stereo-camera captures into editable OpenUSD environments.

The workflow combined PyCuSFM, FoundationStereo, and nvblox to guide object selection and placement with user reviews. Astra assembled generated and Blender-created assets and used USD Content Agents to configure how objects would move and interact within the simulation.

Isaac Sim guided and tested collision and contact fixes for doors and drawers. The workflow connects captured geometry to interaction tests, making it easier to inspect gaps and object behavior.

Explore USD Content Agents.

Start Building with AI Agents

Have a simulation idea? Explore the NVIDIA Omniverse Library to start building with AI agents.