Editor’s note: This article is part of To Omniverse, a series highlighting how developers, 3D artists, and businesses can leverage cutting-edge technology to enhance their workflows, including OpenUSD and NVIDIA Omniverse.
Vision AI agents are increasingly being utilized to seamlessly convert video data from real-world settings into actionable intelligence for factories, urban environments, warehouses, and transportation networks.
The shift towards this technology is being accelerated as more AI workloads are processed closer to the data source. Gartner forecasts that by 2028, over two-thirds of enterprise data will be created and managed outside traditional data centers or the cloud, with more than 66% of global enterprises expected to adopt edge AI by 2029, a significant leap from just 10% in 2025 (1).
However, an increase in edge-generated data does not inherently equate to greater intelligence. According to the same Gartner report, a staggering 90% of existing edge data remains unutilized.
Effectively converting this data into meaningful actions necessitates a Vision AI agent capable of comprehending video content, adapting to real-world scenarios, and integrating insights into operational workflows. These agents generally operate close to cameras, machinery, and sensors, requiring tailored models that cater to specific site conditions while also addressing latency, power, cost, and connectivity constraints.
To develop these agents, developers require a consistent method for producing training data, refining models, and seamlessly deploying AI video applications across both edge and cloud infrastructures.
NVIDIA Metropolis offers agent skills and blueprints, equipping developers with reusable workflows to create, manage, and enhance Vision AI agents throughout their lifecycle.
Using synthetic data is crucial in this process, providing a unified framework for describing, assembling, and reusing 3D environments. Built upon OpenUSD, NVIDIA Omniverse facilitates the development of simulations, synthetic data generation, and digital twin workflows to model real-world scenarios, effectively broadening the range of possible conditions, including variations in lighting, weather, traffic, camera angles, occlusions, and rare occurrences.
Common Pitfalls in Vision AI Agent Projects
As organizations progress towards deploying autonomous Vision AI agents, they often encounter three primary challenges:
- Data Gaps Leading to Stagnant Accuracy: Vision AI agents must detect rare defects, atypical events, and environmental changes. For instance, in manufacturing, while an inspection model might excel at identifying common scratches and dents, it might falter when faced with subtle hairline cracks not represented in the training dataset.
- Insufficient Fine-Tuning Expertise: After identifying performance deficiencies, organizations often struggle to implement improvements effectively. Fine-tuning necessitates labeled datasets, training configurations, experiment documentation, and evaluation, which many companies lack, especially those with limited in-house ML teams managing operations across diverse sites or products.
- Complex and Lengthy Agent Assembly Processes: Deploying a Vision AI agent involves more than mere inference. Developers must integrate video pipelines, AI models, metadata, embedding, indexing, search functionality, alerting, reporting, and system integration. Customizing these workflows for specific environments significantly extends development time and necessitates specialized expertise. Without OpenUSD’s shared scene description framework, teams often recreate 3D environments from scratch each time deployment conditions or locations change.
A Comprehensive Approach to Vision AI Agents
Utilizing NVIDIA agent skills and blueprints, in conjunction with NVIDIA Omniverse for OpenUSD-based simulations, synthetic data generation, and NVIDIA Metropolis for model development and deployment of video AI, provides developers with a reliable starting point for various workflow components.
By harnessing these reusable workflows, developers can expedite the data generation process, enhance models, and deploy Vision AI agents more efficiently.
Visual Inspection: Generating Defect Data Not Found on the Production Line
In manufacturing, as factories become adept at preventing defects, gathering sufficient instances of imminent faults to train subsequent inspection models becomes increasingly challenging.
Roboflow integrates NVIDIA’s Defect Image Generation skill into its platform. NVIDIA Cosmos World Base Model is being utilized to generate synthetic defect images for customers like Corning, enabling near-perfect detection rates while drastically cutting down on the necessity for daily manual image evaluations.
In evaluations conducted with engineers from Corning’s fiber optic production team, a model that was trained using just eight actual defect images (augmented with synthetic data from NVIDIA’s defect image generation capabilities) achieved an average precision of 95% and flawless recall for the most challenging defect types. This performance surpassed that of a baseline model trained exclusively on authentic data and significantly reduced what would have been a multi-quarter inspection project to mere days.
Discover how synthetic data generation workflows can empower developers to produce the necessary data for training and refining physical AI models.
Smart Cities: Evolving from Video Analytics to Autonomous Operations
Large-scale urban operational systems illustrate why Vision AI agents necessitate interconnected workflows rather than just cognitive abilities.
Linker Vision is developing AI systems for smart cities using the NVIDIA Metropolis Blueprint for VSS to hasten the roll-out of video inference agents within city infrastructures. This workflow incorporates VSS skills to streamline common video AI tasks, including search, summarization, alerting, reporting, and stream management into reusable workflows executable by agents.
The OpenUSD-based NVIDIA Omniverse digital twin empowers you to model urban environments and simulate how Vision AI systems react to varying traffic flows, weather conditions, emergencies, and infrastructure modifications. Linker Vision employs NVIDIA Cosmos for Video Data Expansion and NVIDIA TAO for fine-tuning Cosmos models.
In Kaohsiung, Linker Vision achieved an 85% reduction in development effort and slashed incident response times by up to 80% using VSS blueprints. The new AI-GRID extension builds upon this methodology. NVIDIA Nemo Crow serves as a blueprint for secure agent AI, facilitating autonomous video inference across urban and transportation settings.
Industrial Operations: Understanding Work in Progress
In industrial environments, the challenge extends beyond recognizing what appears in a video frame. Your agents must be capable of:
- Verifying if tasks are executed accurately
- Comparing ongoing operations to standard protocols
- Generating insights prior to defects propagating downstream in the process.
At Foxconn, DeepHow Live SOP Validation Agent harnesses NVIDIA Metropolis VSS Blueprints to serve as a video workflow layer for search, summarization, and analysis in production settings. NVIDIA Cosmos enhances reasoning capabilities, assisting agents in interpreting intricate human activities and sequences of work, ensuring that assembly tasks are performed correctly and in the designated order.
This solution was implemented on the NVIDIA GB300 server production line, leading to a 3% increase in first-pass yield, achieving 99% task-level accuracy in understanding the micro-actions associated with crucial SOP steps, while also alleviating redundant tasks by enabling teams to spot issues early in the chain.
To learn how developers can create and deploy video analytics AI agents, check out this technical tutorial. NVIDIA VSS Skills with Coding Agent.
Explore NVIDIA agent skills and blueprints to build, manage, and enhance Video Analytics AI Agents.
Source: Gartner, “Predicting 2026: Physical AI Will Drive I&O to the Edge,” March 3, 2026. Gartner is a trademark of Gartner, Inc. and/or its affiliates.
Source: blogs.nvidia.com