NVIDIA adds company-data tools and prediction models to its AI agent toolkit

NVIDIA adds Cosmos tools to generate training data and test robots in simulation

Developers building robots and autonomous vehicles have new tools for creating hard-to-capture training examples and testing behavior before real-world trials. NVIDIA’s Cosmos expansion, announced at GTC Berlin, adds data workflows, model-training tools and simulation variants. The platform supports world models: AI systems that help machines understand and simulate the physical world.

Cosmos Data Factory combines tools for curating, labeling, generating and evaluating training data. Its new blueprints package those capabilities into five cloud-based workflows, covering driving footage, product defects, still images and video. Cosmos Curator, the platform’s GPU-accelerated video-data preparation tool, also adds support for inputs from multiple types of media and sensors.

Autonomous Vehicle Dataset Generation curates and labels driving data, then adds synthetic examples of scenarios that are difficult to capture.

Image Attribute Augmentation changes objects’ appearance to improve image search.

Event Video Generation turns still images into synthetic videos depicting rare or unsafe events.

Defect Image Generation creates examples of defects in printed circuit boards, metal and glass for inspection-model training.

Video Data Augmentation changes weather and lighting in existing footage while keeping the scene’s structure intact.

Developers can use complete workflows or individual capabilities, with agents coordinating the process. NVIDIA describes these reusable capabilities as core agent skills. Data Factory is available through GitHub, Microsoft Azure, Nebius and CoreWeave.

The Cosmos Framework update extends training, optimization and deployment tools beyond Cosmos 3 to more open models, including GR00T, Alpamayo, QWEN and Pi0.5. Developers and agents can adapt compatible models through fine-tuning and reinforcement learning, then optimize them through distillation and quantization before deployment. NVIDIA says the new post-training and optimization tools are intended to improve world models using less compute.

One example comes from robotics research lab MicroAGI. NVIDIA says the lab further trained Cosmos 3 on videos of humans handling objects, then distilled its 32-billion-parameter reasoning model into an eight-billion-parameter Nano model with no loss in accuracy. That example concerns a model specialized for manipulation viewed from the actor’s perspective.

New Cosmos 3 Nano simulation variants address the testing stage. They generate repeatable scenarios with multiple views and sensor types, including situations difficult to capture physically. Accompanying code lets developers adapt the models to train and evaluate control policies—the behavior governing a machine’s actions—for autonomous vehicles and two-arm robots in interactive simulations before real-world testing.

Porsche is using Data Factory through Voxel51 on Microsoft Azure to develop a shared autonomous-driving data workflow across Volkswagen Group brands. NVIDIA says the effort is expected to reduce repeated physical test drives by up to 90% and development cycle times by up to 85%. Those are projected reductions, not completed results.

For industrial inspection, ANYbotics generated more than 1,500 synthetic video clips to teach robots to recognize liquid spills and other changes between facility visits, according to NVIDIA. It curated real-world data and validated generated outputs using Encord’s platform. Ultralytics also combined real images with targeted synthetic defects to retrain its YOLO vision model. NVIDIA says that improved defect localization and recall—finding defects’ positions and catching more of them.

NVIDIA adds open AI models for business prediction, robots and science

NVIDIA is broadening its open AI portfolio beyond language agents to systems that predict business outcomes, prepare robot training data and design proteins. Its October 21 announcement at GTC Berlin brings seven model families together around a common promise: organizations can customize them with local data and deploy them on infrastructure they control.

The business-facing models split into distinct jobs. Nemotron handles language and visual information for agents. Nemotron Retriever supplies models for making information searchable, reordering results and extracting document contents. Kumo works with structured business data—records organized into tables and relationships—to forecast outcomes, score fraud risk and recommend actions.

Kumo Structured covers prediction, forecasting and anomaly detection, while Kumo Engine supplies libraries for developing and deploying those models. NVIDIA says developers can start with pretrained models, then fine-tune them on proprietary data. Its pitch is to avoid building a separate machine-learning pipeline for every prediction task.

Not everything debuted in Berlin. NVIDIA released the 120-billion-parameter Nemotron 3.5 Super VL on October 15 for specialized, always-on agents. It combines text and visual understanding for work involving PDFs, webpages, tables and charts.

Cosmos supplies world models for robots, autonomous vehicles and vision agents. New post-training and optimization tools aim to reduce computing needs, while simulation checkpoints let developers evaluate driving and robotic manipulation before real-world testing.

Cosmos Data Factory addresses the training material itself: curating, labeling, generating and evaluating data for driving scenarios, product defects and image augmentation. Developers can use complete workflows or individual capabilities, coordinated by agents.

Alpamayo 2 Edge instead works inside a vehicle. NVIDIA describes it as a compact reasoning model for real-time scene understanding and motion planning, built on Cosmos and optimized for its DRIVE AGX Thor platform.

BioNeMo RFD4-Proteina, developed with the University of Washington’s Baker Lab, generates protein sequences and three-dimensional structures. NVIDIA says it generates designs about 13 times faster than RFdiffusion3 in a benchmark using 200-amino-acid proteins. Researchers then evaluate candidates against design goals and molecular constraints before choosing which to test in the lab.

NVIDIA publishes Kumo Forecast to blend AI predictions with historical patterns

NVIDIA published Kumo Forecast on October 21, giving developers a forecasting model that can combine learned patterns with historical records supplied for a particular application. Released under Apache 2.0, it predicts multiple numerical measurements over time and includes optional tools for explaining which earlier observations and input variables influenced a forecast.

The direct route starts with MOMENT-1-large, a pretrained Transformer encoder with approximately 340 million parameters. That component turns the input measurements into representations the forecasting layer can use. NVIDIA keeps the encoder and its embedding weights fixed while training a separate forecasting head—the part that maps those representations to future values.

The second route adds a search through historical data. NVIDIA calls this Domain-Aware Representation and Retrieval, or DARR. It uses a nearest-neighbor method to find related historical patterns, then combines that information with the direct forecast. Retrieval is optional, so the same library supports predictions with or without this extra context.

Crucially, the recent input window and the retrieval archive are separate. The input window supplies observations for the current forecast; a separate historical table supplies patterns to retrieve. Developers can configure how much the final prediction relies on each route, rather than accepting a fixed balance between the two.

Forecast length has another distinction. The library offers configurable horizons, meaning how far ahead it predicts. Beyond the model’s native forecasting capability, it supports recursive extension: predictions feed into further predictions. A longer forecast produced this way is different from one generated directly within the model’s native horizon.

When enabled, the interpretability framework produces explanations for each forecast horizon. These include lag attributions, which identify the influence of earlier observations, and feature attributions, which identify the influence of input variables. The explanation bundle accompanies the forecast without requiring a different forecasting architecture.

The bundle also includes diagnostics for semantic flow and trajectory stability, plus an optional attribution method applied to embeddings. NVIDIA describes the framework as model-agnostic: the explanation tools are separate from the underlying forecaster rather than a replacement for its prediction mechanism.