Physical AI
Collection
Collection of open, commercial-grade datasets for physical AI developers • 54 items • Updated • 175
Dataset Viewer
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
id: string
task: string
category: string
media: string
conversations: list<item: struct<from: string, value: string>>
child 0, item: struct<from: string, value: string>
child 0, from: string
child 1, value: string
_meta: struct<source: string, split: string, expression_id: int64, sentence: string, object_category: strin (... 63 chars omitted)
child 0, source: string
child 1, split: string
child 2, expression_id: int64
child 3, sentence: string
child 4, object_category: string
child 5, image_width: int64
child 6, image_height: int64
child 7, is_no_target: bool
bcq: list<item: struct<id: string, video: string, system_prompt: string, question: string>>
child 0, item: struct<id: string, video: string, system_prompt: string, question: string>
child 0, id: string
child 1, video: string
child 2, system_prompt: string
child 3, question: string
to
{'bcq': List({'id': Value('string'), 'video': Value('string'), 'system_prompt': Value('string'), 'question': Value('string')})}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 489, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2818, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2355, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2380, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 536, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 419, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2369, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2297, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
id: string
task: string
category: string
media: string
conversations: list<item: struct<from: string, value: string>>
child 0, item: struct<from: string, value: string>
child 0, from: string
child 1, value: string
_meta: struct<source: string, split: string, expression_id: int64, sentence: string, object_category: strin (... 63 chars omitted)
child 0, source: string
child 1, split: string
child 2, expression_id: int64
child 3, sentence: string
child 4, object_category: string
child 5, image_width: int64
child 6, image_height: int64
child 7, is_no_target: bool
bcq: list<item: struct<id: string, video: string, system_prompt: string, question: string>>
child 0, item: struct<id: string, video: string, system_prompt: string, question: string>
child 0, id: string
child 1, video: string
child 2, system_prompt: string
child 3, question: string
to
{'bcq': List({'id': Value('string'), 'video': Value('string'), 'system_prompt': Value('string'), 'question': Value('string')})}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
VANTAGE-BENCH
Video ANalysis Tasks Across Generalized Environments
Dataset Description
VANTAGE-BENCH is the first public benchmark purpose-built for evaluating visual understanding on video captured by fixed infrastructure cameras. It spans three real-world domains — warehouse, smart city / Intelligent Transportation Systems (ITS), and smart spaces — across six spatio-temporal video understanding tasks including video question answering (VQA), temporal grounding, dense video captioning, event verification, spatial grounding, and spatio-temporal tracking.
This dataset is for evaluation purposes only.
Dataset Owner(s)
NVIDIA Corporation
Dataset Creation Date
April 24, 2026
License/Terms of Use
This dataset is released under the NVIDIA Evaluation Data License.
Dataset Characterization
Data Collection Method
Hybrid: Human, Synthetic, Automated. Video data is sourced from vendor-provided footage (GoPro captures of warehouse and smart space environments), synthetic generation (DriveSim collision and multi-camera scenarios), and publicly scraped sources (Dubuque highway/ITS footage).
Labeling Method
Hybrid: Human, Synthetic, Pseudolabeled. Annotations for VQA, dense video captions, and temporal localization are primarily human-authored. Spatial grounding labels (2D/3D bounding boxes, referring expressions) use a combination of human annotation and pseudolabeling pipelines (detection + SAM for spatial pointing). Event verification labels are human-curated. Annotations are held server-side for evaluation only.
Directory Structure
VANTAGE-BENCH/
├── vqa/ # Video question answering
├── dense_captioning/ # Dense video captioning
├── temporal_localization/ # Temporal localization
├── event_verification/ # Event verification
├── 2dbbox/ # 2D object localization
├── referring/ # 2D referring expressions
├── pointing/ # 2D spatial pointing
├── tracking/ # Spatio-temporal tracking
└── README.md # Dataset documentation and submission instructions
Evaluation
Tasks and Submission Formats
| Category | Task | Metric |
|---|---|---|
| Semantic | VQA | Accuracy |
| Semantic | Event Verification | Macro F1 |
| Temporal | Dense Video Captioning | SODA-c |
| Temporal | Temporal Localization | mIoU |
| Spatial | 2D Object Localization | F1@0.5 |
| Spatial | 2D Referring Expressions | mIoU |
| Spatial | 2D Spatial Pointing | Accuracy |
| Spatio-Temporal | Single Object Tracking | AUC |
Submit predictions via the VANTAGE-Bench submission portal and track results on the official leaderboard.
Metric Notes
- Accuracy: Percentage of correct predictions.
- SODA-c: Metric for dense video captioning quality across event coverage and language quality.
- Macro F1: Unweighted mean of per-class F1 scores.
- F1@0.5: F1 score at an IoU threshold of 0.5.
- mIoU: Mean Intersection over Union — average overlap between predicted and ground-truth regions (spatial bounding boxes or temporal segments, depending on the task).
- AUC: Area under the ROC curve, measuring the model's ability to distinguish correct detections or tracks from incorrect ones across varying confidence thresholds.
Evaluation Server
The VANTAGE-Bench GitHub repository provides a sample evaluation pipeline for generating model predictions. Predictions are submitted through the submission portal and scored on the official leaderboard.
Dataset Format
Video (mp4) and Images (jpg).
Dataset Quantification
| Category | Task | Media | Entries |
|---|---|---|---|
| Semantic | VQA | 282 videos | 1,195 QAs (MCQ) |
| Semantic | Event Verification | 163 videos | 163 QAs (BCQ) |
| Temporal | Dense Video Captioning | 104 videos | 717 Events |
| Temporal | Temporal Localization | 203 videos | 1,067 Segments / Spans |
| Spatial | 2D Object Localization | 628 images (3 video sequences) | 27,404 Bboxes |
| Spatial | 2D Referring Expressions | 1,503 images | 3,276 Expressions |
| Spatial | 2D Spatial Pointing | 361 images | 1,005 QAs (MCQ) |
| Spatio-Temporal | Single Object Tracking | 102 video clips | 200 Trajectories |
Total Entries (Annotations): 35,027 Total Media Samples (across tasks, with overlaps): 3,346 Total Data Storage: 42 GB
Potential Known Risks
- Ground truth annotations are not publicly released. All evaluation is performed server-side.
- Some warehouse videos are concatenated clips from longer recording sessions.
Citations
@inproceedings{Fujita2020SODA,
author = {Soichiro Fujita and Tsutomu Hirao and Hidetaka Kamigaito and Manabu Okumura and Masaaki Nagata},
title = {{SODA}: Story Oriented Dense Video Captioning Evaluation Framework},
booktitle = {Proc. ECCV},
year = {2020}
}
@inproceedings{Fu2024BLINK,
author = {Xingyu Fu and Yushi Hu and Bangzheng Li and Yu Feng and Haoyu Wang and Xudong Lin and Dan Roth and Noah A. Smith and Wei-Chiu Ma and Ranjay Krishna},
title = {{BLINK}: Multimodal Large Language Models Can See but Not Perceive},
booktitle = {Proc. ECCV},
year = {2024}
}
@article{Sun2025RefDrone,
author = {Zhichao Sun and Yuda Zou and Xian Sun and Yingchao Feng and Wenhui Diao and Menglong Yan and Kun Fu},
title = {{RefDrone}: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes},
journal = {arXiv preprint arXiv:2502.00392},
year = {2025}
}
References
- HuggingFace dataset: nvidia/PhysicalAI-VANTAGE-Bench
- Project website: vantage-bench.org
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal developer teams to ensure this dataset meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
Changelog
See CHANGELOG.md for release history.
- Downloads last month
- 504