FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

  🌐 Website   |   📑 Paper   |   💻 Code   |   🤗 Dataset  

Quick Start

from datasets import load_dataset

ds = load_dataset("AI4Manufacturing/forge", "task1_three_view", split="train")
print(ds[0].keys())
ds[0]["test_image"]  # PIL Image

Configs

Core Tasks

Config	Cases	Task	Modality
task1_image	451	Wrong model detection (MCQ)	Photo
task1_three_view	496	Wrong model detection (letter)	Three-View
task2_three_view	830	Anomaly classification (normal + defect type)	Three-View
task3_image	857	Extra/wrong part detection (MCQ)	Photo
task3_three_view	309	Extra/wrong part detection (letter)	Three-View
task3_missing_part_image	240	Missing part identification (MCQ)	Photo
task3_missing_part_three_view	137	Missing part identification (MCQ)	Three-View

Grounding Ablation (Single-Image)

Config	Cases	Description
grounding_task_a_zero_shot	500	Coord → Letter, zero-shot
grounding_task_a_icl_within	500	Coord → Letter, ICL (same image)
grounding_task_a_icl_outside	500	Coord → Letter, ICL (cross image)
grounding_task_b_zero_shot	500	Letter → Coord, zero-shot
grounding_task_b_icl_within	500	Letter → Coord, ICL (same image)
grounding_task_b_icl_outside	500	Letter → Coord, ICL (cross image)

Grounding Ablation (Cross-Image)

Config	Cases	Description
grounding_cross_letter_to_letter	513	Match parts by letter across images
grounding_cross_coord_to_coord	513	Match parts by coordinate across images

Total: 6,846 cases across 15 configs

Data Fields

Each row is self-contained with all images embedded. Unused image slots hold a 1x1 placeholder. Use n_normal_refs / n_icl_examples to know how many are real.

Task 1/3 Image -- test_image, grounding_image, assembly_name, assembly_description, error_case, ref_image_0..4, icl_ori_image_0..2, icl_grounding_image_0..2, n_normal_refs, n_icl_examples

Task 1/3 Three-View -- test_image, gt_parts (JSON), query_description, scenario_name, error_case, ref_image_0..4, icl_image_0..2, icl_gt_letters (JSON), n_normal_refs, n_icl_examples

Task 2 Three-View -- test_image, defect_type, is_normal, component_type, component_description, ref_image_0..4, icl_image_0..2, icl_metadata (JSON), n_normal_refs, n_icl_examples

Missing Part -- test_image, assembly_name, assembly_description, choices_text, gt_letter, gt_answer, mcq_mapping (JSON), ref_image_0..4, icl_image_0..2, icl_gt_letters (JSON), n_normal_refs, n_icl_examples

Grounding (single) -- test_image, target_coord (JSON), target_letter, choices (JSON), gt_choice_letter, icl_image_0..2, icl_metadata (JSON), n_icl_examples

Grounding (cross) -- ref_image, test_image, ref_hint, ref_hint_coord (JSON), test_choices (JSON), test_mcq_options (JSON), gt_answer

Corpus evaluation lock — not applicable, checked by byte identity (2026-09-19)

This benchmark is not in the AI4Manufacturing unified training schema (no metadata column; per-config fields), so the corpus's per-record metadata.eval_lock stamp cannot be written here. Checked by byte identity instead: every image cell of every config (34,046 cells across 15 configs, 7,346 rows; 6,573 distinct sha256) was hashed and looked up in the corpus overlap manifest (forge_model/common/overlap_manifest.json, revision fe6e286912b0, generated 2026-09-08) — 0 matches, so no FORGE image is byte-shared with any published AI4Manufacturing repository that carries image_sha256; a re-encoded copy would not be caught by byte identity (pixel identity was not checked). FORGE is an evaluation benchmark and is not a training source. Measured on the live repository at d4cd2907b465 (reports/card_disclosure/b6/forge_measure.json).

Evaluation Code

See the FORGE GitHub repo for the full evaluation toolkit supporting OpenRouter, OpenAI, Anthropic, Google, and vLLM backends.

Citation

@misc{jian2026forge,
      title={FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios}, 
      author={Xiangru Jian and Hao Xu and Wei Pang and Xinjian Zhao and Chengyu Tao and Qixin Zhang and Xikun Zhang and Chao Zhang and Guanzhi Deng and Alex Xue and Juan Du and Tianshu Yu and Garth Tarr and Linqi Song and Qiuzhuang Sun and Dacheng Tao},
      year={2026},
      eprint={2604.07413},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.07413}, 
}