1,379 MCQs
Each question is grounded in an independent video clip (5s to 120s), with strict human annotation and verification.
Benchmark Project Page
A Hierarchical Video Counting and Numerical Reasoning Benchmark
Video-based Numerical Reasoning Benchmark
VidNum-1.4K A Comprehensive Benchmark for Video-based Numerical Reasoning
VidNum-1.4K is a strictly human-annotated benchmark with 1,379 video-question pairs across diverse real-world, documentary, and virtual-world videos. It evaluates progressive capability from direct counting to compositional numerical reasoning through Level 1, Level 2, and Level 3 tasks.
About
About the VidNum1.4K
Our Dataset
Three Counting Targets
The benchmark covers Object (647), Action (343), and Event (389) to test broad numerical video reasoning.
Diverse Sources
Videos span real-world, documentary/educational, and virtual-world content with high heterogeneity.
Level 1
Homogeneous Counting
Count a single type of object/action/event under minimal constraints. This tests stable temporal tracking and object permanence.
Level 2
Constrained and Heterogeneous Counting
Count multiple entity types or constrained attributes, often across multi-shot videos requiring robust re-identification.
Level 3
compositional numerical reasoning
Perform temporally grounded arithmetic/comparison operations, moving from perception to multi-step logical deduction.
Demo Questions
Three-Level Benchmark Questions
How many gray-green doors appeared in total in this video?
Answer: D
Reasoning Sketch
How many different dogs appeared in the video?
Answer: C
Reasoning Sketch
A short-haired man in a red jersey shoots twice near the goal. Were there more players in white jerseys near him during the first shot, or more players in yellow jerseys during the second shot?
Answer: A
Reasoning Sketch
Data Construction
Manual Human Annotation Pipeline
Source Video Collection
Videos are curated under a two-tier topic hierarchy with five macro-categories: Knowledge, Life Record, Sports Competition, Artistic Performance, and Film & Television.
Creation and Primary Verification
Group A creates timestamp-grounded numerical questions with a strict visual-description-only policy. Group B independently solves and filters ambiguous or prior-based questions.
Quality Review and Final Audit
Group C refines timestamps and validates reasoning relevance; Group D performs independent final audit. This process finalizes 1,379 verified pairs.
Discussion
Open the Discussion Subpages
Subpage 1
Zero-shot CoT: A Double Edged Sword
Detailed view of when CoT helps and when it hurts across levels and targets.
Subpage 2
Scaling Up + Failure Demo
Joint analysis of scaling behavior and representative failure patterns.
Subpage 3
Evaluation Prompt Protocol
Full NoCoT and ZeroShot-CoT prompts used in benchmark evaluation.
Subpage 4
Use Our Dataset
How to download questions, run evaluation, and access full videos beyond GitHub limits.
Resources
Download Benchmark JSONL
Use the official benchmark JSONL file for reproducible VidNum-1.4K evaluation. Code is available on GitHub.