Repository navigation
[Benchmark] Add TimeLens-Bench for video temporal grounding - #1715
Conversation
Adds Charades-TimeLens, ActivityNet-TimeLens and QVHighlights-TimeLens from TencentARC/TimeLens-Bench. The data is downloaded from Hugging Face, and the prompt and scoring follow the official TimeLens evaluation (R1@0.3/0.5/0.7 and mIoU). Also adds a small shared module for temporal grounding (reading time spans from model answers, temporal IoU, frame timestamps) with unit tests.
|
@aliasgerovs Thanks for adding TimeLens-Bench. I found a few issues that should be addressed before merging:
python -m pytest -q tests/test_dataset_alias.py The timestamp parser and single-span IoU otherwise match the official implementation. multi_span_iou is not used by TimeLens-Bench. |
- Round predicted spans to whole seconds before scoring, as the official eval_dataloader.py does (Python round, so [2.5, 7.5] becomes [2, 8]). - Normalize queries like the official data loader: collapse whitespace and drop trailing periods. - Add the six new presets to the predefined video shortcuts manifest.
TimeLens-Bench scores one span per query, so merge_spans and multi_span_iou are not needed here. MomentSeeker and VUE-TR add them in their own PRs.
|
@ssiq Thanks a lot for the careful review! All three points are fixed in the latest commits.
I also took |
|
Thanks for merging! For the record, here is the full re-run with the cleaned queries and the rounding (Qwen3-VL-8B-Instruct, official settings, all 9,404 answers parsed):
Every score is within 0.7 points of the official numbers. |
2 participants
What this adds
This PR adds TimeLens-Bench, a video temporal grounding benchmark: the model gets a video and a sentence and has to say when the described event happens (start and end time in seconds). The benchmark is a manually cleaned-up version of three widely used grounding datasets:
Data: https://huggingface.co/datasets/TencentARC/TimeLens-Bench (BSD-3-Clause), from the TimeLens paper by TencentARC.
How it works
LMUData/TimeLens-Bench.Charades-TimeLens_2fps,ActivityNet-TimeLens_2fps,QVHighlights-TimeLens_2fps(2 fps is the official setting), plus_1fpsversions.The helpers for reading time spans from an answer, computing temporal IoU and getting frame timestamps are in
vlmeval/dataset/utils/temporal_grounding.py, so other grounding benchmarks can reuse them. They have unit tests intests/test_temporal_grounding.py.Comparison with the official numbers
I ran Qwen3-VL-8B-Instruct with the official TimeLens settings (2 fps, at least 64 visual tokens per frame and at most 14,336 in total, greedy decoding, up to 512 new tokens) and compared with the scores in the TimeLens README:
All 9,404 answers could be parsed. ActivityNet and QVHighlights are within 0.5 points of the official numbers on every metric. The one bigger gap is Charades R1@0.5. Charades labels and answers are mostly whole seconds, so a lot of predictions (6% here) land exactly on IoU 0.5, and tiny differences between machines push them over or under the threshold. The other Charades scores are within 0.3.
I also checked the scoring on its own: on about 47,000 answers from several models, the span parsing and IoU in this PR give exactly the same IoU for every answer as the official
compute_metrics.py.Config used:
{ "model": { "Qwen3-VL-8B-Instruct": { "class": "Qwen3VLChat", "model_path": "Qwen/Qwen3-VL-8B-Instruct", "use_custom_prompt": false, "use_vllm": false, "min_pixels": 65536, "total_pixels": 14680064, "temperature": 0, "max_new_tokens": 512 } }, "data": { "Charades-TimeLens_2fps": {"preset": "Charades-TimeLens_2fps"} } }Checks
pytest tests/test_temporal_grounding.py: 12 passed.