View a PDF of the paper titled What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation, by Farrukh Rahman
Abstract:A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. (1) is the question even temporal, does it need several frames and their order? And (2) when it does, does the model recover the order from the pixels, or read it off the positional encoding (RoPE)? Most of a temporal score answers neither, since a single frame and answer priors often carry it, and the field’s validity checks, frame-shuffle sensitivity and multi-frame gain, speak only to (1). For (2) we score a reversal conflict, frames reversed while RoPE stays forward, two ways: label-free, as the accuracy lost against the forward labels, and against reversed labels wherever labels transform deterministically under time reversal. This separates a model that follows the reversed content from one merely disrupted. The two readings dissociate two similarly scoring models at both model scales. Qwen3-VL answers the reversed event it actually sees, reading the visual sequence. Molmo2 correctly reports a backwards video when positions agree, and flipping only the RoPE indices flips its prediction to the forward events, i.e. it reads positions. We call these regimes visual-sequence-dominant and position-dominant. Controls show reversed content is processable whenever the channels agree, and that both families share a bias toward describing events as running forward (arrow-of-time prior) that, being similar in strength, cannot by itself account for the dissociation. The distinction matters because the two channels fail on opposite inputs: two models with the same score are not interchangeable.
Submission history
From: Farrukh Rahman [view email]
[v1]
Tue, 14 Jul 2026 03:26:33 UTC (104 KB)
[v2]
Wed, 18 Nov 2026 05:07:02 UTC (115 KB)
[v3]
Thu, 19 Nov 2026 05:14:26 UTC (115 KB)
![[2607.12304] What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation [2607.12304] What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation](https://techwavedispatch.com/wp-content/uploads/2026/11/260712304-What-Does-a-Temporal-Benchmark-Score-Measure-Decomposing-Channel-1024x597.png)