The reference distance, recovered from the cadence source never carried

FFmpeg's source is a sign, plus or minus one whatever the distance, so
a P-frame after a run of B-frames reported its whole span's
displacement as one frame's --- three to four times over on B-heavy
encodes. The gap back to the previous reference frame is recoverable
from the picture-type cadence while decoding in display order, and
past-referencing vectors now divide by it, in both the data loop and
the grid. Test-first with a forced I B B P cadence: 6.0 before, 2.0
after, on a block moving 2 px a frame. The residuals that remain ---
multi-reference blocks and future-referencing B vectors --- are
documented where the old false claim about source's magnitude stood.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GFjgT8igZuQu4EPMvSqK6u

Author: alexarje

CHANGELOG.md

@@ -8,6 +8,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Changed
+- **Motion-vector displacements are corrected for the reference cadence.** FFmpeg's
+  `source` carries only the sign of a vector's reference, so a P-frame following a run
+  of B-frames reported its whole multi-frame displacement as one frame's — 3× to 4× on
+  typical B-heavy encodes. Past-referencing vectors are now divided by the
+  display-order gap to the previous reference frame, recovered from the picture-type
+  cadence while decoding. This changes every motion-vector number on B-frame encodes:
+  displacements, `magnitude`, the grams and everything built on them. Two residuals
+  remain and are documented: multi-reference blocks reaching older frames, and
+  future-referencing B-frame vectors, which default filtering excludes.
 - **The motion-vector motiongrams now use the classic orientations, named for the
   picture.** The vertical motiongram is the tall image — the frame's width across,
   time running downward — and carries sideways travel; the horizontal motiongram is

musicalgestures/_motionvectors.py

@@ -104,16 +104,20 @@ def mg_motionvectordata(self: "musicalgestures.MgVideo") -> "MgMotionVectorData"
     predicted. Without it a B-frame's vectors point backwards half the time and averaging
     them gives roughly nothing.
 
-    **What that normalisation cannot fix is the reference distance.** `source` records
-    only the direction, plus or minus one, never how many frames away the reference was.
-    An encoder with multiple reference frames will predict some blocks from two or four
-    frames back, and those read as two or four times the per-frame displacement. Measured
-    on a block moving 4 pixels per frame, 53 per cent of vectors came back as 4 and the
-    rest as 8 or 16, with `source` at plus or minus one throughout. So `median_dx` and
-    `median_dy` are medians for a reason and are robust to it; `magnitude` is a sum and is
-    not, and inherits the over-count. Treat `magnitude` as a quantity to correlate against
-    itself over time, which is what it was validated for, rather than as pixels per
-    second.
+    **The reference distance is corrected from the cadence, with two residuals.**
+    `source` records only the direction, plus or minus one, never how many frames away
+    the reference was, so the distance is recovered instead from the picture-type
+    cadence: past-referencing vectors are divided by the display-order gap to the
+    previous reference frame, which makes a P-frame after a run of B-frames read
+    per-frame displacement rather than the whole span's. What the cadence cannot see:
+    blocks reaching an OLDER reference through multi-reference prediction --- measured
+    on a block moving 4 pixels per frame, such blocks read as 8 or 16 with `source` at
+    plus or minus one throughout --- and B-frame vectors pointing at a future
+    reference, which default filtering excludes. So `median_dx` and `median_dy` are
+    medians for a reason and are robust to the residuals; `magnitude` is a sum and
+    inherits what remains of the over-count. Treat `magnitude` as a quantity to
+    correlate against itself over time, which is what it was validated for, rather
+    than as pixels per second.
 
     Returns:
         MgMotionVectorData: `time` in seconds, `picture_type` as 'I'/'P'/'B',
@@ -152,10 +156,27 @@ def mg_motionvectordata(self: "musicalgestures.MgVideo") -> "MgMotionVectorData"
     magnitude: list[float] = []
     med_dx: list[float] = []
     med_dy: list[float] = []
+    #: The reference distance, recovered from the cadence. `source` carries only the
+    #: SIGN of the reference --- measured on a real encode it is plus or minus one
+    #: throughout, whatever the distance --- so a P-frame that follows a run of
+    #: B-frames reports its whole multi-frame displacement as one frame's. What IS
+    #: recoverable while decoding in display order is the gap back to the previous
+    #: reference frame (I or P), and dividing the past-referencing vectors by that gap
+    #: corrects the cadence part exactly. Two residuals remain and are documented
+    #: rather than hidden: blocks reaching an OLDER reference through multi-reference
+    #: prediction, and B-frame vectors pointing at a future reference --- the toolbox
+    #: excludes B-frames by default, and their vectors were already the worse signal.
+    display_idx = -1
+    last_ref_idx = None
     for frame in container.decode(stream):
+        display_idx += 1
+        kind = _PICTURE_TYPES.get(int(frame.pict_type), "?")
+        past_gap = 1.0 if last_ref_idx is None else max(1.0, display_idx - last_ref_idx)
+        if kind in ("I", "P"):
+            last_ref_idx = display_idx
         time.append(float(frame.pts * stream.time_base) if frame.pts is not None
                     else (time[-1] if time else 0.0))
-        kinds.append(_PICTURE_TYPES.get(int(frame.pict_type), "?"))
+        kinds.append(kind)
         vectors = frame.side_data.get("MOTION_VECTORS")
         table = vectors.to_ndarray() if vectors is not None else None
         if table is None or len(table) == 0:
@@ -165,20 +186,20 @@ def mg_motionvectordata(self: "musicalgestures.MgVideo") -> "MgMotionVectorData"
             med_dy.append(0.0)
             continue
         scale = np.maximum(table["motion_scale"].astype(np.float64), 1)
-        #: Two corrections in one division, and both are needed to get a number that
-        #: means "which way did this move, per frame".
-        #:
-        #: ffmpeg defines the vector as `src = dst + motion / motion_scale`, so it
-        #: points from where the block is now back to where it came from: content
-        #: travelling right carries a NEGATIVE motion_x. And `source` is negative when
-        #: the reference is an earlier frame, positive when it is a later one, with its
-        #: magnitude the distance in frames. Dividing by `source` undoes both --- the
-        #: backwards sense of the vector and the backwards sense of a future reference
-        #: cancel --- and scales a multi-frame prediction down to one frame.
+        #: Two corrections in one division: ffmpeg defines the vector as
+        #: `src = dst + motion / motion_scale`, so it points from where the block is
+        #: now back to where it came from --- content travelling right carries a
+        #: NEGATIVE motion_x --- and `source` is negative for a past reference,
+        #: positive for a future one. Dividing by `source` undoes both senses; the
+        #: cadence gap above then scales the past-referencing multi-frame predictions
+        #: down to one frame.
         source = table["source"].astype(np.float64)
         source[source == 0] = -1
         dx = table["motion_x"].astype(np.float64) / scale / source
         dy = table["motion_y"].astype(np.float64) / scale / source
+        past = source < 0
+        dx[past] /= past_gap
+        dy[past] /= past_gap
         area = table["w"].astype(np.float64) * table["h"]
         counts.append(len(table))
         magnitude.append(float((np.hypot(dx, dy) * area).sum()))
@@ -282,15 +303,25 @@ def motion_vector_grid(filename, deterministic=False, threshold=0.0):
     rows = max(1, -(-height // _GRID))
 
     previous_time = 0.0
+    display_idx = -1
+    last_ref_idx = None
     try:
         for frame in container.decode(stream):
+            display_idx += 1
+            kind = _PICTURE_TYPES.get(int(frame.pict_type), "?")
+            #: The cadence gap to the previous reference frame; see motionvectordata
+            #: for why `source` cannot provide the distance itself.
+            past_gap = (1.0 if last_ref_idx is None
+                        else max(1.0, display_idx - last_ref_idx))
+            if kind in ("I", "P"):
+                last_ref_idx = display_idx
             vx = np.zeros((rows, cols), np.float64)
             vy = np.zeros((rows, cols), np.float64)
             w = np.zeros((rows, cols), np.float64)
             t = (float(frame.pts * stream.time_base)
                  if frame.pts is not None else previous_time)
             previous_time = t
-            is_p = _PICTURE_TYPES.get(int(frame.pict_type), "?") == "P"
+            is_p = kind == "P"
             mvs = frame.side_data.get("MOTION_VECTORS")
             table = mvs.to_ndarray() if mvs is not None else None
             if table is not None and len(table):
@@ -299,6 +330,9 @@ def motion_vector_grid(filename, deterministic=False, threshold=0.0):
                 source[source == 0] = -1
                 dx = table["motion_x"].astype(np.float64) / scale / source
                 dy = table["motion_y"].astype(np.float64) / scale / source
+                past = source < 0
+                dx[past] /= past_gap
+                dy[past] /= past_gap
                 area = (table["w"].astype(np.float64) * table["h"])
                 if threshold > 0:
                     keep = np.hypot(dx, dy) >= threshold

tests/_synth.py

@@ -84,7 +84,7 @@ def decaying_tone(t60, sr=SR, f=440.0, dur=None, onset=0.05):
 
 
 def moving_block_video(path, dx=4, dy=0, frames=40, size=(320, 240), block=48,
-                       fps=25, noise=0):
+                       fps=25, noise=0, x264opts=None):
     """An H.264 clip of one textured block translating by an exact (dx, dy) per frame.
 
     Motion vectors are a claim about displacement, so testing them needs footage whose
@@ -128,10 +128,14 @@ def moving_block_video(path, dx=4, dy=0, frames=40, size=(320, 240), block=48,
             frame[y:y + block, x:x + block] = texture
         raw += frame.tobytes()
 
+    #: `x264opts` pins the encoder's structure where a test needs it deterministic ---
+    #: for example "bframes=2:b-adapt=0:ref=1" for a fixed I B B P cadence with single
+    #: reference, so every P-frame's reference is exactly 3 display frames back.
+    extra = ["-x264opts", x264opts] if x264opts else []
     subprocess.run(
         ["ffmpeg", "-y", "-loglevel", "error", "-f", "rawvideo", "-pix_fmt", "gray",
          "-s", f"{W}x{H}", "-r", str(fps), "-i", "pipe:0",
-         "-c:v", "libx264", "-pix_fmt", "yuv420p", "-g", "12", str(path)],
+         "-c:v", "libx264", "-pix_fmt", "yuv420p", "-g", "12", *extra, str(path)],
         input=bytes(raw), check=True)
     return str(path)
 
@@ -154,7 +158,7 @@ def intra_only_video(path, frames=12, size=(160, 120), fps=25):
 
 
 def oscillating_block_video(path, amplitude=40, period=12, frames=96, size=(320, 240),
-                            block=48, fps=25):
+                            block=48, fps=25, x264opts=None):
     """A block that goes back and forth, so its direction cancels but its motion does not.
 
     The counterpart to `moving_block_video`: both leave the same amount of movement in
@@ -176,9 +180,13 @@ def oscillating_block_video(path, amplitude=40, period=12, frames=96, size=(320,
         x = max(0, min(W - block, x))
         frame[y0:y0 + block, x:x + block] = texture
         raw += frame.tobytes()
+    #: `x264opts` pins the encoder's structure where a test needs it deterministic ---
+    #: for example "bframes=2:b-adapt=0:ref=1" for a fixed I B B P cadence with single
+    #: reference, so every P-frame's reference is exactly 3 display frames back.
+    extra = ["-x264opts", x264opts] if x264opts else []
     subprocess.run(
         ["ffmpeg", "-y", "-loglevel", "error", "-f", "rawvideo", "-pix_fmt", "gray",
          "-s", f"{W}x{H}", "-r", str(fps), "-i", "pipe:0",
-         "-c:v", "libx264", "-pix_fmt", "yuv420p", "-g", "12", str(path)],
+         "-c:v", "libx264", "-pix_fmt", "yuv420p", "-g", "12", *extra, str(path)],
         input=bytes(raw), check=True)
     return str(path)

tests/test_motionvectordata.py

@@ -39,25 +39,14 @@ def test_recovers_the_horizontal_displacement_it_was_given(self, moving_right):
         assert np.median(moved) == pytest.approx(4, abs=1.0)
 
     def test_recovers_the_vertical_displacement_it_was_given(self, moving_down):
-        """3 px a frame, within the spread that the reference distance introduces.
-
-        **The tolerance is wide for a reason, and the reason is a real limitation.**
-        `motion_x`/`motion_y` are divided by `source`, and FFmpeg's `source` carries only
-        the SIGN of the reference -- past or future -- not its DISTANCE. A vector
-        referencing a frame two back therefore reports twice the per-frame displacement,
-        and an encode mixing distance-1 and distance-2 references gives a median between
-        the two.
-
-        That is not hypothetical: ffmpeg 6.1.1 emits only distance-1 references here and
-        this reads exactly 3.00, while CI's newer build mixes them and reads 4.5 -- on
-        all nine matrix jobs, so it is the encoder and not flakiness. The test was
-        written against one encoder and had never run on another, because CI did not
-        install PyAV until 1.21.0.
-
-        The narrow assertions worth keeping are elsewhere in this class: sign, and the
-        absence of a component on the unmoved axis. Whether the reader should divide by
-        the reference distance is a question about the measure rather than about this
-        test.
+        """3 px a frame, within the spread the multi-reference residual introduces.
+
+        The cadence part of the reference distance is corrected now --- see
+        Test_reference_distance --- but a block reaching an OLDER reference through
+        multi-reference prediction still over-reports, and which blocks do that is the
+        encoder's choice: ffmpeg 6.1.1 emits only distance-1 references here and reads
+        exactly 3.00, while CI's newer build mixes references. The tolerance covers
+        that residual, not the cadence, which is asserted tightly below.
         """
         mv = musicalgestures.MgVideo(moving_down).motionvectordata()
         moved = mv.median_dy[mv.n_vectors > 0]
@@ -146,3 +135,31 @@ class Test_it_is_cheap:
     def test_reports_how_many_frames_carried_vectors(self, moving_right):
         mv = musicalgestures.MgVideo(moving_right).motionvectordata()
         assert 0 < int((mv.n_vectors > 0).sum()) <= len(mv.time)
+
+
+class Test_reference_distance:
+    """The cadence part of the reference distance is corrected, and says so in numbers.
+
+    FFmpeg's `source` carries only the sign of the reference, so a P-frame that follows
+    a run of B-frames reports its whole multi-frame displacement as if it were one
+    frame's. The distance to the previous REFERENCE frame is recoverable from the
+    picture-type cadence while decoding, and that is the correction under test: with a
+    forced I B B P cadence and single reference, every P-frame's vectors span exactly
+    3 display frames, and a block moving 2 px a frame must read 2, not 6.
+
+    What this cannot correct, and is not tested: blocks reaching an older reference
+    through multi-reference prediction (suppressed here with ref=1), and B-frame
+    vectors that point at a future reference, which the toolbox excludes by default.
+    """
+
+    @pytest.fixture(scope="class")
+    def cadenced(self, tmp_path_factory):
+        return moving_block_video(tmp_path_factory.mktemp("mv") / "cadence.mp4",
+                                  dx=2, dy=0, frames=60,
+                                  x264opts="bframes=2:b-adapt=0:ref=1")
+
+    def test_a_p_frame_after_b_frames_reports_per_frame_displacement(self, cadenced):
+        mv = musicalgestures.MgVideo(cadenced).motionvectordata()
+        p = (mv.picture_type == "P") & (mv.n_vectors > 0) & (mv.median_dx != 0)
+        assert p.sum() > 5
+        assert np.median(mv.median_dx[p]) == pytest.approx(2, abs=0.75)