How auto-sync works

subsync sync re-times a subtitle so that its speech activity lines up with a reference: either another subtitle (any language) or the speech detected in an audio track. Both sides are reduced to the same object, a binary speech-present signal sampled every 10 ms, so the alignment code never needs to know which kind of reference it was given. It lives in subsync/align.py.

1. Signal model

Different languages have different words but similar dialogue rhythm: who talks when, the pauses between scenes, the long silences. That rhythm is what we correlate.

2. Normalised cross-correlation

For a window w (zero-mean) and reference r, the score at lag k is

ncc(k) = sum_i w[i] * r[i+k] / sqrt( var_sum(r[k:k+m]) * sum(w^2) )

The numerator is one FFT correlation (numpy.fft.rfft); the denominator uses cumulative sums for the local reference variance, so a stretch of reference that is simply "full of speech" cannot win by volume alone. ncc is in [-1, 1] and is 1.0 for a perfect copy. global_offset() runs this for the whole subtitle against the whole reference and returns the constant offset at 10 ms resolution (this is what --mode offset uses).

3. Scale (framerate drift)

A 25 to 23.976 fps mismatch is a 4.27% stretch: a 60 s window drifts 2.6 s across itself and no longer correlates sharply. So before looking at windows, _scan_scale stretches the whole subtitle by every factor in +-12% (step 0.4 / duration, so the residual error stays under 0.4 s) at 100 ms resolution and keeps the factor with the highest global ncc peak. The factor is only accepted if it beats "no stretch" by 8%.

Worked example (examples/broken_drift.srt, the reference scaled by 23.976/25): the scan picks +4.2597% (true value +4.2709%), windows are then computed on the pre-stretched subtitle, and a final step snaps the total to the nearest well-known ratio (25 to 23.976, 24 to 25, 30 to 29.97, ...) when the leftover disagreement across the file would be below 0.25 s. The snapped value is exact, which removes the last millisecond-level drift.

4. Local offsets per window

The (pre-stretched) subtitle is cut into 60 s windows with a 30 s hop. For each window with 4-96% speech density:

  1. coarse search over the whole reference at 100 ms resolution (max-pooled signals, so a handful of FFTs per window);
  2. refine +-150 ms around the winner at the full 10 ms resolution.

The result is a list of points (t, offset, score): "around subtitle time t, the subtitle has to move by offset". Windows below ncc 0.12 are discarded.

5. Segment model: RANSAC + optimal partitioning

After scale compensation the offsets form a staircase: a constant offset between breakpoints (an ad break removed, a second CD, a recap cut) plus perhaps a small leftover slope.

  1. RANSAC line offset = a + b*t over the points (600 random pairs, inlier tolerance 1 s, score = summed ncc of inliers). The largest consistent segment wins; points from other segments are outliers.
  2. Support filter: a point survives only if a neighbouring window (within 2 positions) agrees within 1.5 s. A lone window is noise (an ambiguous correlation peak). Two windows that agree with each other are kept, so a short last segment survives.

  3. Optimal partitioning (dynamic programming). With residuals r_i = o_i - b*t_i and weights w_i, find the split into runs minimising

    sum over runs of weighted SSE(run) + penalty

best[j] = min over i of best[i] + cost(i, j) + penalty, where cost comes from prefix sums of w, w*r, w*r^2 in O(1). The penalty (0.5 s^2) is the price of a breakpoint: splitting must save more squared error than that, which roughly means a jump of at least 0.5 s with a few windows on each side. Runs need >= 2 windows. 4. The slope b is re-estimated from within-run regressions only, and steps 2-3 are repeated (3 iterations), so the staircase does not leak into the slope. 5. Runs whose offsets differ by less than 0.3 s are merged back together.

Worked example (examples/broken_cut.srt, 90 s removed at 20:00): window offsets are 0.00 for t < 1170 s and +90.00 afterwards; one window straddling the cut is an isolated outlier and is dropped. The DP returns two runs, and each run's offset is the weighted mean of its residuals.

6. Locating the breakpoint

Windows are 60 s wide, so the DP only knows the breakpoint to within about a window. _boundary pins it to a cue gap. Between the last window of run A and the first window of run B, every candidate cue index j (cues before j use offset A, the rest offset B) is scored by summing, per cue, the Jaccard overlap between the mapped cue and the reference speech within 0.5 s around it. Jaccard rather than raw overlap matters: when the reference contains the removed material, a cue can overlap some speech at the wrong offset, but only the right offset reproduces its exact boundaries. Ties are broken towards the widest gap between cues.

7. Final refinement and confidence

Each segment's offset is refined once more by correlating all its cues with the reference at 10 ms over +-0.6 s. confidence is how much better than chance the result is:

agreement  = speech time of mapped cues that lies inside reference speech / total cue time
density    = fraction of the reference that is speech over the same span
confidence = clip((agreement - density) / (1 - density), 0, 1)

An unrelated reference scores below 0.5 even though it may overlap by chance; sync refuses to write below 0.3 unless --force is given. With a jittered other-language reference or a VAD, agreement tops out around 90% because cue boundaries and detected speech never match exactly, so a good sync typically shows 0.7-0.9.

Limitations