Why AI Dubs Sound Robotic: 65 Syntheses, Measured

The most common complaint about AI dubbing is not that it mistranslates. Search Reddit for it and the phrasing repeats almost verbatim across subreddits: dubbed videos that sound like robots reading. It is a complaint about delivery, not accuracy — and it is rarely accompanied by any account of which part of the pipeline produces it.

A note on why this post exists, since we normally start from search data: it has none. Keyword Planner returns no meaningful volume for any informational query in this space — of 2,929 candidate keywords pulled for the dubbing niche, three were question-shaped and all three were about anime. The searches that do carry volume (video translation, 90,500/mo worldwide; translate video to english, 12,100) return SERPs that are entirely product pages. So this is written for the people asking the question rather than for a keyword, and everything below is measured rather than argued.

We build AI video dubbing at Voqusa, which means we have an instrumented pipeline and an API key. Over one afternoon we ran 65 syntheses through it and measured the results with the same pitch estimator that ships in production. Three findings held up. A fourth did not, and that section is included too.

What "robotic" is actually made of#

"Sounds like a robot" bundles at least four separable failures, and they live in different stages:

1. Wrong voice — a 92 Hz man given a 242 Hz female voice, or two speakers in a conversation given voices 10 Hz apart so the audience cannot tell them apart. A casting failure.
2. Wrong timing — the translated line runs past the moment it belongs to, so the speaker's mouth stops while the audio continues. A scheduling failure.
3. Flat delivery — one narrator voice reading every line at the same pace regardless of what the original speaker was doing. A synthesis failure.
4. Dead room — the original music and ambience stripped out and not replaced, so the dub floats over silence. A mixing failure.

Only the third is really about the text-to-speech model, which is where the conversation usually stops. The other three are engineering, and the fourth is the one most listeners react to first without being able to name it.

Method#

One English line, chosen to be long enough to have prosody and boring enough to have no obvious emotional reading:

The lighthouse keeper wrote the same three words in his logbook every single evening for thirty years.

Synthesized through MiniMax speech-02-hd at speed 1.0, five times per cell, across 13 cells: six voices at neutral pitch, and one male voice across seven pitch settings. Each result was decoded to 16 kHz mono PCM and passed through the autocorrelation F0 estimator in our own src/lib/dub/pitch.ts — 40 ms windows, 20 ms hop, 70–300 Hz search band, median across voiced frames. Separation used demucs v4.1.0 (htdemucs) locally. Every individual run is reported below rather than summarized away.

Finding 1: the same sentence is a different length every time#

This is the one that reorganizes how you build a dubbing pipeline.

Identical text, identical voice, identical speed, sent five times. Across 13 cells and 65 syntheses, the median line came back at 5.50 seconds — and the spread within a cell, between the shortest and longest of five identical requests, had a median of 0.77 s (14%) and a maximum of 1.93 s (35%).

Cell (5 identical requests each)	Median duration	Spread
male-qn-jingying	5.28 s	0.13 s
female-tianmei	5.41 s	0.20 s
female-chengshu	5.84 s	0.29 s
English_expressive_narrator	5.84 s	0.51 s
Wise_Woman	6.35 s	1.45 s
audiobook_male_1	5.50 s	1.93 s

Nothing about the request changed. The model simply does not produce the same performance twice, and on some voices the variation is a third of the line.

The consequence is structural. A dubbing system that decides "this German line is 14% longer than the English, so trim the text by 14%" is planning against a number it cannot know in advance. You have to synthesize first, measure the audio you actually got, and correct afterwards.

That is what our pipeline does, and the thresholds are arithmetic rather than judgement: a line whose synthesized audio overruns its original window by more than 1.15× gets rewritten roughly 20% shorter and re-synthesized once; anything still over is absorbed by playing it faster, capped at 1.3× because past that the speed-up is audible. A line that fails to shorten ships slightly hurried rather than failing the whole dub.

The academic literature arrived at the same place from the other direction. Tam et al. (Interspeech 2022), from AWS AI Labs, formalize this as isochrony-aware translation — the requirement that target speech segments have roughly the same duration as the source segments they replace, and that pauses project correctly across languages. Their finding, that the simplest approach beat the more elaborate ones, matches what the variance above implies: precision in planning is wasted when the synthesizer is this noisy.

Finding 2: the pitch control is close to a constant ratio, until it floors#

To place a synthetic voice on a real speaker's fundamental, you need to know what the synthesizer's pitch parameter does in Hertz. Ours assumes a constant ratio per unit — a shipped constant of 1.0678 — which is a strong assumption for an undocumented control.

It holds. Measured medians for audiobook_male_1, five syntheses per point:

Pitch setting	Measured median	Predicted (127 × 1.0678^p)	Error
−6	97 Hz	85.7 Hz	+11.3
−4	96 Hz	97.7 Hz	−1.7
−2	110 Hz	111.4 Hz	−1.4
0	127 Hz	127.0 Hz	0.0
+2	142 Hz	144.8 Hz	−2.8
+4	160 Hz	165.1 Hz	−5.1
+6	182 Hz	188.3 Hz	−6.3

From −4 to +6 the model is within 6.3 Hz across a 86 Hz span, and the ratio implied by the endpoints is 1.0661 against the shipped 1.0678 — a 0.16% difference. At −6 it breaks: the voice refuses to go lower and lands 11 Hz above prediction, essentially unchanged from −4.

So the useful range is narrower than the control suggests, and a pipeline that trusts −6 to deliver 86 Hz will hand a very deep speaker a voice 11 Hz too high. Ours clamps at ±6 already; the measurement says the real floor is nearer −4.

Finding 3: the male voices are nearly indistinguishable, and that is a casting problem#

Measured base frequency of each voice in the roster, against the value shipped in our own table:

Voice	Shipped	Measured median	Range across 5 runs
male-qn-jingying	127 Hz	130 Hz	125–134
English_expressive_narrator	128 Hz	136 Hz	124–154
audiobook_male_1	130 Hz	120 Hz	110–136
female-chengshu	176 Hz	174 Hz	167–184
Wise_Woman	203 Hz	205 Hz	182–216
female-tianmei	250 Hz	254 Hz	242–258

Five of six land within 10 Hz of the shipped figure. But look at the male column: three voices spanning 16 Hz, and each with a run-to-run range wider than the gap between them. Give two male speakers in an interview two different male voices and the audience hears one narrator. Voice selection alone cannot separate them — the pitch control from Finding 2 has to do the work, which is exactly why the two mechanisms are coupled in the pipeline rather than offered as independent settings.

It is worth putting these against real speakers. The largest normative study of the speaking voice we could find — Berg et al., Journal of Voice 31(2), 2017, a population-based sample of 2,472 adults (1,154 men, 1,318 women) aged 40–79 from the LIFE-Leipzig cohort — reports conversational-voice means of 111.9 Hz for men and 168.5 Hz for women. The synthetic male voices sit 8–24 Hz above that mean; the female voices bracket it from 174 up to 254, well past it.

That study also exposes a design tension in our own code. We classify below 150 Hz as male, above 175 Hz as female, and deliberately refuse to guess in between — a wrong confident answer in the overlap being worse than an admitted one. But 168.5 Hz, the population mean for women in conversation, falls inside that refusal band. The cohort skews older, which lowers it, and our fallback when the register is unclear is safe. Still: the honest reading is that the band was drawn to avoid embarrassing errors, not to maximize correct classifications, and a meaningful share of female speakers land in it.

Finding 4: what we could not reproduce#

Our production code carries a comment recording a development-time observation: a 92 Hz male speaker measured a confident 198 Hz when the estimate was taken from the demucs vocals stem rather than the original mix. It is the stated reason the pipeline measures pitch on the mix, and it is a plausible mechanism — demucs is trained on music, where "vocals" means a sung voice over instruments, and a low speaking fundamental sits in the same band as a bass line. Hybrid Transformer Demucs is trained and evaluated on MUSDB, a music dataset; nothing about it promises to treat a 92 Hz speaking voice as vocal content.

We tried to reproduce it and could not. Three pitch levels, each in two conditions — bare synthesized speech, and the same speech over a synthetic bed of sustained 55/73/82 Hz tones plus pink noise mixed 6 dB under the voice:

Condition	Input	demucs vocals stem
−6, bare voice	98 Hz	98 Hz
−6, with music bed	98 Hz	98 Hz
−2, bare voice	119 Hz	119 Hz
−2, with music bed	119 Hz	119 Hz
+4, bare voice	165 Hz	165 Hz
+4, with music bed	163 Hz	163 Hz

Every case came back within 4 Hz. Three sine tones are evidently not what triggers it; the failure needs real, dense instrumentation, and we did not manufacture a case that produced it. We are reporting the negative result rather than quietly keeping the anecdote, and the pipeline continues to measure on the mix — the cost of that choice is nothing, and the observation that motivated it came from production audio rather than a test rig.

What to listen for in any dubbing tool#

The measurements suggest a short listening test that separates tools faster than a feature table:

• Play a two-speaker clip. If both speakers come back with the same voice, or two voices you cannot tell apart, casting is not measured — it is picked.
• Watch the end of long sentences. If audio continues after the speaker's mouth stops, the tool is planning timing instead of correcting it.
• Listen for the room. If the music and background disappear the moment the dub starts, the voice track was replaced rather than rebuilt. This is the one most listeners register as "fake" without identifying why.
• Dub the same clip twice. Given Finding 1, the two results will differ. How much they differ tells you how much the tool is compensating downstream.

Nothing here requires a subscription to evaluate.

Reproducing this#

Everything above is 200 lines of throwaway TypeScript against a TTS API and a local demucs install. If you want to run it on a different synthesizer, the shape is: pick one sentence, send it five times per condition, decode to 16 kHz mono, take the median autocorrelation F0 per run, and report the spread rather than the mean. The spread is the interesting number and it is the one nobody publishes.

For the translation half of the problem — getting the text right before any of this matters — see our AI audio translation guide, and for the difference between a translated subtitle file and a dubbed voice track, the YouTube transcript guide covers where each one is the right answer.

Key takeaways#

• The same sentence, same voice, same settings, re-synthesized, varies in length by a median of 14% and up to 35%. Dub timing must be measured after synthesis, not predicted before it.
• A synthesizer's pitch control can follow a constant ratio closely (1.0661 measured against 1.0678 assumed) and still floor out at the extreme, where trusting it puts a deep voice 11 Hz too high.
• Three male voices in one roster spanned 16 Hz — less than their own run-to-run variation. Casting two male speakers apart requires pitch shifting, not voice selection.
• Synthetic voices sit above human conversational norms (111.9 Hz male / 168.5 Hz female across 2,472 speakers), and the female population mean falls inside the band where our own classifier refuses to guess.
• A development-time observation about source separation destroying a low fundamental did not reproduce under synthetic conditions. Reported as a negative result.