AI dubbing and lip-sync: localise a video without reshooting it

AI dubbing lets you take a finished video into another language without going back on set: the software transcribes, translates, recreates the speaker’s voice and, if you want, reworks the lip movements. This guide covers how it works today, what the main tools actually do as of October 2026, where they still fall short, and a workflow for a version that holds up in front of native speakers.

The job isn’t pressing “translate”. It’s deciding what the video needs to say in the other language, in whose voice and with whose permission. The technology handles the rest, but only if the direction comes first.

How AI dubbing works today

An AI dubbing system does, in sequence, the work of several specialists: it separates the voice from music and effects, transcribes the dialogue while tracking who says what, translates it and generates a new vocal track, leaving the background audio untouched.

The voice

This is where cloning comes in. Announcing Eleven v4 on 28 September 2026, ElevenLabs said an Instant Voice Clone can capture a voice with high fidelity from just 10 seconds of audio. Ten seconds: that’s why consent, further down, is not a formality.

Timing

The translation has to fit the same window as the original. IBM guidelines cited by the W3C estimate that English text over 70 characters, translated into a European language, ends up on average at 130% of its original length. ElevenLabs describes its Dubbing v2 as translation that lines up the start and end of each line with the original.

Lip-sync

The last, optional step reshapes the speaker’s mouth to match the new audio. Specialist providers such as sync. offer “production-grade” models built for close-ups, profile shots and objects in front of the mouth: exactly the shots that separate a convincing result from one you notice.

The tools in October 2026: what they actually do

Two families: platforms that dub inside the social network, and production tools that hand you a file.

YouTube, Instagram and Facebook

In February 2026 YouTube opened auto dubbing to everyone, with 27 languages; Expressive Speech, which carries over the energy of the original voice, works in eight of them, Italian included. Lip-sync is still a pilot for selected channels. But according to the official help page, a video originally in Italian can only be auto-dubbed into English.

Auto dubs can’t be edited, but you can review them before they go live or delete them. If you want control, upload your own tracks with multi-language audio, opened to all creators in September 2025: during the pilot, TechCrunch reported, creators using it got over 25% of their watch time, on average, from views in a non-primary language.

Meta translates Reels for free with Meta AI, mimicking the sound and tone of the creator’s voice, with optional lip-sync. On Instagram it works for public accounts, and on 14 July 2026 Meta announced the expansion to Italian; on Facebook you need at least 1,000 followers. Every translated reel carries the label “Translated with Meta AI”.

Meta advises creators to use face-to-camera videos, no more than two speakers on Facebook and minimal background music, and lets you approve each translation before it goes live.

Production tools

• ElevenLabs Dubbing: according to its documentation, 90+ languages, up to 32 speakers per file, music and effects preserved. The default model, Dubbing v2, is still in alpha, and the dubbed video is described as a high-quality preview, not a production master.
• HeyGen: claims 175+ languages and dialects, voice cloning, lip-sync, a review step for the translated script and a glossary with protected terms and forced pronunciations.
• Higgsfield, one of our partners: its AI Video Translator dubs into 18 languages, Italian included, and re-syncs the lips. It handles one language per run, doesn’t separate speakers, generates a new voice rather than cloning the original and doesn’t export SRT subtitles.

None of them delivers a broadcast-ready master on its own: they give you excellent material to edit, mix and check.

Where AI still gets it wrong

Machine translation is fluent, but fluent doesn’t mean right. Check these every time:

• Idioms and humour. A joke translated literally arrives accurate and dead. It needs rewriting, sometimes replacing altogether.
• Names and brands. Brands, surnames and places get anglicised, or translated when they shouldn’t be.
• Numbers, dates and currencies. “1.000” versus “1,000”, “15/10” versus “10/15”, euros versus pounds: they’re adapted, not translated.
• On-screen text. Lower thirds, graphics, signs and calls to action stay in the original language. The voice changes; the picture doesn’t.
• Speed and overlapping voices. YouTube won’t auto-dub speech that’s too fast, as it would produce a sped-up, unlistenable track. YouTube and Meta both advise against overlapping voices.

Voice and consent: what the rules say

Since 2 August 2026 the transparency obligations in Article 50 of the EU AI Act have applied. A deepfake is defined as AI-generated or manipulated image, audio or video content that resembles existing people and would falsely appear authentic. Whoever deploys it must disclose that it has been generated or manipulated, at the latest at first exposure; for evidently artistic, creative or fictional work, it’s enough to flag it in a way that doesn’t hamper enjoyment of the work.

The Commission’s guidelines, published on 20 July 2026, make clear, according to Paul, Weiss, that no intent to deceive is required, that digital replicas fall within the definition and that audio-only content needs an audible disclaimer. The voluntary Code of Practice published on 10 June 2026 points to an EU icon for labels. Our reading: a real person speaking a language they never spoke, in their cloned voice and with retouched lip movements, should be labelled.

In Italy, Law 132/2025 added Article 612-quater to the Criminal Code: anyone who causes unjust harm by disseminating, without consent, images, videos or voices falsified or altered with AI and capable of misleading people about their authenticity faces one to five years in prison.

Performers have organised too. The 2023 SAG-AFTRA TV/Theatrical contract requires consent to create and use digital replicas, with a reasonably specific description of the intended use; the agreement ratified in June 2026, running until 30 June 2030, expands those protections.

This is general information, not legal advice: for a specific project, have a lawyer check your releases and contracts.

The workflow, step by step

The sequence that turns a translation into a proper language version:

1. Adapt the script with a native speaker. Don’t translate: rewrite lines, wordplay and calls to action for the listener, keeping each line within its original length. Prepare a glossary of names, brands and pronunciations.
2. Cast the voice. Cloning the original voice needs written consent covering languages, channels and duration of use. The alternative is a new voice or an actor.
3. Generate and fix the timing. Work line by line. If a sentence doesn’t fit, rewrite it shorter rather than speeding it up.
4. Use lip-sync only where it’s needed. Frontal close-ups call for it; wide shots, voiceover and shots from behind don’t.
5. Write subtitles from the adapted script, not from an automatic transcription of the dubbed audio.
6. Rebuild graphics and lower thirds in the new language, from the edit project rather than the exported file.
7. Run QA with native reviewers. They watch the whole video, not the script: pronunciation, numbers, sync, tone, AI label. Then the final mix.

One master, several versions

Localisation works better when it’s planned from the start: a script written knowing it will be adapted, graphics kept editable, voice rights agreed in writing before work begins. At Kult, language versions start that way, in pre-production, and are finished in post-production alongside colour, formats and delivery.

We work in Italian and English. If you have a video that needs to speak to another audience, take a look at our adaptation and localisation service or get in touch.

Frequently asked questions

Can AI dubbing replace a professional voice actor?

For tutorials, social content and internal comms it’s often enough. For ads, brand films and drama, performance, adaptation choices and voice direction still matter: use AI to test versions, then choose the final voice once you’ve heard them.

Do I need permission to clone someone else’s voice?

Yes: get written consent covering languages, uses and duration. Some platforms check: for Professional Voice Clones, ElevenLabs asks the voice owner to read out and record a verification text. This is general information, not legal advice.

Can you dub a video with AI for free?

Yes, within limits. Meta AI translations for Reels are free, HeyGen’s free plan lets you translate up to 3 videos a month of up to 3 minutes each, and ElevenLabs watermarks dubs made on its free tier.

Sources

1. Introducing Eleven v4 · ElevenLabs, 2026-09-28
2. Text size in translation · W3C Internationalization, 2007-07
3. ElevenLabs Launches Dubbing v2 API · Speech Technology Magazine, 2026-08-11
4. Lipsync models · sync., 2026-06-01
5. Unlocking a global audience with auto dubbing · YouTube Official Blog, 2026-02-04
6. Use automatic dubbing · YouTube Help, 2026-10
7. YouTube auto dubbing, explained · YouTube Official Blog, 2026-05-15
8. YouTube’s multi-language audio feature for dubbing videos rolls out to all creators · TechCrunch, 2025-09-10
9. Discover Reels From Around the World With Meta AI Translation (updated 14 July 2026) · Meta Newsroom, 2026-07-14
10. Expanding Meta AI Translations for Creators on Facebook and Instagram · Facebook for Creators, 2025-08-19
11. Dubbing (capabilities) · ElevenLabs Documentation, 2026-10
12. Professional Voice Cloning quickstart · ElevenLabs Documentation, 2026-10
13. AI Video Translator: Translate Videos into 175+ Languages · HeyGen, 2026-10
14. AI Video Translator - Translate & Dub Any Video · Higgsfield, 2026-10
15. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 3(60) and 50 · EUR-Lex, Official Journal of the European Union, 2024-06-13
16. Guidelines on transparency obligations for providers and deployers of AI systems · European Commission, 2026-07-20
17. EU Finalises Transparency Rules for AI-Generated Content · Paul, Weiss, 2026-08-04
18. European Commission Publishes Final Code of Practice on Marking and Labelling AI-Generated Content · Jones Day, 2026-06
19. Legge 23 settembre 2025, n. 132 - Disposizioni e deleghe al Governo in materia di intelligenza artificiale (GU n. 223 del 25-9-2025) · Ministero della Giustizia - Procura Generale di Perugia, 2025-09-25
20. Artificial Intelligence: 2023 TV/Theatrical Contracts · SAG-AFTRA, 2023-12
21. SAG-AFTRA Approves New Contract With Stronger AI Protections · Los Angeles Magazine, 2026-06-05