Microsoft has filed a patent for an AI system that can change what people say in live video and redraw their lip movements to match, in near real time. The application, published at the end of October, describes software that picks out words, swaps them for new ones, and generates fresh video frames so the speaker’s mouth looks like it said the replacement all along.
The edits would not be the same for everyone. According to the filing, the system tailors changes to each viewer’s profile, including their age, location and languages. In other words, two people watching the same live broadcast could hear two different versions of it.
How Microsoft’s live speech system would work
The patent was first spotted by patent tracker Patentlyze. Based on its breakdown, the process has three stages. First, the system scans live video and audio. Next, an AI agent decides which words to change for a given viewer. Finally, a generative AI model replaces the audio, smooths over the edit and generates new frames of the speaker’s mouth so the picture matches the sound.
The filing calls this “near-real-time”, which Patentlyze takes to mean a delay of a few seconds. It says the system could run on phones, tablets, cameras and AR glasses.
From cutting “um” to live translation
Some of the proposed uses are harmless. Others are the kind that make people nervous about AI-edited video.
| Proposed use | What it would do |
|---|---|
| Clean-up | Remove filler words such as “um” and “like” |
| Content filter | Replace profanity or offensive language |
| Corrections | Fix slang or mispronounced words, or swap in facts pulled from an internet search |
| Translation | Translate speech a viewer does not understand, with lips redrawn to match |
| Summaries | Condense long recordings into shorter videos with newly generated footage |
Live translation is the most obviously useful idea. Watching a speech or stream in a language you do not speak, with the speaker appearing to talk in yours, would be a genuine accessibility win. Instant fact-checking, by contrast, raises a harder question: who decides what the “correct” version of someone’s words is?
Why the patent is already raising deepfake concerns
The worry is not hard to see. A system built to put new words in someone’s mouth, live and convincingly, overlaps heavily with what deepfakes already do. Futurism, which also covered the filing, noted that the patent mentions storing the original video and audio as “ground truth”, which suggests Microsoft is aware the edited and unedited versions could be confused.
There is also the personalisation problem. If every viewer gets a version of a broadcast shaped by their profile, it becomes harder for people to agree on what was actually said. That matters for news, for politics, and for live entertainment such as esports broadcasts and streams, where casters and players speak off the cuff.
AI and the human voice have already been a flashpoint in games. Fortnite’s AI-voiced Darth Vader drew a complaint from the actors’ union, and SAG-AFTRA’s fight over AI and performers has shaped contracts across the industry. A tool that edits a real person’s live speech goes a step further than generating a voice from scratch.
A patent is not a product
It is worth keeping this in proportion. Big tech companies file thousands of patents, and many never become products. Microsoft has not announced any plans to use this technology, and it did not immediately respond to Futurism’s request for comment. It is also not the first eyebrow-raising filing from the company, which previously patented a system that would pause games to show adverts.
Even so, the filing shows where research is heading. Real-time translation with matching lip movements is clearly on Microsoft’s radar, and if a version of it does ship, the debate over how to label AI-edited live video will arrive with it.
