I wanted to take movie scenes or other video, find the parts that were useful, create an AI song, and make the people in the footage appear to sing it. That's the idea behind Lip-match Studio.
My original plan also included lip reading, meaning figuring out what someone was saying from their mouth movements. What I can describe most concretely is the face-tracking, clip-selection, music, timeline, and audio-driven mouth-redraw workflow. Those are big systems on their own.
As I remember it, the first demo got roughly 1,500 views on YouTube within about a week. That was an early response to one video. It gave me something to point to while the tool was still a work in progress.
Preparing footage for the job
The app starts by analyzing the uploaded video. It makes a playable copy and looks for faces it can follow across frames. The code uses face detection along with a detailed face mesh to find and track the mouth.
That analysis matters because not every frame is equally usable. A small face, a profile, a visual glitch, or text over the mouth can make a clip a bad candidate for redrawing. The app rejects some of those cases while it builds the clip library.
That turns a long video into shorter stretches I can actually use. The footage is still real footage, but the app now knows where suitable faces appear and how long those usable stretches last.
Building a shared audio library
The song is another input with its own prep work. The app accepts an uploaded song or audio track, and its music path can generate a song with Google Lyria. Gemini can help turn a subject into lyrics, or I can write the lyrics myself.
The app separates the voice from the accompaniment. That gives the mouth-redraw step a cleaner vocal to follow, and it keeps the full track for the final soundtrack.
Separating audio as its own stage also makes reuse easier. A song or its isolated vocal doesn't need to be treated as a brand-new object every time I rearrange the clips.
Matching clips to vocal phrases
The React interface has a timeline under the audio waveform. I can drag clips, reorder them, and trim their edges. That matters because a creative result needs more control than picking a folder and pressing render.
The autofill works through the vocal lines and fits clips to their lengths. It handles longer lines first, so they have a better shot at a clip long enough to cover them. The selection rules also prevent reusing the same clip and prevent the same shot from showing up back to back.
This part is closer to editing than to image generation. A face that works technically still has to fit a particular spot in the song.
Redrawing the mouth while keeping the scene
During generation, the system works on the lower part of the face instead of replacing the whole frame. The code uses MuseTalk for the audio-driven redraw, then blends the generated region back into the original footage using the tracked face geometry.
That local approach keeps more of the original shot. The body, the setting, the lighting, and most of the face stay as they were. Only the mouth changes to match the vocal.
Clips with no singing over them skip the redraw. The finished video uses the full audio track, and the renderer puts the processed frames back together into a playable file.
A system around a creative idea
The project combines a FastAPI backend with a React editing interface. Video analysis, tracking, clip storage, audio prep, timeline decisions, and final rendering each have their own role.
That's why I call it an AI media system and not just a model at the end of a pipeline. A mouth-redraw model still needs good input, correct timing, the right audio, and a final assembly step. The app around it decides how much of that work I can manage.
There are limits to what this shows. Mouth tracking and synchronization don't prove that the system can read lips from silent footage. One good demo doesn't prove that any footage will work as well. The original lip-reading idea and the working sync pipeline need to be described separately.
The useful story for me is that I took a playful idea far enough to build an editor, connect several processing stages, and publish a demo people watched. The entertaining output sits on top of a lot of practical engineering.