PopTok: a fully local pipeline from one prompt to a lip-synced music video (ACE-Step, LTX-2.5, InfiniteTalk)
Give it a song and lyrics, or only a prompt. Simulated songwriter and director 'workshops' plan the piece, then local models sing it, draw it, animate it as one continuous take and lip-sync the sung lines. You need serious hardware.
Julian Harris shared PopTok on Oct 25, and LTX featured it the same day in its weekly community roundup. The repo (Apache-2.0) is new and has few stars, but the design is worth studying. There are three ways in: a finished song plus lyrics; lyrics plus a music style, where ACE-Step 1.5 makes several takes and keeps the one Whisper transcribes best; or a single prompt, where three LLM 'songwriters' write the lyrics and style first.
Then a 'directors' workshop' of three simulated music-video directors writes the visual treatment: one universe style that leads every image prompt, a shot for every lyric line and every 6–8 s of instrumental, and where the performer appears or sings. Stills come from MOSS-Image and Qwen-Image-Edit, with an automatic check that re-rolls bad frames. LTX-2.5 with an LTX-2.3 Transition LoRA turns them into one continuous take. Scene changes are then moved onto downbeats, and InfiniteTalk lip-syncs only the sung passages, cut in from the vocal stem. One flag stops the run after the stills, so you can review them before hours of video rendering.
Hardware is the catch. The author says the full pipeline wants a 128 GB DGX-class box. Parts run on an RTX 4090 (analysis, stills, int8 lip-sync) and an M5 Max Mac (LTX video), and the README publishes per-step timings for each machine. The local LLM is Qwen 3.8 27B on llama.cpp.
Why it's on ReelStack: beat-snapped cuts, Whisper-aligned lyrics and lip-sync only where the singer is on camera are the right instincts for music videos, whatever models you use. A lyric_loop tool re-spells words the singer garbles until Whisper hears them correctly. That's a neat trick for any AI-music workflow.
Our take: most people should borrow the planning and beat-alignment steps and use hosted models for the heavy renders. If you do music videos for clients, use your own performer photo with consent.
Related tools on ReelStack
Skills & repos in our directory
- PopTok · Repo, Lip-sync & timing
- Massless · Repo, Generation
- ACE-Step 1.5 · Repo, Audio & music
- DiffRhythm · Repo, Audio & music
- MMAudio · Repo, Audio & music
- HunyuanVideo-Foley · Repo, Audio & music
More from the Lab
- Markerboard draws whiteboard explainers to your voiceover, plus a Blender face-capture stack, ComfyUI blockout tips and Kevin Ngo's JavaScript drawing prompt
- #OpenSourceHiggsfield closes: five last-hour ComfyUI builds
- Three open-source video tools picking up steam: Mago Studio, Rigby2D and a MiniMax H3 'short-drama workbench' for ComfyUI
- Freefield: an open-weights Higgsfield-style studio on ComfyUI (120 apps, Modal or your GPU)
Topics: Audio, voice & music · Open-weight video & audio models