Audio, voice & music
The sound side of AI video: text-to-speech and voice, music generation, foley, stem separation and word-level timing for captions and cuts. Most of it can run locally. Check each model's licence before commercial use.
Lab posts (38)
- A 3-minute interview video cut entirely by Claude: the transcript-first workflow behind it
Ubon, a business analyst, turned 16 raw interview clips into a 3-minute church Youth Week video with captions, title cards, a logo, music and cleaned-up audio, all without opening an editing app. The thread explaining how passed 1,300 bookmarks in a day. Here's the method in our own words.
- Two open TTS models worth knowing for voiceovers (ZONOS3-Mini and Phonon), plus Comfy's official skills plugin for coding agents
A curator list of open repos for a research-to-video-to-voice pipeline flagged two TTS models we hadn't listed. ZONOS3-Mini is Zyphra's Apache-2.0 multilingual voice-cloning model that runs on a GPU, a Mac or in the browser. Phonon is Gradium's tiny on-device model. Separately, Comfy's official skills repo turns Comfy Cloud into slash commands for Claude Code.
- Markerboard draws whiteboard explainers to your voiceover, plus a Blender face-capture stack, ComfyUI blockout tips and Kevin Ngo's JavaScript drawing prompt
A brand-new open skill turns a topic into a marker-on-whiteboard explainer timed word by word to the narration. Also today: a Claude-scripted Blender face-capture stack, ComfyUI's advice on how detailed a blockout should be, the prompt behind Kevin Ngo's viral JavaScript drawing, and LTX's write-up of its SDR-to-HDR research.
- Mariko Mori's Claude-coded animation installation, ElevenCreative Dub & Sync, and HyperFrames 0.9.36 to 0.9.39
Three smaller items from overnight. A major artist's retrospective now includes a video piece whose animation Claude wrote as code. ElevenLabs folded dubbing and lip sync into one step in ElevenCreative. HyperFrames shipped four releases covering motion-blur previews, template safety and Studio fixes.
- Taan v1-preview scores music to your cuts, Riverside's Tonality edits a single spoken word, and HyperFrames 0.9.40
Two audio launches aimed at the part of the edit people redo most. Taan, a YC startup, says its music model reads your video's cuts and follows timing instructions like 'drums at 10 seconds', with stems and MCP access. Riverside's Tonality changes one word in an existing recording and keeps the voice and room. HyperFrames 0.9.40 stops long 4K exports from filling your disk.
- Fable 5.2's breakout use for video: whole short films written as Python and rendered in Blender
Two days after launch, the Fable 5.2 clips getting the most attention aren't motion-graphics templates. They are complete short films (a fake '90s cereal ad, a show intro, a 13-shot room scene) where the model writes Python that builds, animates and renders everything in Blender, then scores it. Anthropic's own account is featuring them. Here's what the creators say about how they were made.
- ComfyUI v0.45.0 for video makers: RelayVSR video upscaling, H3-Image, Seedream 5.5 and Breeze TTS 2 nodes
ComfyUI tagged v0.45.0 on Nov 10. For video work, the headline is native RelayVSR, a 2x/4x video upscaler that runs a big model only on keyframes and a small decoder on everything in between. The release also adds MiniMax H3-Image, Seedream 5.5 and Breeze TTS 2 nodes, plus the Kling 4.0 Flash, Seedance 2.5 4K and Gemini Omni 1.2 Flash partner-node updates from last week.
- HyperFrames /reverse-any-video: rebuild a finished clip as an editable project (plus free music, SFX and stock)
HeyGen's open-source HTML-to-video framework added a skill that takes an existing video and rebuilds it as an editable HyperFrames project, then diffs the render against the original frame by frame. The reverse step is a Pro feature (about $25/month, per users quoting the pricing); the same refresh made a big signed-in asset library free. Update with npx hyperframes skills update.
- MoneyPrinterTurbo is trending again: topic in, finished short out
The 2024 short-video generator by harry0703 is back on GitHub's daily Trending lists (all languages and Python) on Nov 4, a day after @RoundtableSpace featured it. It's MIT-licensed, sits near 139K stars, and has quietly grown from a stock-footage slideshow maker into a front end for Seedance, MiniMax H3 and a dozen voice engines.
- #OpenSourceHiggsfield closes: five last-hour ComfyUI builds
Entries for ComfyUI's #OpenSourceHiggsfield challenge closed at 9 AM PT on Nov 3, and the hashtag filled with last-minute submissions. Winners are due Nov 10. These five stood out for working video makers: a 3D previs tool, a kinetic-type engine, a MiniMax H3 film studio, a song-sectioned music-video tool and a take-based shot manager.
- Papermorph: a Claude skill that turns books into narrated films
Papermorph is an MIT-licensed agent skill for Claude Code and OpenCode. It turns a textbook PDF into narrated, animated web lessons, a story into a paper-puppet film, or a book into a documentary and a set of vertical shorts. It just passed 1,000 stars, and a curator post on Monday brought it to a wider audience.
- Fish Audio Drama 3 final weights top open TTS arenas
Fish Audio pushed upgraded Drama 3 weights to Hugging Face this morning, and Artificial Analysis now ranks it the top open-weights model on both of its speech arenas. For narration and multi-voice scenes, it's the open TTS to try first.
- ElevenLabs SFX v3 and VEED Dubbing in ComfyUI: two audio upgrades
ElevenLabs shipped a new sound-effects model that handles whole sequences from one prompt, and ComfyUI added a VEED node that translates, voice-clones and lip-syncs a clip. Both cover the audio half of AI video, which still takes the most hand work.
- Mickmumpitz's one-model H3 short film and Singularity V2
Mickmumpitz made an anime short using MiniMax H3 for everything (character sheets, turnarounds, voices, sound effects and the shots) and gave away the ComfyUI workflows. The same day, tutorials landed for Singularity V2, a community H3 fine-tune aimed at action.
- Same script, same footage: a creator rebuilt his talking-head graphics in Remotion and HyperFrames, then retired Remotion
Guney Topcu had Claude Code build motion graphics for one talking-head video twice. Remotion got his months-refined workflow; HyperFrames got the script, the voice track and no revisions. He liked the HyperFrames cut more and says why, and HyperFrames itself moved house today.
- PopTok: a fully local pipeline from one prompt to a lip-synced music video (ACE-Step, LTX-2.5, InfiniteTalk)
Give it a song and lyrics, or only a prompt. Simulated songwriter and director 'workshops' plan the piece, then local models sing it, draw it, animate it as one continuous take and lip-sync the sung lines. You need serious hardware.
- Lock the voice first: a reference-role Seedance prompt that one-shots talking ads, and ElevenLabs' $100k fake-product ad contest (closes Nov 1)
X creator @ZssBecker says he one-shots about 90% of his ads by generating the voice separately (ElevenLabs or a real recording), pasting the exact script into the prompt, and giving every reference one job. His post drew 2,000+ bookmarks. The same week, ElevenLabs is running a $100k contest for 15–45 s ads for fictional brands.
- HeyGen Voice Turbo tops the TTS leaderboard, is free on the API until Nov 30, and now has ComfyUI nodes
HeyGen's faster, cheaper voice mode took #1 on Artificial Analysis' blind-listening TTS arena, with HeyGen's standard voice at #2. The Turbo endpoint costs nothing through Nov 30, and ComfyUI shipped clone and text-to-speech nodes the same day.
- Concat: an open-source, local-first CapCut replacement that Claude Code can drive through a built-in MCP server
A Rust video editor with local Whisper captions, text-to-speech and voice cloning, background removal, keyframes and 4K export, with no account, watermark or upload. The part we care about: `concat --mcp` lets an agent cut a project with the same commands the window uses.
- Google's gemini-api CLI lets your coding agent make video, music, voiceover and SRTs with one command each
gemini-api is an experimental, open-source, zero-dependency binary built for agents. The image, video, tts and music commands write files to disk and print only the path. transcribe outputs SRT. Any agent with a shell (Claude Code, Codex, Cursor) can drive Google's media models without writing SDK scripts.
- Kapwing's MCP lets Claude or Codex caption, translate and voice-clone-dub a video, and the team published three slash-command skills
Add https://www.kapwing.com/mcp as a custom connector in Claude (or a one-liner in Codex), sign in with OAuth, and the agent can upload a clip, burn word-by-word captions, translate them on the original timing, and dub it with each speaker's voice cloned. Kapwing's video ships /captions, /translate and /dubbing skills.
- Don't start an Opus 5.5 film from a blank prompt: the open repos that show their prompts and code
@RoundtableSpace rounded up eight repos where every Opus 5.5 video comes with its prompt and source. Four of them are worth cloning: a 51-style film skill, a 15-style motion kit with a review rubric, a hand-painted cartoon base, and a 4K60 karaoke music video.
- Stop re-explaining your edits: Kyle Balmer's habit of turning each correction into a rule for the agent
Part 3 of Kyle Balmer's series is about the boring step that makes agent editing compound. Sort each fix into 'this video only' or 'every video', save only the second kind to your editing skill, and check the rules on fresh footage.
- Sirio open-sourced his YouTube editing system: a teleprompter that logs your takes, an AI cut you review, and 267 animated boards
Most AI editing setups start with a messy recording. Sirio's Eyeline starts earlier. The teleprompter follows your voice and records which line each take belongs to, so the agent's cut and the motion-graphics boards all key off clean, labelled data.
- huashu-motion: an open skill that writes the launch film, scores it, and lets you point at the frame to fix it
Anthropic's Claude Motion is still a Team and Enterprise beta. Two days after 花叔 open-sourced his own take, it had more than 1,100 stars: a skill that pitches three directions, builds the film as one HTML page, writes a score that real sampled instruments play, and opens an editor where a click on the frame becomes a precise fix.
- BaoCut: an AI video agent where every caption, dub and cut stays editable
Most AI video tools render once and hand you a file. BaoCut makes the timeline the source of truth: transcripts, translations, subtitle layers, voice-overs and code-made motion graphics all live in one editable video that you and your agent edit through the same undo stack.
- No Final Cut at all: Ben Vallack had Claude Code build his edit suite, then directed it with voice notes
Instead of adopting someone else's AI editor, a YouTuber who has edited in Final Cut Pro for years asked Claude Code to build the few tools his own process needs: a transcript cutter, a searchable B-roll library, a phone app for fine-tuning cuts, and a player where he leaves spoken notes on the timeline.
- Voxtral 3 for video people: cheaper captions that know your product names, and narration with emotion tags
Mistral's Voxtral 3 is pitched at voice agents, but two parts matter for editors: transcription you can bias with product and speaker names, and $13-per-million-character TTS with emotion and pace tags.
- Tandem: an MIT video editor with a CLI and MCP, where agent edits stay purple until you've watched them
Mike Cann had Opus 5.5 build him a native Mac editor overnight to replace Filmora. It's now on GitHub under MIT, with a CLI, an MCP server, a Claude skill, and a review gate that shows exactly what the agent changed.
- VoiceStudio: local voice cloning, dubbing and an MCP server for your video agent
A desktop app (AGPL-3.0) for cloning, voice design, timed video dubbing and transcription in 646 languages. It runs on NVIDIA, Apple Silicon or CPU, and serves an MCP server so Claude Code can voice your renders. Check the model licences first.
- earshot gives coding agents ears: hear the soundtrack, score the sync, forge SFX in code
Agents can now render a whole launch film but can't hear it. earshot is a CPU-only CLI, Python API and MCP server that turns sound into evidence: what a file sounds like, where the hits land against the cuts, and a scorecard with timestamped fixes.
- Fish Audio Drama 3 in ComfyUI: directed multi-speaker voice, then H3 talking video
35B open-weight TTS with inline cues ([sighs], [whispers]) and up to 16 speakers, running in ComfyUI from 8 GB VRAM by keeping MoE experts in system RAM. Then hand the take to MiniMax H3 for the picture.
- HyperFrames this week: storyboard boards, an audio command, beats without Chrome
Five releases in about a day, and they all push agents toward the same habits the best creators use: stills before motion, and cuts measured against real audio.
- Kevin Ngo's 'Apple HomeView' spec ad: web pages mapped onto 3D glass
An 82-second fan-made commercial for a product that doesn't exist, where Claude Opus 5.5 orchestrated Blender, HTML screens, Midjourney plates and code-composed music.
- anything2explainer: a topic in, a sourced, narrated explainer out (non-commercial licence)
A Claude Code / Codex skill that researches a topic with sources, writes and voices the narration, then sends parallel agents to draw every shot in Remotion. Check the licence before any client work.
- Two open recipes for AI music videos: Nate McGuire's full credits list and the song-video-director skill
One is a generative-footage pipeline posted credit by credit. The other is a Claude Code skill for music videos rendered in code. Both treat the song as data before any picture gets made.
- claude-motion teaches an agent taste, then makes it check its own frames
Motion rules with actual numbers, a contact-sheet review loop, and a sound engine written in plain Python.
- An open-source audio stack for video: music, foley, lip-sync, timing
Five repos that cover most of the sound side of an AI video pipeline without a subscription.
Skills, repos & workflows in the directory (69)
- OpenMotion Repo
OpenDesign (nexu-io) Apache-2.0 agentic video studio, open-sourced Nov 17 2026 (v0.1.0): give your agent a reference video or social link and it rebuilds the shots as editable HTML with your Brand Kit, then refine in a Studio timeline and render MP4; 300+ MOTION.md shot/timing/transition guides, 150+ brand kits, 22 local agents (Claude Code, Codex, OpenCode, Hermes, Gemini CLI and more), MCP server, ChatGPT plugin, BYOK image/video/music/voice providers; imports HyperFrames projects. Install only from the official repo.
- ZONOS3-Mini Repo
Zyphra's multilingual voice-cloning TTS (sparse MoE, 100M active / 800M total params; Apache-2.0 weights on Hugging Face): one HTTP API across CUDA, Apple Silicon MLX, C++/ggml (no Python) and in-browser WebGPU backends, with a local Voice Studio and a lite cloning page. Clone voices only with consent.
- Phonon (xn-ptts) Repo
Gradium's tiny on-device TTS (31M/40M) with a Rust runtime and no PyTorch: ptts CLI, Python and npm (browser) packages, streaming with fast first audio; EN/FR/DE/ES/PT. Code MIT or Apache-2.0; English 31M weights Apache-2.0, 40M weights CC BY-NC 4.0 (non-commercial).
- Comfy Skills Skill
Comfy-Org's official MIT plugins for coding agents: comfy-cloud bundles the hosted Comfy Cloud MCP server plus skills for image, video, audio and 3D generation, voice cloning, background removal, upscaling, consistent characters and searching models, nodes and templates (Claude Code: /plugin marketplace add Comfy-Org/comfy-skills, install comfy-cloud, sign in via /mcp); comfy-sdk is guidance for apps on the Comfy SDK and Router. Runs on your Comfy Cloud plan.
- FireRed-OpenStoryline Repo
FireRedTeam's Apache-2.0 conversational editing agent: finds and segments footage, drafts storyline and narration, recommends music with beat sync, voiceover and fonts, refines by natural-language edits; save a workflow as a reusable skill for batch edits; ASR rough-cut skill for speech videos; MCP and OpenClaw skills; Hugging Face demo. AI transitions call paid third-party generators.
- SupoClip Repo
FujiwaraChoki's AGPL-3.0 open-source OpusClip alternative: long video (upload, YouTube, Twitch/Kick VOD) → LLM picks 3–7 moments with hook/engagement scores → 9:16 clips with face-centred crop, word-synced captions, hook titles, optional B-roll; editor, MCP server, REST API; Docker self-host (AssemblyAI + LLM key, or local Whisper + Ollama) or hosted.
- Otopilot Repo
wj-sidiss's MIT local topic-to-video pipeline for Shorts and YouTube: script, CC0/museum/Wikimedia images or stock video, voice-over (Edge with OS-voice fallback, optional Azure/Piper/VoxCPM2), subtitles + SRT, thumbnail and publishing pack with image credits; localhost panel or drive it from Claude Code/Codex; no key on the free path. Docs mostly Turkish, UI in 15 languages.
- Motion Canvas Repo
MIT TypeScript library that programs animations with generators, plus an editor with real-time preview and tweakable parameters; built for informative vector animations synced to voice-over. Listed by @RoundtableSpace among open-source repos for agent-made animation alongside remotion-dev/skills and Manim.
- Markerboard Skill
arifbuildsai's MIT agent skill (Claude Code or any command-running agent) for dry-erase whiteboard explainers: asks whether you supply voiceover/timings, writes a beat-numbered script and waits for approval, makes VO (ElevenLabs, MCP voice tool or local) + word timestamps, then renders one continuous board where a marker draws each mark on its spoken word as the camera follows; eight automated checks; Node + Chromium + ffmpeg on macOS/Windows/Linux.
- Cinematic Skill Skill
exploraX's MIT agent skill (Claude Code, Codex, Cursor, Gemini CLI, OpenCode) for narrated documentaries and explainers drawn in Python/Skia: fact-checked one-idea-per-line script, scratch-then-final voice with auto re-timing, Wikimedia Commons images with licence records, one continuous camera over a designed world, word-timed stamps/maps/counters, synthesised SFX, four caption styles + SRT. No video model.
- cangshu koubo video skills Skill
QQTangBa's Apache-2.0 pair of Claude Code skills for Chinese talking-head edits on Apple silicon: cut flubs/retakes/pauses with optical-flow cut smoothing, breathing zoom, keyword captions, 9:16 render with lip-sync drift check; motion version alternates speaker with hand-written Remotion scenes in 3:4. Local mlx-whisper; Chinese docs.
- Taan MCP MCP
Taan (YC) v1-preview video-scoring music model via MCP/API: give it a video, script, reference audio, hum or MIDI and it writes an original score timed to your cuts, then edits single instruments or sections; stems + commercial rights, trained on licensed music per the company. Orchestral/cinematic only for now; paid plans from $15/mo.
- NovelVids Repo
Anning01's MIT self-hostable (Docker) workspace for illustrated dramas and short dramas: import a manuscript (TXT/MD/DOCX/PDF), reuse character/location/voice assets across chapters, edit storyboards on an infinite canvas, generate image/video with the models you configure, assemble episodes (FFmpeg). Also a hosted version; Chinese-first skill pack, English UI supported.
- Rigby2D Repo
Apache-2.0 2D skeletal animation editor (browser or desktop): AutoRig layered PSD/SVG into bones, meshes and IK; Film workspace with camera, audio tracks, viseme lip-sync via on-device Whisper, renders to PNG/WebM-alpha/MP4; built-in MCP server so Claude can rig and animate; runtimes for Godot, Unity, Phaser, PixiJS and more.
- AICG3D Short-Drama Workbench ComfyUI
GPL-3.0 ComfyUI pack (Chinese UI) for long, segmented MiniMax H3 videos and local short dramas: segment parsing, second-pass upscale on the joined cut to avoid seams, exposure lock, per-character voice locking across segments; workflows in the repo. MiniMax H3 licence applies.
- OneShot MV ComfyUI
VisualFrisson's MIT ComfyUI nodes + studio app: a song plus one line of setting becomes a lip-synced music video (beat/section analysis, invented cast via Krea 2, LTX-2.3 singing clips, SyncNet-checked retakes, auto-edit with drum-driven FX). Single Artist mode only with a consenting artist's photo. #OpenSourceHiggsfield honorable mention.
- cinecode Skill
MIT skill for Claude Code or Cursor by sarpstar: one sentence to a finished MP4 with script, Canvas/three.js frames rendered in headless Chrome, Web Audio sound design and a -14 LUFS master. Quick mode (up to ~3 min) and staged Film mode for 10+ min documentaries; no video model.
- Stickman Video Director Skill
MIT agent skill that turns a topic, article or script into a stick-figure explainer with voiceover, captions and SFX, in a dozen visual styles aimed at faceless YouTube channels.
- Scenario Agent Skills Skill
MIT skills pack by Scenario for Claude Code, Cursor, Codex and 70+ agents: images, video (Seedance storyboards, music videos, ads, color grade), audio and 3D through the Scenario MCP, with per-job model choice, a cost quote first and an identity library for consistent characters. Includes '3D OS' expert teams that drive Blender, Maya, ZBrush, Unreal and Unity. Generations use Scenario credits.
- 37Studio Repo
MIT, self-hosted in Docker by Shaitrish: imports brand assets, Claude Code builds and revises HyperFrames videos, faster-whisper transcribes long recordings for clips, and it schedules posts to nine platforms with a shared Meta inbox and drafted replies. Bring your own AI keys and platform developer apps.
- Video Studio Skill Skill
MIT Claude Code skill (/devrel-video) by aaitor: Playwright records real browser flows, HyperFrames composes title, frame, chapters and CTA, FFmpeg masters and QA-checks. Optional narration, music, subtitles. 16:9 only for now.
- Screenplay (agent-screenplay) MCP
MIT MCP server + CLI by gmpetrov: your agent explores a site or desktop app, writes a scenario, and Screenplay films it Screen Studio-style (smooth cursor, auto zooms, callouts, captions, optional voice-over) to MP4. No model key needed.
- MulmoCast CLI Repo
AGPL-3.0 CLI by receptron: an LLM writes a MulmoScript (JSON/YAML beats with speakers, text and images) and MulmoCast renders it to narrated video, podcast, slides, PDF or manga via ffmpeg, with OpenAI, Gemini, ElevenLabs and Replicate options.
- MOTNGRFX ComfyUI
Apache-2.0 ComfyUI node pack + browser app by okkazoo: open-weights kinetic-type motion graphics where Gemma 4 plans, LTX 2.5 shoots with subject mattes so type passes in front of and behind the subject, and ACE-Step scores; 12/16/24 GB graphs.
- Papermorph Skill
MIT Claude Code/OpenCode skill by DozenTwelve that turns textbook PDFs into narrated animated web lessons, stories into paper-puppet films and books into documentaries plus vertical shorts; SVG engine, free Edge TTS or Kokoro narration, optional paid voices and images.
- Claude Animated Shorts Skill
MIT Claude skill by mromidev that turns a topic into a 55-70 s vertical cartoon story short (2D characters in a painted 3D town) rendered with three.js in Remotion, with synthesized music, SFX and frame QA; no video model.
- Massless Repo
MIT browser studio (live at massless.chucksi.com) that designs a shot with camera, lens and film choices, renders a still with open-weight FLUX.1 schnell or SDXL, then flies 28 camera moves (dolly zoom, crash zoom, arcs, bullet time) through it in WebGL with a storyboard, Kokoro voiceover and image-to-3D. No accounts; work stays in your browser.
- video-editor-skills Skill
12 Claude Code skills for a first-pass edit in DaVinci Resolve Studio: cut retakes and dead air, B-roll, voice cleanup, music ducking, captions, Remotion overlays timed to your words, YouTube export. Needs Resolve Studio, ffmpeg and a word-timestamp STT API; raw footage untouched. CC BY-NC 4.0 (no commercial use).
- story-to-handdrawn-video Skill
MIT agent skill: a Chinese story or ordered images in, a 3:4 hand-drawn animated video out via Remotion, with about 300 style recipes and 30 palettes. Output is a silent MP4 (add your own narration and music).
- video-to-game Skill
lovis' agent skill (Claude Code, Codex, Cursor, Gemini) that turns a 10 to 15 s MiniMax H3 Max gameplay clip into a playable single-file Three.js browser game: it studies the clip frame by frame, generates every image, 3D model, sound and music track with fal, writes the game, then plays and films it as proof. The README lists 31 games for $202 in total (about $6 each), with a live gallery. MIT.
- YouTube AI Motion Graphics Repo
Sirio's (TEIN AI) library of 74 HTML/GSAP explainer boards (diagrams, charts, workflows) for YouTube videos, synced to Whisper word-level timings and exported frame-exact to MP4 with Chrome and FFmpeg, plus every prompt from the build and three Claude skills (board brief, board system, board sync).
- Kitty Open Studio MCP
MIT self-hosted AI video studio on your own ComfyUI: 33 curated open-weights workflows as cards, a timeline editor, and MCP so Claude or ChatGPT can direct a music video. Runs on Runpod (pod or pay-per-second serverless) or your own GPU.
- OpenKatana ComfyUI
Apache-2.0 agent-driven editing engine (Studio, ComfyUI apps, CLI and MCP): photos, footage and music in, a finished edit with cuts, transitions, motion type and music out. On the Comfy Registry; standard editing needs no dedicated GPU, per the FAQ.
- OpenFilm Skill
MIT open-source video agent: your coding agent writes a film.html where each shot is a web page drawn by frame(t) (three.js, canvas, SVG, GSAP, video), then you edit by hand in a local browser Studio with a timeline. npx skills add openfilm/openfilm, or desktop app for macOS/Windows; bring your own keys for voice, music and generation.
- OpenCreator (formerly KrillinAI) Repo
Apache-2.0 creator workspace powered by Codex: video download, translation and dubbing (120+ languages per the project), plus agent-driven video, image, voice, avatar and edit tasks; Mac and Windows apps.
- PopTok Repo
Local prompt-or-song → music video pipeline: LLM songwriter and director 'workshops', ACE-Step 1.5 song takes ranked by Whisper, MOSS/Qwen-Image-Edit stills, LTX-2.5 + LTX-2.3 Transition LoRA as one continuous take, cuts snapped to downbeats, InfiniteTalk lip-sync on sung lines. Spreads jobs over DGX Spark, RTX 4090, Strix Halo or M5 Max boxes. Apache-2.0.
- VoxCPM2 Repo
OpenBMB's open 2B tokenizer-free TTS: 30 languages, voice design from a text description, controllable cloning from a short clip, 48 kHz output, real-time streaming (README: RTF ~0.3 on an RTX 4090). Clone only voices you have permission to use. Apache-2.0.
- Kapwing MCP MCP
Official remote MCP (OAuth, any Kapwing plan incl. free, runs on workspace credits) for Claude, Claude Code, Codex and ChatGPT: upload media, grab latest export, burn subtitles, translate subtitles, and AI-dub with per-speaker voice cloning. Kapwing's YouTube walkthrough ships free /captions, /translate and /dubbing skills.
- gemini-api CLI (Google) Repo
Experimental Apache-2.0 zero-dependency CLI for agents: video (Gemini Omni 1.2 Flash GA as of Nov 5; was 1.1), image (Nano Banana 2.1), multi-speaker tts, music (Lyria 3.5), transcribe to SRT, analyze. Media commands write to disk and print only the path; --schema/--dry-run/--jq for agents. v0.4.3 preview.
- HeyGen Voice Turbo (ComfyUI Partner Nodes) ComfyUI
Instant Clone node (free during HeyGen's preview), Text to Speech node with Turbo or Standard, and a HeyGen Voice 1 clone-and-speak template. Turbo ranked #1 on Artificial Analysis' Controlled Voice TTS Arena (Oct 21); HeyGen's Turbo API endpoint is free through Nov 30, 2026 (best effort). Cloud nodes billed via Comfy.
- Concat MCP
Local-first Rust CapCut replacement (macOS/Windows/Linux/Android): local Whisper captions, TTS + voice cloning, background removal, keyframes, 4K60 export, no account or watermark. Built-in MCP server (concat --mcp, window or headless mode) and JSON-RPC API so agents can cut, caption, keyframe and export. Beta; AGPL-3.0.
- Eyeline (YouTube AI Video Editing System) Repo
Sirio's pipeline: voice-following teleprompter that logs takes per line, AI candidate cut (never overwrites plan.json) reviewed in Eyeline Edit with waveforms and word timings, 267 HTML boards (246 animated, window.seek) for Claude Design, narration-aligned renders and a Resolve/Premiere handoff. Placeholder skills; early release. MIT (TEIN branding excluded).
- lemo-opuscar Skill
Lemomo's Claude Code plugin: 51 film styles (style prompt + demo film each), all picture, score and TTS narration made in code; tuned for Opus 5.5. MIT.
- pdoom-video Repo
Code-rendered three.js karaoke music video: Demucs + forced-alignment word timings, beat grid, 1080p60/4K60 with sub-frame motion blur. MIT (song is third-party).
- Retake MCP
Demo-as-code: your agent writes a YAML walkthrough, dry-runs every selector, records a real browser and returns an MP4 with burned-in captions, scene stills and a proof log; optional Edge/Kokoro voiceover. Rerun the file when the UI changes. npm retake-demos. MIT.
- earshot MCP
Ears for coding agents (CPU-only CLI, Python API, MCP): describe/match/find sounds, a listening timeline with beat grid and sound-to-cut sync, an annotated sheet PNG, a launch-film critique with timestamped fixes, and forge/sfx/kit for code-made SFX (201-sound bank). MIT.
- claude-motion-explainer Skill
One sentence to a narrated explainer animation on HyperFrames: research, storyboard, TTS, one sub-agent per scene in parallel, captions, three-colour theme, contact-sheet self-review; includes 26 lessons and a fix_voice.py for mispronounced words. Traditional Chinese. MIT.
- whiteboard-teacher-video Skill
Turns a lecture script into a hand-drawn whiteboard explainer: rough.js strokes drawn in code, text written on as it is spoken, five styles, local Chrome render, edge-tts voice, no image-API cost; re-renders only changed segments. Chinese docs; licence not stated.
- claude-codes-influencers Repo
Talking-mascot pipeline: script → voice (macOS say, ElevenLabs, Voicebox or mic) → Rhubarb 9-mouth-shape lip sync + Whisper word timings → poses, blinks, captions → 1080×1920 MP4. Fork and add your own mascot art. MIT.
- VoiceStudio MCP
Local ElevenLabs-style desktop app: voice cloning, voice design, timed video dubbing, transcription (646 languages) on NVIDIA, Apple Silicon or CPU, with an MCP server at /mcp. App AGPL-3.0; default OmniVoice weights are CC-BY-NC.
- BashCut MCP
Native macOS (AVFoundation) video editor with a docked Claude Code/Codex terminal; everything the UI does is available through the bashcut CLI or MCP as undoable operations. Plugins for Whisper captions and silence markers. MIT.
- papercut-explainer Skill
Nine Agent Skills + a Remotion library for vertical cut-paper explainer reels: paper-cut kit, 24 backdrops, 17 camera moves, 10 transitions, 79 SFX and 8 music beds; strategy → script → storyboard → voice → build → QA. MIT.
- YuE2 Studio MCP
Local song generator on YuE2 (via yue2.cpp): full songs with vocals plus an editable sheet-music score you can re-render. Windows app (6 GB+ NVIDIA), MCP server at 127.0.0.1:8791/mcp for agents. MIT.
- ComfyUI-DBS ComfyUI
Split a soundtrack into music, SFX and speech stems in ComfyUI with the official Dynamic Band Splitting models (tiny, CPU or CUDA). Remove dialogue, pull it out for dubbing, or rebalance a mix. Unofficial integration, MIT.
- ComfyUI-Drama3 ComfyUI
Fish Audio Drama 3 (35B TTS) natively in ComfyUI from 8 GB VRAM via experts-in-RAM: voice cloning, up to 16 speakers, inline cues, Qwen3-ASR, saved voices. Code GPL-3.0; weights under Fish Audio's licence.
- drama3-h3-talking ComfyUI
One ComfyUI workflow: text → Drama 3 voice → MiniMax H3 Ref2VA lip-synced talking video (take anchored at frame 0).
- HyperFrames skills Skill
HTML → deterministic MP4; /hyperframes router picks launch, explainer, PR-to-video, recut or music workflows.
- heditor Skill
Talking head → styled 9:16 Reel on HyperFrames: named styles, word-timed cut, voice-first mix, vision QA. MIT.
- animate Skill
Canvas + Playwright + FFmpeg animation in 7 styles; storyboard gates; Whisper word timing.
- ElevenLabs MCP MCP
Official MCP for TTS, voices and sound effects.
- song-video-director Skill
Claude Code skill for code-rendered music videos (Remotion, p5.js, three.js): BPM/beat grid + Whisper word times, director rules, contact-sheet review. MIT.
- ACE-Step 1.5 Repo
Local song generation, covers, stems, LoRA from a few songs, BPM/key extraction.
- DiffRhythm Repo
Lyrics → full-length song, fast.
- MMAudio Repo
Video-to-audio synced foley (~6 GB VRAM). Check licence for commercial use.
- HunyuanVideo-Foley Repo
Foley generation from video + text.
- Demucs Repo
Stem separation, used for beat detection on drums and clean vocal timing.
- WhisperX Repo
Word-level timestamps + diarization for captions and cut timing.
- faster-whisper Repo
Fast local transcription; used by the animate skill for word timing.
- SlopMonster Skill
Lint scripts for AI tells before voice-over.
Pipelines (6)
- Code-only animated explainer: JS canvas → Playwright → FFmpeg · Based on the creator's video
- HyperFrames product launch video from a URL
- HyperFrames: talking-head footage → captioned, designed recut
- Higgsfield MCP + code motion: two-layer cinematic launch · Based on the creator's video
- Music video: own the track, cut on the beat
- Own-footage character replacement (consent-based) · Based on the creator's video