Creative Space

Konten

Provide: Your song file, original lyrics, a lead character (photo or description), and a visual direction.

Receive: An MV with lip-sync singing and narrative shots, plus an audio map, storyboard and editable project folder. Default to a chorus test before the full song; individual shots can later be replaced and reassembled.

Analyze beats, sections and lyric timing locally; storyboard against that map; use Flova for characters, keyframes, lip-sync and narrative footage; align everything to the original song in FFmpeg. Keep lyric units intact, distinguish lip-sync from narrative shots, lock continuity with keyframes, and never retime lip-sync footage. Local timing and assembly must remain inspectable.

Tools

  • Local Python audio analysis: Typically librosa for beats, faster-whisper or whisperx for word timing, optionally demucs for vocals. Either analysis tools or user-supplied timed lyrics are required.
  • FFmpeg, required: Probe audio, cut driving slices, assemble, overlay lyric layers and export.
  • storyboard planning: Plan shots and elements.
  • Flova, required: Generate character views, settings, keyframes, lip-sync singing and narrative footage.
  • Captions (optional): If requested, offer a suitable HyperFrames, Remotion or FFmpeg post-production workflow.

References

Read for steps 3, 6, 7 and 8; read when writing generation requests in steps 4, 5 and 7.

Boundaries

Use the user's own music or music they may use. For a real person's photo, confirm self-use or consent; do not imitate a real celebrity's face or voice. Full songs commonly need 20–30 shots, with lip-sync comparatively costly. Default to a 20–40 second chorus test and expand after approval; make the full song directly when explicitly requested.

Workflow

Inputs/environment → audio map (approval) → elements/storyboard (approval) → element images (approval) → keyframes (approval) → driving slices → shot videos (approval) → assembly → delivery.

1. Inputs and environment

Obtain the actual song file; probe duration, sample rate and channels with ffprobe. For a link, use available media acquisition tools; if unreadable, request the file. Ask for original lyrics (recommended because transcription can be wrong), chorus/full-song scope, aspect ratio, lead photo/description, and visual style. Check FFmpeg, Python audio tools and Flova. If analysis tools are absent, request timed lyrics such as LRC, or explain that listening-based timing estimates are less accurate.

2. Audio map — approval

Write docs/audio-map.md and machine-readable docs/audio-map.json:

  • BPM, meter and downbeats.
  • Intro, verse, pre-chorus, chorus, bridge and outro boundaries, in milliseconds.
  • Each lyric line's start_ms / end_ms. Align transcription to supplied original lyrics, which take precedence. Without them, ask the user to correct the transcription.
  • When vocal separation is available, save audio/vocals.wav for more accurate driving slices.

Show section boundaries and lyric timing for approval before continuing; timing errors propagate through every shot.

3. Elements and storyboard — approval

Use storyboard planning and the timing reference. Define characters per outfit, important props, and settings with layout, orientation and light; descriptions must match provided images. Keep each lyric line or phrase wholly inside one shot, using lyric boundaries and about one second of padding at each end.

Choose lip-sync only when a visible person is singing/speaking and their visible mouth must match the vocal. Instrumentals, dance, acting, wide shots, reactions, or a singer merely appearing during a chorus are narrative/performance footage. Do not alternate narrative and lip-sync mechanically.

Query the selected model's actual driving-audio duration limit. Split long singing passages at pauses between complete phrases. Prefer downbeats and section transitions for narrative cuts. Record shot ID, start/end milliseconds, lyrics, song section, lip-sync/narrative type, setting, elements, framing, angle, camera motion and action. Obtain approval of storyboard and timing.

4. Element images — approval

Choose mode/model through Flova Skill, usually Quick because FFmpeg handles assembly; follow the prompting reference. Use the supplied lead photo when present, otherwise the description. Generate complete front, side and rear character views for each outfit, plus setting/prop sheets. Approved images become references for subsequent shots.

5. Keyframes — approval

Generate one keyframe per shot to lock composition, light and pose. Reference the appearing elements plus the preceding same-scene keyframe. Lip-sync frames must clearly show the face/mouth in a ready-to-sing pose. Show all keyframes and confirm continuity before generating video.

6. Driving-audio slices

With FFmpeg, cut one slice per lip-sync shot into audio/slices/<shot-id>.wav. Prefer isolated vocals; if using the original mix, explain the possible accuracy loss. Include planned padding, respect model limits, and record each slice's source start time for assembly. Do not generate lip-sync without driving audio. If the user explicitly wants an early attempt, explain the synchronization risk and proceed only after acceptance.

7. Shot generation — approval

  • Lip-sync: an audio-driven model, the approved keyframe and that shot's driving slice.
  • Narrative: a multi-reference video model, the keyframe and relevant element sheets.
  • Describe changes/motion rather than repeating static image details. For singing, specify camera movement, emotion, audio-driven performance and body action.
  • Every video request excludes background music and subtitles, avoiding unwanted sound and onscreen text.

Inspect synchronization, action, camera movement and identity. Regenerate only failed shots. Rewrite blocked descriptions; for repeated failure or unavailable entitlement, switch to another model of the same type and tell the user which one. Show all shots for approval before assembly.

8. FFmpeg assembly

Use the original song as the only soundtrack; mute generated clips. Place lip-sync shots using recorded slice start times at 1.0× speed—no stretching or retiming. Fill remaining intervals with narrative footage, trimming or mildly retiming only those shots. Resolve padding overlaps at natural cuts while keeping complete lyric units.

Export MP4 and inspect representative singing shots. If synchronization is visibly off, correct the affected clip's in-point.

9. Delivery

Deliver the MP4 (identify the covered section for a test), audio map, storyboard and project folder containing element images, keyframes, slices, footage and assembly scripts. List unresolved or skipped shots. Preserve replaceable shot-level assembly.

Project layout

mv-<song>/
├── audio/        original song, vocals, slices/
├── docs/         audio-map.md / .json, storyboard
├── elements/     character views, settings, props
├── keyframes/    one per shot
├── shots/        generated shot videos
└── output/       final video and assembly script

Lip-sync selection and timing

Use for storyboard, driving slices, generation and assembly (steps 3/6/7/8).

Selection

Both conditions are required: a visible character is speaking/singing, and their visible mouth must precisely match the corresponding vocal. Record the decision and its reason per shot. Instrumentals, dance, acting, wide stage views, audience reactions and rhythmic movement use narrative/performance generation. Being an MV, containing lyrics, being a chorus/rap section, showing a singer, using a medium/close shot or showing expressive action alone does not establish a need for lip-sync.

Complete lyric units

Never split a lyric line or phrase across two shots. Plan from start_ms / end_ms, adding about one second before the first lyric and after the last. This accommodates fast delivery and short gaps. If padding overlaps adjacent shots, move cuts to natural phrase/musical pauses or adjust boundaries; every shot must still contain its complete lyric unit.

Each storyboard entry includes setting/elements; narrative, singing, dance or performance action; framing/angle/camera movement; song section, lyrics and start_ms / end_ms; lip-sync decision/reason; and its driving slice when applicable.

Driving audio and keyframes

Do not generate lip-sync until storyboard, keyframes and driving audio are ready and approved. Obtain missing slices from vocals or original mix; verify language, content, singing state, timing and matching lyric unit. If the user insists on generation without final driving audio, explicitly warn that missing driving audio reduces synchronization accuracy and obtain acceptance first.

Each keyframe references element sheets plus the most similar earlier frame in the same sequence. Later batch shots reference already generated earlier shots to retain details absent from element sheets: position, mutual orientation and expression. Check composition, positions, relative orientation, expression, setting layout and style.

Inspection and assembly

  • Lip-sync: mouth movement, singing state, audio content and time range agree.
  • Other footage: correct generation route, subject/camera action, setting continuity, music-section fit and negative constraints.
  • If blocked or unsupported, substitute a model of the same class, retaining keyframe and driving audio: lip-sync for lip-sync; multi-reference video for narrative footage.
  • The original song alone defines soundtrack and timeline; generated video is muted. Align every shot by timestamps and lyrics. Lip-sync stays at 1.0× and at its corresponding source-time placement; never shift it merely to fill duration.
  • Fill gaps between lip-sync shots with suitable existing narrative/performance footage, matching musical mood and rhythm rather than stretching singing footage.
  • Inspect aspect ratio, music synchronization, lyric boundaries, lips, shot/color continuity, visual rhythm and unwanted watermarks or model-generated subtitles.

MV generation prompts

Applies to element images, keyframes and shot videos (steps 4/5/7). Use Flova Skill's general request structure, with this MV-specific guidance taking precedence; model parameters remain governed by live capability discovery.

Language

Use Chinese prompts when the user uses Chinese. Dialogue/narration follow the output language. Retain English style anchors, model-specific phrases, templates and examples as written.

Image prompts

Write like a director/colorist briefing a team: how to depict the image matters as much as its contents. Use connected natural language for subject/action/environment and short phrases for style/color/light/composition. Avoid literary motives, inner thoughts, offscreen events and invisible details.

Integrate these six rules into one flowing description, across realistic, anime and 3D styles:

  1. Specific visual anchors: e.g. Neo-Noir style, David Fincher Style, inspired by Se7en, rather than only Neo-Noir.
  2. Cinematic framing: e.g. Over-the-shoulder shot, Dutch angle, Close-up, Medium shot.
  3. Explicit key light and negative fill: e.g. Strong chiaroscuro contrast, deep facial shadows emphasizing facial structure.
  4. Restrained grading, roughly 90% dominated by one tone: e.g. deep teal-cyan shadows dominating 90%, zero warm fill, muted watercolor wash. Avoid red/blue neon contrast unless requested.
  5. Overall rendering/detail: e.g. fine rendering quality, rich and intricate details, soft-focus effect with subtle Gaussian blur.
  6. Emotion/subtext through micro-expressions and a tense instant rather than posed acting: e.g. oppressive suffocation, predatory stillness; describe object atmosphere/state, e.g. cold and detached atmosphere.

Specify important materials where relevant (worn leather with creases, thick knit wool fabric), and quote any exact required text.

Character and element sheets

Prefer full-body/medium-wide/full views; tight half-body or torso crops omit continuity information. Show front, side or three-quarter, and rear with complete head, torso, legs, feet and clothing silhouette. Facial close-ups supplement rather than replace these views.

Describe age, build, distinctive features and voice characteristics. Name outfits separately and reference the approved identity for later looks. Include eyes, jaw, hairstyle/color, makeup, clothing/accessory material, cut, color and condition. Props need shape, material, color, scale, age and distinguishing marks. Settings need important object positions/orientations, especially the performance area, plus mood such as tension, melancholy, hope, warmth, nostalgia or isolation.

Keyframes

Locate the storyboard instant first; if underspecified, mentally simulate subject, camera and environmental motion to resolve it. Describe only the frozen instant:

  • Start: pre-action weight, gaze and body preparation, initial environment and camera composition/angle.
  • End: completed pose/expression, changed environment and camera placement leading into the next shot.
  • Highlight: the strongest visual/emotional instant, with precise pose, expression, light and composition.

For lip-sync, keep face and mouth clearly visible.

Video prompts

The keyframe already defines appearance; describe change and motion. Briefly anchor the starting subject (the elderly man, the woman in sunglasses), then movements, expressions and orientation changes (a character turning from a profile view to face the camera directly). Specify stillness explicitly when intended (the character stands perfectly still, staring forward).

Use simple subject+motion, background+motion and camera+motion clauses. Include degree (quickly, violently, large amplitude, high frequency). Order multiple actions/subjects chronologically: subject1 + motion1 + motion2 or subject1 + motion1, subject2 + motion2.

Describe orbit, aerial, zoom, pan, track/follow or handheld naturally. For a generated cut, write cut to or shot cut and describe the new view. Write non-fixed camera for movement, static camera or fixed shot for a locked shot.

Lip-sync performance

State the audio-driven behavior (singing, speaking, rapping, crying while talking). Use: camera movement + singer emotion + performance state + specific body action + optional background event.

Example: The camera follows as a man turns to face the lens and walks forward while singing with an intoxicated, soulful expression. He first touches his collar with both hands, then spreads his arms wide and tilts his head back, appearing completely immersed. Light and shadow in the background pulse with the rhythm.

Required exclusions

Do not put narration text in video prompts when narration has its own track. Every shot specifies no music and no subtitles; use the original song.

Blocked generations

For a blocked/interrupted request, check these in order and rewrite naturally, rather than adding a separate prohibition line:

  1. Real celebrity/public figure: virtual [appearance description] actor/character.
  2. Copyrighted character/IP (Disney, Marvel, anime, games): retain visual color/clothing/pose features but remove names/logos; use original virtual character + [colors] + [clothing or equipment style] + [action or ability] + no copyright identifiers.
  3. Famous brand name/logo: original [type], [color + shape + material], no brand identifiers.
  4. Gore/violent fight close-ups: tense atmosphere, no gore, no violent close-ups.
  5. Nudity/sexual suggestion: simple clothing, natural movement, no nudity.
  6. Emphasized body curves/parts: balanced figure, natural posture, without naming specific parts.
  7. Horror/ghost/supernatural content: suspenseful mood, no horror elements.

The source workflow does not attempt rewrites for national leaders, political parties/military, sensitive historical events; separatism, extremism, hate/discrimination; drugs, gambling, fraud/illegal finance; fabricated news/official-media impersonation; or child-inappropriate material. Explain the generation limitation instead.

Storyboard and asset handoff

When new shot planning is needed, the host uses this workflow directly; reuse a clear existing shot list or edit brief. Record each S01 segment’s script link, visual/information goal, duration/time, subject/action, camera or element motion, exact text/sound, transition and production method. Link E01 target elements to user material and actual resources; keep REF-E01 source identities separate and mark unavailable assets as pending. Check identity, space, action and sound continuity; retime to actual speech. For marketing, label applicable hook/turn/proof/cta roles and the viewer gain, using grounded demonstrations; teaching and narrative work retain their own structure. For an unclear expression, find a real reference and record what to borrow/replace. No separately named planning skill or file is needed.

Proses kreatif

  1. Python

    Python

    Analyze the song structure

    Mark song sections, beats, and lyric timings.

  2. Flova

    Flova

    Generate music-video shots

    Create performance and story visuals.

  3. FFmpeg

    FFmpeg

    Edit to the music

    Join shots to the original song rhythm and align performances and lyrics.