The Deterministic AI Video Pipeline

STAGE 01

Claude "Skills" — Director & Worldbuilder ระบบ AI ผู้กำกับClaude 技能 — 导演与世界构建器

Claude Sonnet 4.6 prompt caching JOEY's "skills"

The creator describes building two custom Claude "Skills" over two weeks. In API terms a Skill is a long system prompt that pins behaviour and output schema. Cache it once and every subsequent character/scene call hits the cache.

Skill A — Banana Pro Director. Owns character physical lock and wardrobe continuity. Emits a character sheet exactly once per actor; every later prompt prepends the sheet's lock_phrase.

# pip install anthropic
from anthropic import Anthropic
import json

client = Anthropic()  # picks up ANTHROPIC_API_KEY

DIRECTOR_SKILL = """You are the Banana Pro Director. For every character
introduced, emit exactly this JSON and nothing else:

{
  "character_id":  "kebab-case-slug",
  "physical": {
    "build": "...", "face_shape": "...", "skin": "...",
    "hair":  {"length": "...", "color": "...", "style": "..."},
    "eyes":  "...", "distinguishing": ["..."]
  },
  "wardrobe": [{
    "scene_tag": "...",
    "outfit":    "fabric-specific description",
    "fabric_notes": "thread count, weave, behaviour under motion",
    "accessories": ["..."]
  }],
  "lock_phrase": "60-word paragraph prepended to every still prompt"
}

Rules: once physical is set it never changes. Wardrobe entries may
be added but never retconned. The lock_phrase is the only string
that ever ships to the image model."""

def lock_character(brief: str) -> dict:
    r = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=2048,
        system=[{
            "type": "text",
            "text": DIRECTOR_SKILL,
            "cache_control": {"type": "ephemeral"},  # cache hit on call 2+
        }],
        messages=[{"role": "user", "content": brief}],
    )
    return json.loads(r.content[0].text)

Skill B — Cinema Worldbuilder. Converts shot descriptions into cinematography language. "Nice sunset shot" is rejected. Output uses real-world gear: lens, aperture, lighting setup, sensor, film stock, atmosphere.

WORLDBUILDER_SKILL = """You are the Cinema Worldbuilder. Convert a shot
description into a single dense prompt. You MUST specify:
  • lens     — focal length + character (35mm anamorphic, 85mm portrait)
  • aperture — depth-of-field intent (f/1.4 isolated, f/8 deep)
  • lighting — named technique (Rembrandt, split, butterfly, chiaroscuro)
  • sensor   — film or digital reference (Alexa LF, 16mm Kodak Vision3)
  • atmosphere — practical haze, anamorphic flare, lens breathing
  • grade    — film-stock reference (Kodak 2383, Fuji 3513)

Forbidden words: cinematic, beautiful, nice, stunning, epic.
Treat the diffusion model as a cinematographer who only speaks gear."""

def shot_prompt(shot: str, characters: list[dict]) -> str:
    locks = " ".join(c["lock_phrase"] for c in characters)
    r = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        system=[{"type": "text", "text": WORLDBUILDER_SKILL,
                 "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user",
                   "content": f"Characters: {locks}\nShot: {shot}"}],
    )
    return r.content[0].text

STAGE 02

Scene-duration planning วางแผนเครดิตเป็นวินาที镜头时长规划

Claude credit budget

Seedance 2.0 charges per clip-second; Higsfield charges per credit. Asking Claude to emit a per-shot duration plan up front prevents the most common money sink: generating 10-second clips that get cut to 2 seconds in the NLE.

def budget(script: str) -> list[dict]:
    r = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=2048,
        system="""Given a script, return JSON list:
            [{"shot_id": "s01", "intent": "...", "seconds": float}]
        Sum of seconds must equal the script's target runtime.
        Round each shot up to the nearest 0.5s. No clip < 2s.""",
        messages=[{"role": "user", "content": script}],
    )
    plan = json.loads(r.content[0].text)
    print(f"Total seconds: {sum(s['seconds'] for s in plan)}")
    return plan

The total seconds × per-second engine rate is your hard budget. If it exceeds what you're willing to spend, the right answer is to cut shots, not shorten them — short clips lose more in pacing than long clips lose in cost.

STAGE 03

Static stills + vision feedback loop วงจรตรวจสอบภาพ静帧 + 视觉反馈回路

Nano Banana Pro Claude vision up to 5 characters

Nano Banana Pro (Gemini 3 Pro Image) is the asset-locking workhorse. Its "Consistency by Design" architecture accepts up to 14 reference images of up to 5 distinct people in a single call — the only consumer-tier image model that holds multi-character identity reliably. Generate the still, then feed it back to Claude with vision input. Claude's correction list rewrites the next prompt before any motion credit is spent.

from google import genai
from google.genai import types
import base64, pathlib

gem = genai.Client()

def render_still(prompt: str, refs: list[bytes], out: str):
    parts = [types.Part.from_bytes(data=r, mime_type="image/png") for r in refs]
    parts.append(prompt + " 16:9 cinematic")
    r = gem.models.generate_content(
        model="gemini-3-pro-image-preview",
        contents=parts,
        config=types.GenerateContentConfig(response_modalities=["IMAGE"]),
    )
    for p in r.candidates[0].content.parts:
        if p.inline_data:
            pathlib.Path(out).write_bytes(p.inline_data.data)

def verify(image_path: str, intent: str) -> list[str]:
    img = base64.standard_b64encode(pathlib.Path(image_path).read_bytes()).decode()
    r = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": img}},
            {"type": "text", "text": f"""Director intent: {intent}

                Compare image to intent. Return JSON only:
                {{ "matches_intent": bool, "drift": ["..."], "fix_prompt": "..." }}
                Look for: wrong outfit, wrong hair, wrong number of subjects,
                wrong lens character, wrong lighting direction."""}
        ]}],
    )
    return json.loads(r.content[0].text)

# Loop until intent matches
for attempt in range(3):
    render_still(prompt, refs, "out/s01.png")
    check = verify("out/s01.png", intent)
    if check["matches_intent"]: break
    prompt = check["fix_prompt"]

STAGE 04

Motion synthesis — two engines, one router เครื่องยนต์สร้างการเคลื่อนไหว动作合成 — 双引擎路由

Seedance 2.0 Higsfield route by constraint

Different shots have different hardest constraints. The pipeline routes each one to whichever engine handles its constraint best, rather than picking a single engine and hoping.

Seedance 2.0 ingests up to 9 images + 3 video clips + 3 audio references in one call. That audio input is the key — for a music video you pass the song stem and the model generates lip-sync directly from the waveform rather than from a separate sync pass.

import replicate

def seedance(still_path: str, motion_prompt: str, audio_path: str | None):
    inputs = {
        "image":    open(still_path, "rb"),
        "prompt":   motion_prompt,
        "duration": 5,
        "fps":      24,
    }
    if audio_path:                          # lip-sync from waveform
        inputs["audio"] = open(audio_path, "rb")
    return replicate.run("bytedance/seedance-2.0", input=inputs)

Higsfield is the cinematic-control layer that aggregates Sora 2, Veo 3, and its own proprietary models behind a single UI. The two features that justify reaching for it: Canvas (mask and regenerate a localised region across moving frames — the only way to fix a single-second hallucination without scrapping the whole clip) and Soul (binds video generation to the Nano Banana Pro reference sheet so faces don't drift). Higsfield is currently GUI-driven for most users; the programmatic path is to render the clean shots through Seedance and route only the broken ones to Higsfield Canvas manually.

STAGE 05

Assembly — CapCut, or scripted ffmpeg ตัดต่อ剪辑合成

CapCut PC ffmpeg (scripted)

The source workflow uses CapCut PC's two-timeline feature for primary footage + B-roll layering, auto-captions, motion tracking, and "MagnatesMedia documentary"–style animated curve lines and 3D map overlays. That's a taste pass — opening CapCut and editing with the timeline is fine here.

If you want the assembly step to stay scripted (CI-friendly, re-runnable), ffmpeg does the structural cut and lets you keep CapCut for the final 30-second polish. The patterns the basic pipeline uses (concat demuxer + drawtext + amix) still apply, plus two additions for the multi-engine case:

# 1. Mixed-source xfade — different engines produce different colour spaces.
#    Force every clip through a normalisation filter BEFORE concat.
for i, clip in enumerate(clips):
    subprocess.run([
        "ffmpeg", "-y", "-i", clip,
        "-vf", "scale=3840:2160:flags=lanczos,format=yuv420p,setsar=1",
        "-c:v", "libx264", "-crf", "18", "-preset", "medium",
        f"out/norm_{i:02d}.mp4",
    ], check=True)

# 2. Lip-sync proof — extract audio and check Seedance's sync drift
#    against the source song. ffmpeg + a simple cross-correlation suffices.
subprocess.run([
    "ffmpeg", "-i", "out/norm_00.mp4",
    "-af", "silencedetect=noise=-30dB:d=0.3", "-f", "null", "-",
])

STAGE 06

Upscale + master เพิ่มความคมและส่งออก升采样与成片

Aiarty VideoProc Real-ESRGAN (free)

Generative video frequently outputs sub-4K with mild compression artefacts. The source workflow uses Aiarty Video Enhancer for AI denoising and upscaling, then VideoProc Converter AI for H.265 encoding. Both are paid GUI tools.

The free programmatic equivalent: ffmpeg with Real-ESRGAN as a CUDA filter, or render the upscale through Replicate's nightmareai/real-esrgan model. Quality is comparable for the 1080p→4K case; the paid tools are faster, not better.

# Free path: Real-ESRGAN via Replicate, then H.265 via ffmpeg
out = replicate.run(
    "nightmareai/real-esrgan",
    input={"image": open("out/final_1080.mp4", "rb"), "scale": 2},
)
pathlib.Path("out/final_4k_raw.mp4").write_bytes(out.read())

subprocess.run([
    "ffmpeg", "-y", "-i", "out/final_4k_raw.mp4",
    "-c:v", "libx265", "-crf", "20", "-preset", "medium",
    "-tag:v", "hvc1",  # Safari/iOS playback
    "-c:a", "aac", "-b:a", "192k",
    "out/final_4k.mp4",
], check=True)