Text-to-Image & Image-to-Video in Python用 Python 跑文生图与图生视频

★The big idea — ComfyUI is just a wrapper核心思想——ComfyUI 只是一层外壳

ComfyUI looks like magic because it paints a graph of nodes. But under the hood every node is calling Hugging Face diffusers + PyTorch — the same library you can import yourself. A diffusers pipeline is one Python object that bundles every node ComfyUI wires by hand:ComfyUI 看起来很神奇，因为它画出一张节点图。但底层每个节点都在调用 Hugging Face 的 diffusers + PyTorch——也就是你自己可以 import 的那个库。一个 diffusers pipeline 就是一个 Python 对象，把 ComfyUI 手工连接的每个节点都打包在一起：

1Setup — one pip install环境准备——一次 pip 安装

You need Python 3.10+ and these packages. torch CPU wheels are large but install with one command; no CUDA, no drivers required.你需要 Python 3.10+ 和以下这些包。torch 的 CPU 版本体积较大，但一条命令即可安装；无需 CUDA、无需驱动。

# a clean virtual environment keeps this off your system Python
python -m venv .venv
# Windows:  .venv\Scripts\activate    ·    macOS/Linux:  source .venv/bin/activate

pip install diffusers transformers accelerate safetensors
pip install torch                        # CPU build is the default
pip install imageio imageio-ffmpeg pillow  # for saving the video file

# 1 — pull it
docker pull ddtraveller/diffusers-cpu

# 2 — make two host folders: one for outputs, one to cache downloaded models
mkdir -p ~/diffusers/work ~/diffusers/cache

# text -> image (first run downloads sd-turbo into the cache, then it's instant)
docker run --rm \
  -v ~/diffusers/work:/work \
  -v ~/diffusers/cache:/cache \
  ddtraveller/diffusers-cpu \
  t2i --prompt "misty Doi Suthep temple at dawn, lanna style" --out /work/temple.png

# image -> video (SVD is gated — pass your Hugging Face token)
docker run --rm -e HF_TOKEN=hf_xxx \
  -v ~/diffusers/work:/work \
  -v ~/diffusers/cache:/cache \
  ddtraveller/diffusers-cpu \
  i2v --image /work/temple.png --out /work/temple.mp4

↓Download the model — two ways (on a Mac)下载模型——两种方法（以 Mac 为例）

Using stabilityai/sd-turbo as the example: it is ungated — no Hugging Face login, no license to accept — and the download is about 2.5 GB. Both methods below write to the same cache, so your script finds the weights either way. You only need one.以 stabilityai/sd-turbo 为例：它是非受限模型——不需要登录 Hugging Face、不需要接受许可——下载约 2.5 GB。下面两种方法写入的是同一个缓存，所以脚本都能找到权重。你只需选其一。

Method A — let the script download it (nothing extra to run)方法 A——让脚本自动下载（无需额外操作）

The first time from_pretrained() runs, it fetches the weights and caches them; every run after that loads from disk. This is the whole "download step" — there isn't a separate one.第一次执行 from_pretrained() 时会拉取权重并缓存；之后每次运行都从磁盘加载。这就是全部的“下载步骤”——并没有单独的一步。

# run the t2i.py from section 2 — the download happens on this line, once:
#   pipe = AutoPipelineForText2Image.from_pretrained("stabilityai/sd-turbo", ...)
python t2i.py        # first run pulls ~2.5 GB into the cache, then a few seconds per image

Method B — pre-download with the hf CLI方法 B——用 hf 命令行提前下载

Use this to grab the weights ahead of time (offline prep, or to inspect the files). It installs nothing your script doesn't already need.用它可以提前把权重拉下来（离线准备，或想查看文件）。它安装的东西脚本本来也需要。

pip install -U "huggingface_hub[cli]"

# downloads the repo into ~/.cache/huggingface/hub — your script finds it automatically
hf download stabilityai/sd-turbo

# …or put the files in a folder you can see, instead of the hidden cache:
hf download stabilityai/sd-turbo --local-dir ./sd-turbo

# confirm what landed (size + on-disk path):
hf cache scan

hf download is the current command; the older huggingface-cli download still works but is deprecated. No hf auth login is needed for sd-turbo — that step is only for gated models like SVD in section 3.hf download 是现在的命令；旧的 huggingface-cli download 仍可用但已弃用。sd-turbo 不需要 hf auth login——那一步只对第 3 节里 SVD 这类受限模型才需要。

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sd-turbo", torch_dtype=torch.float16, variant="fp16",
).to("mps")

Cache location on macOS: ~/.cache/huggingface/hub. Delete that folder (or run hf cache delete) to free the space. To pull only the smaller fp16 weights via the CLI: hf download stabilityai/sd-turbo --include "*.json" "*.txt" "*fp16*".macOS 上的缓存位置：~/.cache/huggingface/hub。删除该目录（或运行 hf cache delete）即可释放空间。只想用命令行拉取较小的 fp16 权重：hf download stabilityai/sd-turbo --include "*.json" "*.txt" "*fp16*"。

2Text → image文生图

The whole program is four lines of real work. We use SD-Turbo: a distilled model that produces a 512×512 image in 1–4 steps (regular Stable Diffusion needs ~25), so it finishes in seconds even on CPU. Turbo models are trained without classifier-free guidance, so set guidance_scale=0.0.真正干活的代码就四行。我们用 SD-Turbo：一个蒸馏模型，1–4 步就能生成 512×512 的图（普通 Stable Diffusion 需要约 25 步），所以即便在 CPU 上也几秒完成。Turbo 模型训练时没有使用 classifier-free guidance，所以要设 guidance_scale=0.0。

# t2i.py  —  text to image, runs on CPU
import torch
from diffusers import AutoPipelineForText2Image

# pick the best device available; CPU needs float32 (float16 is GPU-only)
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype  = torch.float16 if device == "cuda" else torch.float32

# 1 — Load Checkpoint (downloads + caches on first run)
pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sd-turbo", torch_dtype=dtype,
).to(device)
pipe.enable_attention_slicing()        # less RAM on small machines

# 2 — CLIP Text Encode + KSampler + VAE Decode, all in one call
image = pipe(
    prompt="misty Doi Suthep temple at dawn, golden light, lanna architecture, highly detailed",
    num_inference_steps=4,      # sd-turbo: 1 is fastest, 4 is a touch nicer
    guidance_scale=0.0,         # turbo models: CFG OFF
    generator=torch.Generator(device).manual_seed(42),  # reproducible
).images[0]

# 3 — Save Image
image.save("temple.png")
print("saved temple.png")

python t2i.py        # first run downloads ~2.5 GB, then a few seconds per image on CPU

Swapping models — change one string换模型——改一个字符串

The only thing that changes between models is the repo name in from_pretrained() and the step/guidance settings. The pipeline class auto-detects the architecture.在不同模型之间，唯一要改的就是 from_pretrained() 里的仓库名，以及步数 / guidance 设置。pipeline 类会自动识别模型结构。

3Image → video图生视频

Same shape, different pipeline. Stable Video Diffusion (SVD) takes one still image and predicts a short clip of motion from it. The base model makes 14 frames; ...-xt makes 25. Feed it the temple.png you just generated.套路一样，换个 pipeline。Stable Video Diffusion（SVD）接收一张静态图，并据此预测一小段运动画面。基础模型生成 14 帧；...-xt 生成 25 帧。把你刚生成的 temple.png 喂给它。

# i2v.py  —  image to video
import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype  = torch.float16 if device == "cuda" else torch.float32

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid", torch_dtype=dtype,
).to(device)
pipe.enable_attention_slicing()

# SVD wants a specific size; resize the still first
image = load_image("temple.png").resize((1024, 576))

frames = pipe(
    image,
    num_frames=14,           # base = 14, the -xt model = 25
    decode_chunk_size=2,     # lower = less memory (decode fewer frames at once)
    generator=torch.Generator(device).manual_seed(42),
).frames[0]

export_to_video(frames, "temple.mp4", fps=7)
print("saved temple.mp4")

huggingface-cli login        # paste a token from huggingface.co/settings/tokens

4Reference many .md files & images in one render在一次渲染中引用多个 .md 文件和图片

Real projects don't hard-code the prompt string. A cartoon series keeps its look, its characters, and each shot in separate .md files, and locks the actual appearance in reference .pngs — a "show bible". At render time a script reads them all, concatenates the text into one prompt, and feeds the images into the pipeline. That is the whole trick to keeping frame #800 on-model with frame #1. (See the companion guide Prepare Once, Animate Forever for how the bible is authored.)真实项目不会把提示词写死。一部卡通剧集把画风、角色和每个镜头分别放在不同的 .md 文件里，并用参考 .png 锁定真实外观——一本「设定集」。渲染时，脚本会把它们全部读入、把文本拼成一条提示词、并把图片喂进 pipeline。这就是让第 800 帧与第 1 帧保持一致的全部诀窍。（设定集如何撰写见姊妹篇 准备一次，永远开拍。）

A — Concatenate the text from several .md filesA——把多个 .md 文件的文本拼接起来

Each bible file is plain YAML after its # title line, so one tiny loader reads any of them. The scene names which characters and location to use; the script resolves those names to the locked tokens and joins everything into a single prompt. One extra dependency: pip install pyyaml.每个设定集文件在 # 标题 行之后就是普通 YAML，所以一个极小的加载函数就能读取任意一个。场景文件指明用哪些角色和地点；脚本把这些名字解析成锁定的 token，再把一切拼成一条 prompt。多一个依赖：pip install pyyaml。

# render_scene.py — assemble ONE prompt from many .md files, then render
import yaml, torch
from pathlib import Path
from diffusers import AutoPipelineForText2Image

def fields(path):
    # a .md bible file is YAML once you drop its '# title' line
    body = "".join(l for l in open(path, encoding="utf-8") if not l.startswith("#"))
    return yaml.safe_load(body)

style = fields("style.md")                  # render_tokens, negative, model
chars = {p.stem: fields(p) for p in Path("characters").glob("*.md")}
scene = fields("scenes/scene_01.md")        # cast, action, camera, motion

# resolve every character named in the scene -> their locked look_tokens
looks  = [chars[name]["look_tokens"] for name in scene["cast"]]
prompt = ", ".join(looks + [scene["action"], scene["camera"], style["render_tokens"]])

pipe = AutoPipelineForText2Image.from_pretrained(style["model"], torch_dtype=torch.float32)
image = pipe(
    prompt=prompt,
    negative_prompt=style["negative"],
    num_inference_steps=28, guidance_scale=6.5,   # animagine-xl (a normal SDXL model, not turbo)
).images[0]
Path("renders").mkdir(exist_ok=True)
image.save("renders/scene_01.png")

B — Feed several reference images at once (IP-Adapter)B——一次喂入多张参考图片（IP-Adapter）

Words drift; pictures don't. To pin the actual drawing — the character turnaround, the location backdrop — pass them as pixels. IP-Adapter is the standard way, and it takes more than one image: load a list of adapters, set a list of scales, and pass a list to ip_adapter_image — one slot per adapter. Here adapter 1 holds the character, adapter 2 holds the place.文字会走样，图片不会。要锁定真实画作——角色转面图、场景背景——就把它们作为像素传入。IP-Adapter 是标准做法，而且能接收不止一张图：加载一个列表的适配器、设置一个列表的权重、并把一个列表传给 ip_adapter_image——每个适配器对应一个槽位。这里适配器 1 装角色，适配器 2 装地点。

# reference the actual PICTURES, not just words — several at once
from diffusers.utils import load_image

pipe.load_ip_adapter(
    "h94/IP-Adapter", subfolder="sdxl_models",
    weight_name=[
        "ip-adapter-plus-face_sdxl_vit-h.safetensors",  # adapter 1 -> the character
        "ip-adapter_sdxl_vit-h.safetensors",            # adapter 2 -> the place / style
    ],
)
pipe.set_ip_adapter_scale([0.7, 0.4])    # per-adapter strength (a list, one per adapter)

image = pipe(
    prompt=prompt,
    negative_prompt=style["negative"],
    ip_adapter_image=[                       # a LIST -> one entry per loaded adapter
        load_image(chars["ploy"]["refs"][0]),         # characters/ploy_turnaround.png
        load_image("locations/village_lane_wide.png"),
    ],
    num_inference_steps=30, guidance_scale=6.5,
).images[0]

The refs: list comes straight from characters/ploy.md — the YAML loader already parsed it, so chars["ploy"]["refs"][0] is the turnaround path. A single still you want to lightly redraw instead goes through AutoPipelineForImage2Image with image= + a strength= (0.3 keeps it close, 0.8 reimagines it).refs: 列表直接来自 characters/ploy.md——YAML 加载器已经解析好了，所以 chars["ploy"]["refs"][0] 就是转面图路径。若只想对单张图做轻度重绘，则用 AutoPipelineForImage2Image 配 image= 和 strength=（0.3 贴近原图，0.8 大幅重构）。

C — The same references carry into the videoC——同一组参考延续到视频

Because the still you just rendered already is every reference composed into one frame, animating it needs no new references — the composed PNG becomes the video's first frame.因为你刚渲染的静帧本身就是所有参考合成的一帧，让它动起来无需新的参考——这张合成 PNG 直接成为视频的第一帧。

# the composed still IS the video's reference: feed it straight to SVD as frame 0
from diffusers.utils import load_image, export_to_video
seed   = load_image("renders/scene_01.png").resize((1024, 576))
frames = pipe_i2v(seed, num_frames=14, decode_chunk_size=2).frames[0]
export_to_video(frames, "renders/scene_01.mp4", fps=7)

# SVD takes no text prompt. For a text-driven i2v model (e.g. Wan 2.2), the
# scene's `motion:` line becomes the motion prompt, with this still as frame 0.

Where every source lands in the call每个来源在调用中的落点

5Making it fit & run faster让它跑得下、跑得快

On a 16–24 GB machine, two flags do most of the work. They trade a little speed for a lot less peak memory — the difference between "runs" and "killed: out of memory".在 16–24 GB 内存的机器上，两个开关就能解决大部分问题。它们用一点速度换取大幅降低的峰值内存——这正是“能跑”与“被系统杀掉：内存不足”的区别。

?Recap小结