Text-to-Image & Image-to-Video in Python用 Python 跑文生图与图生视频

★The big idea — ComfyUI is just a wrapper核心思想——ComfyUI 只是一层外壳

ComfyUI looks like magic because it paints a graph of nodes. But under the hood every node is calling Hugging Face diffusers + PyTorch — the same library you can import yourself. A diffusers pipeline is one Python object that bundles every node ComfyUI wires by hand:ComfyUI 看起来很神奇，因为它画出一张节点图。但底层每个节点都在调用 Hugging Face 的 diffusers + PyTorch——也就是你自己可以 import 的那个库。一个 diffusers pipeline 就是一个 Python 对象，把 ComfyUI 手工连接的每个节点都打包在一起：

1Setup — one pip install环境准备——一次 pip 安装

You need Python 3.10+ and these packages. torch CPU wheels are large but install with one command; no CUDA, no drivers required.你需要 Python 3.10+ 和以下这些包。torch 的 CPU 版本体积较大，但一条命令即可安装；无需 CUDA、无需驱动。

# a clean virtual environment keeps this off your system Python
python -m venv .venv
# Windows:  .venv\Scripts\activate    ·    macOS/Linux:  source .venv/bin/activate

pip install diffusers transformers accelerate safetensors
pip install torch                        # CPU build is the default
pip install imageio imageio-ffmpeg pillow  # for saving the video file

# 1 — pull it
docker pull ddtraveller/diffusers-cpu

# 2 — make two host folders: one for outputs, one to cache downloaded models
mkdir -p ~/diffusers/work ~/diffusers/cache

# text -> image (first run downloads sd-turbo into the cache, then it's instant)
docker run --rm \
  -v ~/diffusers/work:/work \
  -v ~/diffusers/cache:/cache \
  ddtraveller/diffusers-cpu \
  t2i --prompt "misty Doi Suthep temple at dawn, lanna style" --out /work/temple.png

# image -> video (SVD is gated — pass your Hugging Face token)
docker run --rm -e HF_TOKEN=hf_xxx \
  -v ~/diffusers/work:/work \
  -v ~/diffusers/cache:/cache \
  ddtraveller/diffusers-cpu \
  i2v --image /work/temple.png --out /work/temple.mp4

↓Download the model — two ways (on a Mac)下载模型——两种方法（以 Mac 为例）

Using stabilityai/sd-turbo as the example: it is ungated — no Hugging Face login, no license to accept — and the download is about 2.5 GB. Both methods below write to the same cache, so your script finds the weights either way. You only need one.以 stabilityai/sd-turbo 为例：它是非受限模型——不需要登录 Hugging Face、不需要接受许可——下载约 2.5 GB。下面两种方法写入的是同一个缓存，所以脚本都能找到权重。你只需选其一。

Method A — let the script download it (nothing extra to run)方法 A——让脚本自动下载（无需额外操作）

The first time from_pretrained() runs, it fetches the weights and caches them; every run after that loads from disk. This is the whole "download step" — there isn't a separate one.第一次执行 from_pretrained() 时会拉取权重并缓存；之后每次运行都从磁盘加载。这就是全部的“下载步骤”——并没有单独的一步。

# run the t2i.py from section 2 — the download happens on this line, once:
#   pipe = AutoPipelineForText2Image.from_pretrained("stabilityai/sd-turbo", ...)
python t2i.py        # first run pulls ~2.5 GB into the cache, then a few seconds per image

Method B — pre-download with the hf CLI方法 B——用 hf 命令行提前下载

Use this to grab the weights ahead of time (offline prep, or to inspect the files). It installs nothing your script doesn't already need.用它可以提前把权重拉下来（离线准备，或想查看文件）。它安装的东西脚本本来也需要。

pip install -U "huggingface_hub[cli]"

# downloads the repo into ~/.cache/huggingface/hub — your script finds it automatically
hf download stabilityai/sd-turbo

# …or put the files in a folder you can see, instead of the hidden cache:
hf download stabilityai/sd-turbo --local-dir ./sd-turbo

# confirm what landed (size + on-disk path):
hf cache scan

hf download is the current command; the older huggingface-cli download still works but is deprecated. No hf auth login is needed for sd-turbo — that step is only for gated models like SVD in section 3.hf download 是现在的命令；旧的 huggingface-cli download 仍可用但已弃用。sd-turbo 不需要 hf auth login——那一步只对第 3 节里 SVD 这类受限模型才需要。

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sd-turbo", torch_dtype=torch.float16, variant="fp16",
).to("mps")

Cache location on macOS: ~/.cache/huggingface/hub. Delete that folder (or run hf cache delete) to free the space. To pull only the smaller fp16 weights via the CLI: hf download stabilityai/sd-turbo --include "*.json" "*.txt" "*fp16*".macOS 上的缓存位置：~/.cache/huggingface/hub。删除该目录（或运行 hf cache delete）即可释放空间。只想用命令行拉取较小的 fp16 权重：hf download stabilityai/sd-turbo --include "*.json" "*.txt" "*fp16*"。

2Text → image文生图

The whole program is four lines of real work. We use SD-Turbo: a distilled model that produces a 512×512 image in 1–4 steps (regular Stable Diffusion needs ~25), so it finishes in seconds even on CPU. Turbo models are trained without classifier-free guidance, so set guidance_scale=0.0.真正干活的代码就四行。我们用 SD-Turbo：一个蒸馏模型，1–4 步就能生成 512×512 的图（普通 Stable Diffusion 需要约 25 步），所以即便在 CPU 上也几秒完成。Turbo 模型训练时没有使用 classifier-free guidance，所以要设 guidance_scale=0.0。

# t2i.py  —  text to image, runs on CPU
import torch
from diffusers import AutoPipelineForText2Image

# pick the best device available; CPU needs float32 (float16 is GPU-only)
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype  = torch.float16 if device == "cuda" else torch.float32

# 1 — Load Checkpoint (downloads + caches on first run)
pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sd-turbo", torch_dtype=dtype,
).to(device)
pipe.enable_attention_slicing()        # less RAM on small machines

# 2 — CLIP Text Encode + KSampler + VAE Decode, all in one call
image = pipe(
    prompt="misty Doi Suthep temple at dawn, golden light, lanna architecture, highly detailed",
    num_inference_steps=4,      # sd-turbo: 1 is fastest, 4 is a touch nicer
    guidance_scale=0.0,         # turbo models: CFG OFF
    generator=torch.Generator(device).manual_seed(42),  # reproducible
).images[0]

# 3 — Save Image
image.save("temple.png")
print("saved temple.png")

python t2i.py        # first run downloads ~2.5 GB, then a few seconds per image on CPU

Swapping models — change one string换模型——改一个字符串

The only thing that changes between models is the repo name in from_pretrained() and the step/guidance settings. The pipeline class auto-detects the architecture.在不同模型之间，唯一要改的就是 from_pretrained() 里的仓库名，以及步数 / guidance 设置。pipeline 类会自动识别模型结构。

Batch from a JSON file — prompt & filename as data从 JSON 文件批量生成——把提示词与文件名当数据

Hard-coding one prompt is fine for a test, but the real win of a script is running many unattended. Move the two things that change — the description and the output filename — out of the code and into a small prompts.json. Now anyone can add images by editing data, never the Python.写死一个提示词用来测试没问题，但脚本真正的价值是无人值守地批量运行很多个。把会变的两样东西——描述文本和输出文件名——从代码里挪到一个小小的 prompts.json 中。这样谁都能通过改数据来添加图片，而不用碰 Python。

// prompts.json  —  a list of { description, filename } jobs
[
  { "prompt": "misty Doi Suthep temple at dawn, lanna architecture, highly detailed", "out": "temple.png" },
  { "prompt": "lantern festival on the Ping River at dusk, warm light, reflections",   "out": "lanterns.png" },
  { "prompt": "a bowl of khao soi on a wooden table, steam, soft window light",        "out": "khaosoi.png" }
]

Then change t2i.py to load that file and loop. Build the pipeline once (the slow part), then reuse it for every job:然后把 t2i.py 改成读取该文件并循环。pipeline 只构建一次（最慢的一步），之后每个任务都复用它：

# t2i.py  —  read prompts + filenames from prompts.json, generate each
import json, torch
from diffusers import AutoPipelineForText2Image

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype  = torch.float16 if device == "cuda" else torch.float32

# build the pipeline ONCE, outside the loop (loading ~2.5 GB is the slow step)
pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sd-turbo", torch_dtype=dtype,
).to(device)
pipe.enable_attention_slicing()

# load the job list — encoding="utf-8" so Thai / Chinese prompts survive
with open("prompts.json", encoding="utf-8") as f:
    jobs = json.load(f)

for job in jobs:
    image = pipe(
        prompt=job["prompt"],            # description ← from JSON
        num_inference_steps=4,
        guidance_scale=0.0,
        generator=torch.Generator(device).manual_seed(42),
    ).images[0]
    image.save(job["out"])              # filename ← from JSON
    print("saved", job["out"])

python t2i.py        # generates temple.png, lanterns.png, khaosoi.png in one run

3Image → video图生视频

Same shape, different pipeline. Stable Video Diffusion (SVD) takes one still image and predicts a short clip of motion from it. The base model makes 14 frames; ...-xt makes 25. Feed it the temple.png you just generated.套路一样，换个 pipeline。Stable Video Diffusion（SVD）接收一张静态图，并据此预测一小段运动画面。基础模型生成 14 帧；...-xt 生成 25 帧。把你刚生成的 temple.png 喂给它。

# i2v.py  —  image to video
import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype  = torch.float16 if device == "cuda" else torch.float32

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid", torch_dtype=dtype,
).to(device)
pipe.enable_attention_slicing()

# SVD wants a specific size; resize the still first
image = load_image("temple.png").resize((1024, 576))

frames = pipe(
    image,
    num_frames=14,           # base = 14, the -xt model = 25
    decode_chunk_size=2,     # lower = less memory (decode fewer frames at once)
    generator=torch.Generator(device).manual_seed(42),
).frames[0]

export_to_video(frames, "temple.mp4", fps=7)
print("saved temple.mp4")

huggingface-cli login        # paste a token from huggingface.co/settings/tokens

4Making it fit & run faster让它跑得下、跑得快

On a 16–24 GB machine, two flags do most of the work. They trade a little speed for a lot less peak memory — the difference between "runs" and "killed: out of memory".在 16–24 GB 内存的机器上，两个开关就能解决大部分问题。它们用一点速度换取大幅降低的峰值内存——这正是“能跑”与“被系统杀掉：内存不足”的区别。

?Recap小结