在国内本地跑文生图与图生视频Text-to-Image & Image-to-Video, locally, inside China

不用翻墙、不用 Hugging Face、不用 Google。同样的 diffusers 代码，权重改从魔搭社区（ModelScope）下载——国内直连、速度快。文生图用可图 Kolors，图生视频用通义万相 Wan 2.2 或 CogVideoX，提示词支持中文。No VPN, no Hugging Face, no Google. The same diffusers code — you just download the weights from ModelScope (魔搭), China's domestic model hub, instead. Text-to-image with Kolors, image-to-video with Wan 2.2 or CogVideoX, Chinese prompts supported.

模型库 hub: ModelScope 魔搭 文生图 t2i: Kolors 可图 图生视频 i2v: Wan2.2 / CogVideoX 中文 prompt OK 无需翻墙 no VPN

★核心思想——Hugging Face 只是其中一个镜像The big idea — Hugging Face is just one mirror

在中国大陆，huggingface.co 和 Google 托管的权重通常无法直连。但模型本身没有国界——同样的 .safetensors 权重在 魔搭社区 ModelScope（阿里达摩院运营，相当于国内版 Hugging Face）上一份不少地重新托管，而且可图、混元、通义万相、CogVideoX 这些国产模型本来就首发在魔搭。你写的 diffusers 代码一行都不用改，只是把“从哪里下载权重”从 Hugging Face 换成魔搭：In mainland China, huggingface.co and Google-hosted weights are usually unreachable. But the models themselves have no borders — the same .safetensors weights are re-hosted on ModelScope (魔搭), run by Alibaba's DAMO Academy as China's equivalent of Hugging Face — and China-origin models (Kolors, Hunyuan, Wan, CogVideoX) launch there first. Your diffusers code doesn't change at all; you only swap where the weights come from:

墙外 / 国际版Blocked / international	国内直连的对应China-accessible equivalent
huggingface.co	modelscope.cn （魔搭社区）(ModelScope)
pip install huggingface_hub · hf download	pip install modelscope · modelscope download
from huggingface_hub import snapshot_download	from modelscope import snapshot_download
PyPI（pypi.org，常超时）PyPI (pypi.org, often times out)	阿里云 / 清华 pip 镜像Aliyun / Tsinghua pip mirror
diffusers 的 pipelinepipelines	完全相同——pipeline 代码不变identical — the pipeline code is unchanged

1环境准备——一次 pip 安装（走国内镜像）Setup — one pip install (via a China mirror)

需要 Python 3.10+。用阿里云 pip 镜像装包，避免 PyPI 超时。modelscope 负责下载，diffusers 负责推理。You need Python 3.10+. Install through the Aliyun pip mirror so PyPI doesn't time out. modelscope does the downloading, diffusers does the inference.

# 干净的虚拟环境 / a clean virtual environment
python -m venv .venv
# Windows: .venv\Scripts\activate   ·   macOS/Linux: source .venv/bin/activate

# 走阿里云镜像安装，速度快 / install via Aliyun mirror (fast in China)
pip install -i https://mirrors.aliyun.com/pypi/simple/ \
    modelscope diffusers transformers accelerate safetensors sentencepiece
pip install -i https://mirrors.aliyun.com/pypi/simple/ torch
pip install -i https://mirrors.aliyun.com/pypi/simple/ imageio imageio-ffmpeg pillow  # 视频导出用 / for saving video

2从魔搭下载模型——两种方法Download from ModelScope — two ways

两种方法下载到同一个缓存，脚本都能找到。关键点：snapshot_download() 返回一个本地路径，把这个路径喂给 diffusers 的 from_pretrained() 即可——和直连 Hugging Face 的写法只差这一步。下面以可图 Kolors 为例。Both methods write to the same cache, so your script finds the files either way. The key: snapshot_download() returns a local path — hand that path to diffusers' from_pretrained(). That one line is the only difference from the Hugging Face version. Example: Kolors.

方法 A —— Python 里 snapshot_downloadMethod A — snapshot_download in Python

from modelscope import snapshot_download

# 从魔搭下载，返回本地目录 / downloads from ModelScope, returns a local dir
model_dir = snapshot_download("Kwai-Kolors/Kolors-diffusers")
print(model_dir)   # 例如 / e.g.  ~/.cache/modelscope/hub/Kwai-Kolors/Kolors-diffusers

方法 B —— 命令行 modelscope downloadMethod B — the modelscope CLI

# 下载到缓存 / download into the cache
modelscope download --model Kwai-Kolors/Kolors-diffusers

# 或下载到看得见的文件夹 / or into a folder you can see
modelscope download --model Kwai-Kolors/Kolors-diffusers --local_dir ./Kolors

在 modelscope.cn/models 搜模型名就能拿到准确的 组织/模型 id。下一节会列出本页用到的全部 id。Search a model name at modelscope.cn/models to get its exact org/model id. The next sections list every id used here.

3文生图——可图 KolorsText → image — Kolors (可图)

可图 Kolors（快手开源）是一个中英双语文生图模型，中文提示词原生支持，中文字形渲染尤其出色。它用 EulerDiscrete 调度器，推荐 guidance_scale=5.0、num_inference_steps=50。注意：Kolors 自带一个 ChatGLM 文本编码器，模型较大（约 10 GB+），建议用 GPU；CPU 能跑但很慢。Kolors (open-sourced by Kuaishou) is a bilingual CN/EN text-to-image model with native Chinese prompt support and excellent Chinese-glyph rendering. It uses the EulerDiscrete scheduler; the recommended settings are guidance_scale=5.0 and num_inference_steps=50. Note: Kolors bundles a ChatGLM text encoder and is large (~10 GB+), so a GPU is recommended; CPU works but is slow.

# t2i.py  —  文生图 / text to image, weights from ModelScope
import torch
from diffusers import KolorsPipeline
from modelscope import snapshot_download

# 选设备 / pick a device: NVIDIA → cuda, 苹果芯片 Apple Silicon → mps, 否则 cpu
device = "cuda" if torch.cuda.is_available() \
    else "mps" if torch.backends.mps.is_available() else "cpu"
dtype  = torch.float16 if device != "cpu" else torch.float32

# 1 — 从魔搭下载（首次下载，之后走缓存）/ download from ModelScope (cached after first run)
model_dir = snapshot_download("Kwai-Kolors/Kolors-diffusers")

# 2 — 加载 pipeline / load the pipeline from the LOCAL path
pipe = KolorsPipeline.from_pretrained(
    model_dir, torch_dtype=dtype, variant="fp16",
).to(device)
pipe.enable_attention_slicing()        # 省内存 / less RAM

# 3 — 中文提示词直接可用 / Chinese prompt works directly
image = pipe(
    prompt="清晨薄雾中的清迈双龙寺，金色阳光，兰纳建筑，高细节",
    negative_prompt="模糊, 变形, 低质量",
    guidance_scale=5.0,            # Kolors 推荐 / recommended
    num_inference_steps=50,        # Kolors 推荐 / recommended
    generator=torch.Generator(device).manual_seed(42),  # 可复现 / reproducible
).images[0]

image.save("temple.png")
print("saved temple.png")

python t2i.py        # 首次下载约 10 GB / first run downloads ~10 GB, then reuses the cache

换模型——改一个字符串Swapping models — change one string

不同模型之间，要改的只是 snapshot_download() 里的魔搭 id、对应的 pipeline 类，以及步数 / guidance。Between models, you change only the ModelScope id in snapshot_download(), the matching pipeline class, and the step/guidance settings.

模型（魔搭 id）Model (ModelScope id)	Pipeline 类 / 设置Pipeline / settings	适用于Use it for
Kwai-Kolors/Kolors-diffusers 可图 · 快手Kolors · Kuaishou	KolorsPipeline 50 步，CFG 550 steps, CFG 5	中英双语，文字渲染强——上面的默认。Bilingual, strong text rendering — the default above.
Xorbits/HunyuanDiT-v1.2-Diffusers-Distilled 混元 · 腾讯（蒸馏版）Hunyuan-DiT · Tencent (distilled)	HunyuanDiTPipeline 25 步25 steps	中英双语，蒸馏版步数更少、更轻——CPU 更友好。Bilingual, distilled to fewer steps — friendlier on CPU.
Wan-AI/Wan2.2-TI2V-5B-Diffusers 通义万相 · 阿里（图也能转文）Wan2.2 · Alibaba (also t2i)	WanPipeline	同一模型也做文生图；主用途见第 4 节图生视频。Same model also does text-to-image; mainly for i2v in §4.

4图生视频——通义万相 Wan 2.2 / CogVideoXImage → video — Wan 2.2 / CogVideoX

把第 3 节生成的 temple.png 喂给图生视频模型，让它“动起来”。两个国产开源选择：通义万相 Wan2.2-TI2V-5B（阿里，中文提示词原生支持，720P）和 CogVideoX-5b-I2V（智谱，提示词仅英文，需把中文翻成英文）。Feed the temple.png from §3 to an image-to-video model to make it move. Two open-source Chinese options: Wan2.2-TI2V-5B (Alibaba, native Chinese prompts, 720P) and CogVideoX-5b-I2V (Zhipu, English prompts only — translate Chinese first).

# i2v.py  —  图生视频 / image to video (Wan 2.2, Chinese prompts OK)
import torch
from diffusers import WanImageToVideoPipeline
from diffusers.utils import load_image, export_to_video
from modelscope import snapshot_download

model_dir = snapshot_download("Wan-AI/Wan2.2-TI2V-5B-Diffusers")

pipe = WanImageToVideoPipeline.from_pretrained(model_dir, torch_dtype=torch.bfloat16)
pipe.to("cuda")                  # 视频强烈建议 GPU / video really wants a GPU
pipe.enable_model_cpu_offload()    # 显存不够时省显存 / spill to RAM if VRAM is tight

image = load_image("temple.png")
frames = pipe(
    image=image,
    prompt="云雾在山间缓缓流动，清晨光线，镜头缓慢推进",   # 中文 OK / Chinese OK
    height=704, width=1280, num_frames=121,        # 见模型卡 / see model card
    num_inference_steps=50,
).frames[0]

export_to_video(frames, "temple.mp4", fps=24)
print("saved temple.mp4")

5在一次渲染中引用多个 .md 文件和图片Reference many .md files & images in one render

真实项目不会把提示词写死。一部卡通剧集把画风、角色和每个镜头分别放在不同的 .md 文件里，并用参考 .png 锁定真实外观——一本「设定集」。渲染时，脚本会把它们全部读入、把文本拼成一条提示词、并把图片喂进 pipeline。这就是让第 800 帧与第 1 帧保持一致的全部诀窍。可图 Kolors 与通义万相 Wan 都原生支持中文，所以整本设定集都能用中文写。（设定集如何撰写见 准备一次，永远开拍。）Real projects don't hard-code the prompt string. A cartoon series keeps its look, its characters, and each shot in separate .md files, and locks the actual appearance in reference .pngs — a "show bible". At render time a script reads them all, concatenates the text into one prompt, and feeds the images into the pipeline. That is the whole trick to keeping frame #800 on-model with frame #1. Both Kolors and Wan accept Chinese natively, so the whole bible can be written in Chinese. (See Prepare Once, Animate Forever for how the bible is authored.)

A —— 把多个 .md 文件的文本拼接起来A — Concatenate the text from several .md files

每个设定集文件在 # 标题 行之后就是普通 YAML，所以一个极小的加载函数就能读取任意一个。场景文件指明用哪些角色和地点；脚本把这些名字解析成锁定的 token，再把一切拼成一条 prompt。style.md 里的 model 字段填魔搭 id，照样用 snapshot_download 下载。多一个依赖：pip install pyyaml。Each bible file is plain YAML after its # title line, so one tiny loader reads any of them. The scene names which characters and location to use; the script resolves those names to the locked tokens and joins everything into a single prompt. The model field in style.md is a ModelScope id, downloaded the same way with snapshot_download. One extra dependency: pip install pyyaml.

# render_scene.py — 用多个 .md 文件拼出一条提示词再渲染 / assemble ONE prompt from many .md files
import yaml, torch
from pathlib import Path
from diffusers import KolorsPipeline
from modelscope import snapshot_download

def fields(path):
    # 去掉 '# 标题' 行后就是 YAML / a .md bible file is YAML once you drop its '# title' line
    body = "".join(l for l in open(path, encoding="utf-8") if not l.startswith("#"))
    return yaml.safe_load(body)

style = fields("style.md")                  # render_tokens, negative, model（魔搭 id）
chars = {p.stem: fields(p) for p in Path("characters").glob("*.md")}
scene = fields("scenes/scene_01.md")        # cast, action, camera, motion

# 把场景点名的每个角色解析成各自锁定的 look_tokens / resolve scene.cast -> locked look_tokens
looks  = [chars[name]["look_tokens"] for name in scene["cast"]]
prompt = "，".join(looks + [scene["action"], scene["camera"], style["render_tokens"]])

model_dir = snapshot_download(style["model"])   # 例如 Kwai-Kolors/Kolors-diffusers
pipe = KolorsPipeline.from_pretrained(model_dir, torch_dtype=torch.float16).to("cuda")
image = pipe(
    prompt=prompt,
    negative_prompt=style["negative"],
    num_inference_steps=50, guidance_scale=5.0,   # Kolors 推荐 / recommended
).images[0]
Path("renders").mkdir(exist_ok=True)
image.save("renders/scene_01.png")

B —— 一次喂入多张参考图片（Kolors IP-Adapter）B — Feed several reference images at once (Kolors IP-Adapter)

文字会走样，图片不会。要锁定真实画作——角色转面图、场景背景——就把它们作为像素传入。可图自带 Kolors IP-Adapter（魔搭 id Kwai-Kolors/Kolors-IP-Adapter-Plus），它需要显式加载一个 image_encoder；给 ip_adapter_image 传一个列表就能同时参考多张图。Words drift; pictures don't. To pin the actual drawing — the character turnaround, the location backdrop — pass them as pixels. Kolors has its own IP-Adapter (ModelScope id Kwai-Kolors/Kolors-IP-Adapter-Plus), which needs an explicit image_encoder; pass a list to ip_adapter_image to reference several images at once.

# 用图片当参考，而不只是文字 / reference the actual PICTURES, not just words
from transformers import CLIPVisionModelWithProjection
from diffusers.utils import load_image

ipa = snapshot_download("Kwai-Kolors/Kolors-IP-Adapter-Plus")   # 参考图适配器 / the IP-Adapter
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    ipa, subfolder="image_encoder", torch_dtype=torch.float16)

pipe = KolorsPipeline.from_pretrained(
    model_dir, image_encoder=image_encoder, torch_dtype=torch.float16).to("cuda")
pipe.load_ip_adapter(ipa, subfolder="", weight_name="ip_adapter_plus_general.safetensors")
pipe.set_ip_adapter_scale(0.5)    # 参考强度 / how strongly the refs steer it

image = pipe(
    prompt=prompt, negative_prompt=style["negative"],
    ip_adapter_image=[                          # 传一个列表 = 同时参考多张图 / a LIST = several refs at once
        load_image(chars["ploy"]["refs"][0]),  # characters/ploy_turnaround.png
        load_image("locations/village_lane_wide.png"),
    ],
    num_inference_steps=50, guidance_scale=5.0,
).images[0]

refs: 列表直接来自 characters/ploy.md——YAML 加载器已解析好，chars["ploy"]["refs"][0] 就是转面图路径。若想分别独立控制「人物 vs 背景」（每个适配器单独设权重），SDXL 系可加载多个 IP-Adapter——做法见国际版 diffusers 模型指南 第 4 节。只想对单张图轻度重绘，则用 KolorsImg2ImgPipeline 配 image= 和 strength=。The refs: list comes straight from characters/ploy.md — already parsed by the YAML loader, so chars["ploy"]["refs"][0] is the turnaround path. To control "character vs background" independently (a separate scale per adapter), SDXL-family pipelines can load multiple IP-Adapters — see §4 of the international guide. For a light redraw of a single still, use KolorsImg2ImgPipeline with image= + strength=.

C —— 同一组参考延续到视频C — The same references carry into the video

因为你刚渲染的静帧本身就是所有参考合成的一帧，让它动起来无需新的参考——这张合成 PNG 直接成为视频的第一帧。而通义万相 Wan 接收文本提示词，所以场景的 motion: 字段可原样当作中文运动提示词。Because the still you just rendered already is every reference composed into one frame, animating it needs no new references — the composed PNG becomes the video's first frame. And since Wan takes a text prompt, the scene's motion: field can be used directly as the (Chinese) motion prompt.

# 合成好的静帧就是视频的参考 / the composed still IS the video's reference (frame 0)
from diffusers import WanImageToVideoPipeline
from diffusers.utils import load_image, export_to_video

vdir = snapshot_download("Wan-AI/Wan2.2-TI2V-5B-Diffusers")
pipe_i2v = WanImageToVideoPipeline.from_pretrained(vdir, torch_dtype=torch.bfloat16).to("cuda")

seed   = load_image("renders/scene_01.png")
frames = pipe_i2v(
    image=seed,
    prompt=scene["motion"],        # 场景的 motion: 字段直接当中文运动提示词 / the scene's motion: line, Chinese OK
    num_frames=121, num_inference_steps=50,
).frames[0]
export_to_video(frames, "renders/scene_01.mp4", fps=24)

每个来源在调用中的落点Where every source lands in the call

来源文件Source file	字段Field	进入Goes into
style.md	model	snapshot_download(...) 魔搭 idModelScope id
style.md	render_tokens	附加到每条appended to every prompt
style.md	negative	negative_prompt=
characters/*.md	look_tokens	由resolved from scene.cast 解析进into prompt
characters/*.md	refs (png)	ip_adapter_image= / 初始图init image=
locations/*.png	—	ip_adapter_image= / i2v 第一帧i2v seed frame
scenes/scene_NN.md	action, camera	prompt
scenes/scene_NN.md	motion	Wan i2v 提示词（中文原生）Wan i2v prompt (native Chinese)
scenes/scene_NN.md	seconds	片段时长 /clip length / num_frames

6让它跑得下、跑得快Making it fit & run faster

调用Call	作用What it does
pipe.enable_attention_slicing()	分块算注意力——单项最省内存。Attention in chunks — biggest single RAM saver.
pipe.vae.enable_tiling()	分块解码画面，视频模型尤其需要。Decodes frames in tiles — key for video.
pipe.enable_model_cpu_offload()	GPU 用——空闲子模型放回内存，按需搬上显存，省 VRAM。GPU only — idle sub-models stay in RAM, moved to VRAM just in time.
pipe.enable_sequential_cpu_offload()	更激进的换出，显存极少时用（更慢）。More aggressive offload for very low VRAM (slower).
num_inference_steps ↓	步数越少越快；蒸馏 / Turbo 模型为此而生。Fewer steps = faster; distilled/turbo models are built for it.

?小结Recap

	文生图Text → image	图生视频Image → video
下载来源Download from	魔搭 ModelScope — snapshot_download("组织/模型")ModelScope — snapshot_download("org/model")
Pipeline	KolorsPipeline	WanImageToVideoPipeline
推荐模型Model	Kwai-Kolors/Kolors-diffusers	Wan-AI/Wan2.2-TI2V-5B-Diffusers
中文提示词Chinese prompts	原生支持native	Wan 原生 / CogVideoX 需翻译Wan native / CogVideoX needs translation
硬件Hardware	GPU 佳，CPU 可（慢）GPU best, CPU ok (slow)	需要 GPUneeds a GPU