⚡ vLLM instead of Ollama · ใช้ vLLM แทน Ollama · 用 vLLM 取代 Ollama

1 One person, or a class of thirty · คนเดียว หรือทั้งห้องสามสิบคน · 一个人，还是三十人一个班

2 They both speak the same API · ทั้งคู่พูด API เดียวกัน · 它们说的是同一套 API

3 One container, and the flags that matter · คอนเทนเนอร์เดียว กับแฟล็กที่สำคัญ · 一个容器，和那几个要紧的参数

📄 serve_vllm.sh

#!/bin/sh
# serve_vllm.sh — start one vLLM server, and leave the rest of the machine
# alone.
#
# Ollama is a service you install once and then forget. vLLM is a process
# you start with the model already chosen. That is the first difference,
# and every other difference follows from it.
#
# The table on the page explains every flag below. The two settings most
# worth changing are up here as variables.

set -eu

# What to serve. vLLM fetches this from Hugging Face the first time, so the
# name is a repository, not a nickname. The -AWQ build is four bits and
# 5.6 GB; drop that suffix for the full sixteen bit weights and 15 GB.
MODEL=${MODEL:-Qwen/Qwen2.5-7B-Instruct-AWQ}

PORT=${PORT:-8000}

# Which image. This line is not a detail. The ordinary
# vllm/vllm-openai:latest carries a PyTorch built for cards up to sm_120,
# and this machine's Blackwell chip is sm_121, so the server dies on
# startup saying no kernel image is available for execution on the device.
# vLLM's own DGX Spark writeup names the CUDA 13 nightly track instead.
# A nightly moves under you, so pin the digest once something works.
IMAGE=${IMAGE:-vllm/vllm-openai:cu130-nightly}

# The share of the card vLLM claims at startup and keeps. The default is
# 0.9, which takes almost everything. This box runs other things.
FRACTION=${FRACTION:-0.25}

# The longest prompt plus answer the server will accept. It also decides
# how big the reserved cache has to be. Qwen2.5 offers 32768, which needs
# far more memory than one demo should take.
MAXLEN=${MAXLEN:-8192}

exec docker run --rm --name vllm \
  --gpus all \
  --ipc=host \
  -p "${PORT}":8000 \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  "${IMAGE}" \
  --model "${MODEL}" \
  --gpu-memory-utilization "${FRACTION}" \
  --max-model-len "${MAXLEN}" \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

🖥️ WHAT IT PRINTED · starting the server, first time, on an NVIDIA GB10

$ sh serve_vllm.sh

Resolved architecture: Qwen2ForCausalLM
Using max model len 8192
The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
Model loading took 5.2 GiB memory and 843.057112 seconds
GPU KV cache size: 395,872 tokens
Maximum concurrency for 8,192 tokens per request: 48.32x
init engine (profile, create kv cache, warmup model) took 66.10 s
INFO:     Started server process [1]
INFO:     Application startup complete.

Stopping it and starting the same command again, with the model now cached:

Model loading took 5.2 GiB memory and 36.585409 seconds
Maximum concurrency for 8,192 tokens per request: 48.65x
INFO:     Application startup complete.

4 Ask it what the model is called · ถามมันว่าโมเดลชื่ออะไร · 问它模型叫什么名字

📄 check_vllm.py

"""check_vllm.py — is the server up, and what is it actually called?

Ollama and vLLM both answer at /v1, the shape OpenAI invented. That is the
whole reason you can swap one for the other. But they disagree about what a
model is called, and that one difference breaks more first attempts than
anything else on this page.

Run this before you write any other code against a new server.
"""
import json
import os
import sys
import urllib.error
import urllib.request

# Two servers, two ports. Ollama picks 11434, vLLM picks 8000. Neither is
# a standard; they are just the defaults each project chose.
SERVERS = {
    "vLLM": os.environ.get("VLLM_URL", "http://localhost:8000/v1"),
    "Ollama": os.environ.get("OLLAMA_URL", "http://localhost:11434/v1"),
}


def ask(url, path):
    """GET one endpoint and hand back the parsed JSON, or an excuse."""
    # Neither server requires this header, and both ignore what is in
    # it. Every OpenAI client sends one anyway, so it is here to match.
    request = urllib.request.Request(
        url + path, headers={"Authorization": "Bearer none"})
    try:
        with urllib.request.urlopen(request, timeout=15) as response:
            return json.load(response), None
    except urllib.error.URLError as trouble:
        # A server that is not running looks exactly like one that is
        # running somewhere else. Say which URL failed, not just that it did.
        return None, str(getattr(trouble, "reason", trouble))
    except Exception as trouble:
        return None, str(trouble)


def main():
    for name, url in SERVERS.items():
        print(name, "at", url)
        listing, trouble = ask(url, "/models")
        if trouble:
            print("   not answering:", trouble)
            print()
            continue

        # The reply is the same shape from both: {"data": [{"id": ...}]}.
        # What differs is what is in the id, and that is the point.
        for entry in listing.get("data", []):
            print("   model id:", entry["id"])
        print()

    print("The id is not a nickname you choose.")
    print("Ollama names a model it downloaded:   qwen2.5:14b")
    print("vLLM names the thing you started it with, in full.")
    print("Send the wrong one and you get a 404, not a guess.")


if __name__ == "__main__":
    main()

🖥️ WHAT IT PRINTED · both servers, asked what they serve

vLLM at http://spark.local:8000/v1
   model id: Qwen/Qwen2.5-7B-Instruct-AWQ

Ollama at http://spark.local:11434/v1
   model id: qwen2.5:1.5b
   model id: qwen2.5:7b
   model id: nomic-embed-text:latest
   model id: gemma2:9b
   model id: llama3.1:8b
   model id: scb10x/typhoon2.1-gemma3-12b:latest
   model id: qwen2.5:14b
   model id: llama3.2:3b

The id is not a nickname you choose.
Ollama names a model it downloaded:   qwen2.5:14b
vLLM names the thing you started it with, in full.
Send the wrong one and you get a 404, not a guess.

5 One question, no library · หนึ่งคำถาม ไม่ต้องใช้ไลบรารี · 一个问题，不用任何库

📄 ask_vllm.py

"""ask_vllm.py — one question, no SDK, and the same code for both servers.

There is no vLLM library here. No OpenAI library either. Just a POST with
some JSON in it. That is all any of those libraries do underneath.

Read it once and the swap stops being mysterious. You change a URL and a
model name. Everything else is the same bytes on the wire.
"""
import json
import os
import sys
import time
import urllib.request

# Which server, and what it calls the model. The pair always travels
# together: a vLLM model id sent to Ollama is a 404, and the other way too.
SERVERS = {
    "vllm": (os.environ.get("VLLM_URL", "http://localhost:8000/v1"),
             os.environ.get("VLLM_MODEL", "Qwen/Qwen2.5-7B-Instruct-AWQ")),
    "ollama": (os.environ.get("OLLAMA_URL", "http://localhost:11434/v1"),
               os.environ.get("OLLAMA_MODEL", "qwen2.5:7b")),
}

QUESTION = "What is the past tense of go? Answer with one word."


def ask(url, model, question):
    """Send one chat completion and hand back the answer and the timing."""
    # This body is the OpenAI chat format. Both servers read it. The only
    # required fields are the model and the messages.
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": question}],

        # Sampling settled at 0 so two servers can be compared fairly.
        # Leave it out and each picks its own default, and you end up
        # measuring the dice rather than the server.
        "temperature": 0,
        "max_tokens": 20,
    }).encode()

    request = urllib.request.Request(
        url + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 # Ignored by both and required by neither. Every
                 # OpenAI client sends one, so this matches them.
                 "Authorization": "Bearer none"})

    started = time.time()
    with urllib.request.urlopen(request, timeout=300) as response:
        reply = json.load(response)
    took = time.time() - started

    # The answer is buried the same distance down in both servers.
    said = reply["choices"][0]["message"]["content"].strip()

    # usage is how many tokens went in and came out. Both servers fill
    # it in, so it is the one number you can compare across them.
    usage = reply.get("usage") or {}
    return said, took, usage


def main():
    # One name on the command line, or both if you give none.
    wanted = sys.argv[1:] or list(SERVERS)

    for name in wanted:
        if name not in SERVERS:
            print(name, "is not a server I know about")
            continue
        url, model = SERVERS[name]
        print(name, "->", url, "model", model)
        try:
            # The first question of the day also loads the model, and on a
            # cold server that is nearly all of the time measured. Ask once
            # and throw it away, or the number below is a lie.
            _, cold, _ = ask(url, model, "Say ok.")
            print("   first call: %.2f s (loading the model, not answering)"
                  % cold)
            said, took, usage = ask(url, model, QUESTION)
        except Exception as trouble:
            print("   failed:", trouble)
            print()
            continue

        print("   answer :", repr(said))
        print("   took   : %.2f s" % took)
        print("   tokens : %s in, %s out" % (usage.get("prompt_tokens", "?"),
                                             usage.get("completion_tokens", "?")))

        # No tokens-a-second here on purpose. A two word answer is mostly
        # network and prompt reading, so that number would say more about
        # the wifi than the server. race.py measures speed properly.
        print()


if __name__ == "__main__":
    main()

🖥️ WHAT IT PRINTED · the same question, both servers

vllm -> http://spark.local:8000/v1 model Qwen/Qwen2.5-7B-Instruct-AWQ
   first call: 17.53 s (loading the model, not answering)
   answer : 'went'
   took   : 2.95 s
   tokens : 42 in, 2 out

ollama -> http://spark.local:11434/v1 model qwen2.5:7b
   first call: 5.92 s (loading the model, not answering)
   answer : 'went'
   took   : 0.18 s
   tokens : 42 in, 2 out

6 Same tool, same agent, other server · เครื่องมือเดิม เอเจนต์เดิม คนละเซิร์ฟเวอร์ · 同一个工具、同一个智能体，换个服务器

📄 tool_vllm.py

"""tool_vllm.py — the agent from the classes page, pointed at vLLM instead.

The tool is copied from agent_window.py without a character changed. So is
the way the agent is built. The only difference is the two lines at the top
that say where the server is and what the model is called.

That is the claim this file is here to test. Run it against both servers
and read the two transcripts side by side.

    python tool_vllm.py vllm
    python tool_vllm.py ollama
"""
import json
import os
import sys
import time
import urllib.request

from smolagents import ActionStep, FinalAnswerStep
from smolagents import OpenAIServerModel, ToolCallingAgent, tool

SERVERS = {
    "vllm": (os.environ.get("VLLM_URL", "http://localhost:8000/v1"),
             os.environ.get("VLLM_MODEL", "Qwen/Qwen2.5-7B-Instruct-AWQ")),
    "ollama": (os.environ.get("OLLAMA_URL", "http://localhost:11434/v1"),
               os.environ.get("OLLAMA_MODEL", "qwen2.5:7b")),
}

BITKUB = "https://api.bitkub.com/api/v3/market/ticker"
MAX_STEPS = 6

QUESTIONS = ["What is the past tense of go?",
             "What is BTC worth in baht right now?"]


@tool
def coin_price(symbol: str) -> str:
    """Look up what one coin is worth in Thai baht, right now.

    Args:
        symbol: the coin's ticker, for example BTC or ETH
    """
    # No double quotes above. smolagents shows that description to the
    # model, and a model that copies them back breaks its own JSON.
    pair = symbol.upper().strip() + "_THB"
    with urllib.request.urlopen(BITKUB + "?sym=" + pair, timeout=30) as response:
        coins = json.load(response)
    if not coins:
        return "no coin called " + symbol + " on Bitkub"
    return ("1 " + symbol.upper() + " = "
            + format(float(coins[0]["last"]), ",.2f") + " baht")


def main():
    name = sys.argv[1] if len(sys.argv) > 1 else "vllm"
    url, model_id = SERVERS[name]
    print("===", name, "===")
    print("server :", url)
    print("model  :", model_id)
    print()

    # This is the whole swap. OpenAIServerModel does not know or care
    # which program is answering at that address.
    llm = OpenAIServerModel(model_id=model_id, api_base=url, api_key="none")

    for question in QUESTIONS:
        agent = ToolCallingAgent(tools=[coin_price], model=llm,
                                 instructions="You are Kru Nari, a patient "
                                 "English teacher. Teach in short, plain "
                                 "English, and keep every reply under three "
                                 "sentences. If, and only if, someone asks "
                                 "what a coin is worth, you may use your "
                                 "coin_price tool to look it up.",
                                 verbosity_level=0)
        print("You:", question)
        used, answer = [], "?"
        started = time.time()
        try:
            for step in agent.run(question, stream=True, max_steps=MAX_STEPS):
                if isinstance(step, ActionStep):
                    used += [call.name for call in (step.tool_calls or [])]
                if isinstance(step, FinalAnswerStep):
                    answer = str(step.output)
        except Exception as trouble:
            answer = "failed: " + str(trouble)[:80]
        print("   tools called:", ", ".join(used) or "none")
        print("   Kru Nari    :", answer)
        print("   took        : %.1f s" % (time.time() - started))
        print()

    # The number the tool really returned, so the answer above can be
    # checked against it rather than believed.
    print("coin_price itself says:", coin_price("BTC"))


if __name__ == "__main__":
    main()

🖥️ WHAT IT PRINTED · the same agent against vLLM

=== vllm ===
server : http://spark.local:8000/v1
model  : Qwen/Qwen2.5-7B-Instruct-AWQ

You: What is the past tense of go?
   tools called: final_answer
   Kru Nari    : went
   took        : 22.1 s

You: What is BTC worth in baht right now?
   tools called: coin_price, final_answer
   Kru Nari    : 1 BTC is worth 2,796,092.45 baht right now.
   took        : 1.9 s

coin_price itself says: 1 BTC = 2,796,092.45 baht

🖥️ WHAT IT PRINTED · the same agent against Ollama

=== ollama ===
server : http://spark.local:11434/v1
model  : qwen2.5:7b

You: What is the past tense of go?
   tools called: final_answer
   Kru Nari    : went
   took        : 7.2 s

You: What is BTC worth in baht right now?
   tools called: coin_price, final_answer
   Kru Nari    : 1 BTC is currently worth 2,790,025.63 baht.
   took        : 2.0 s

coin_price itself says: 1 BTC = 2,790,025.63 baht

7 The measurement that decides it · การวัดที่ตัดสินเรื่องนี้ · 一测就知道该用哪个

📄 race.py

"""race.py — one request each, then eight at once. This is the whole point.

One person asking one question does not need vLLM. Ollama is easier and it
is fast enough. The difference shows up when several people ask at the same
moment, which is what a classroom is.

So this asks once, then asks eight times at once, and compares the wall
clock. A server that answers eight in about the time it took to answer one
is batching them. A server that takes eight times as long is queueing them.

Set AT_ONCE to send more than eight. That is challenge 4: raise it until
the wall clock stops being flat, and you have found your class size limit.
"""
import json
import os
import statistics
import sys
import threading
import time
import urllib.request

SERVERS = {
    "vllm": (os.environ.get("VLLM_URL", "http://localhost:8000/v1"),
             os.environ.get("VLLM_MODEL", "Qwen/Qwen2.5-7B-Instruct-AWQ")),
    "ollama": (os.environ.get("OLLAMA_URL", "http://localhost:11434/v1"),
               os.environ.get("OLLAMA_MODEL", "qwen2.5:7b")),
}

# Eight different questions, so no server can win by caching an answer it
# has already given. They are all about the same length on purpose.
QUESTIONS = [
    "Name one fruit that grows in northern Thailand.",
    "Name one river that runs through Chiang Mai.",
    "Name one animal that lives in a rice field.",
    "Name one tool a carpenter uses every day.",
    "Name one vegetable that is green and leafy.",
    "Name one instrument you blow air into.",
    "Name one bird that cannot fly at all.",
    "Name one metal that rusts when it gets wet.",
]

# Short answers keep the test about the server and not about the model
# writing an essay.
MAX_TOKENS = 30

# How many ask at the same moment. Eight is a small group; set AT_ONCE
# higher to find where your server stops keeping up. Challenge 4.
AT_ONCE = int(os.environ.get("AT_ONCE", "8"))


def crowd(size):
    """Build `size` questions, none of them the same as another.

    Past eight it starts reusing the list, so a number is added to keep
    every prompt distinct. Two identical prompts would let a server look
    fast by answering the second one from a cache.
    """
    out = []
    for number in range(size):
        question = QUESTIONS[number % len(QUESTIONS)]
        if number >= len(QUESTIONS):
            question = "%s (answer number %d)" % (question, number)
        out.append(question)
    return out


def ask(url, model, question, out):
    """One request. Appends (seconds, tokens out) to a shared list."""
    body = json.dumps({"model": model,
                       "messages": [{"role": "user", "content": question}],
                       "temperature": 0, "max_tokens": MAX_TOKENS}).encode()
    request = urllib.request.Request(
        url + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer none"})
    started = time.time()
    try:
        with urllib.request.urlopen(request, timeout=600) as response:
            reply = json.load(response)
        took = time.time() - started
        tokens = (reply.get("usage") or {}).get("completion_tokens", 0)

        # list.append is safe from several threads at once. Python's own
        # lock covers it, so no lock of our own is needed here.
        out.append((took, tokens))
    except Exception as trouble:
        out.append((time.time() - started, 0))
        print("      one request failed:", str(trouble)[:70])


def one_at_a_time(url, model, question):
    """The warm single request. Also loads the model if it is cold."""
    out = []
    ask(url, model, question, out)
    return out[0]


def all_at_once(url, model, questions):
    """Fire every question on its own thread and wait for the lot."""
    out = []
    threads = [threading.Thread(target=ask, args=(url, model, q, out))
               for q in questions]

    # The clock starts before the threads, so it includes the queueing
    # that a busy server does. That queueing is exactly what we measure.
    started = time.time()
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    return time.time() - started, out


def main():
    wanted = sys.argv[1:] or list(SERVERS)
    for name in wanted:
        url, model = SERVERS[name]
        print("===", name, "-", model, "===")

        # A cold model loads on the first request and that time is not
        # the server's fault. Ask once and throw it away.
        print("   warming up...")
        try:
            one_at_a_time(url, model, "Say ok.")
        except Exception as trouble:
            print("   not answering:", str(trouble)[:80], "\n")
            continue

        alone_took, _ = one_at_a_time(url, model, QUESTIONS[0])
        print("   one request alone     : %.2f s" % alone_took)

        wall, results = all_at_once(url, model, crowd(AT_ONCE))
        each = [t for t, _ in results]
        tokens = sum(n for _, n in results)
        print("   %d all at once%s: %.2f s wall clock"
              % (AT_ONCE, " " * (10 - len(str(AT_ONCE))), wall))
        print("      slowest of them   : %.2f s" % max(each))
        print("      median of them    : %.2f s" % statistics.median(each))
        print("      tokens out, total : %d" % tokens)
        if wall:
            print("      tokens a second   : %.1f" % (tokens / wall))

        # If they take about as long as one did, they overlapped. If
        # they take N times as long, they stood in a queue.
        print("   %d cost %.1fx one request" % (AT_ONCE, wall / alone_took))
        print()


if __name__ == "__main__":
    main()

🖥️ WHAT IT PRINTED · one request, then eight at the same moment

=== vllm - Qwen/Qwen2.5-7B-Instruct-AWQ ===
   warming up...
   one request alone     : 0.69 s
   8 all at once         : 0.70 s wall clock
      slowest of them   : 0.70 s
      median of them    : 0.69 s
      tokens out, total : 206
      tokens a second   : 292.9
   8 cost 1.0x one request

=== ollama - qwen2.5:7b ===
   warming up...
   one request alone     : 0.85 s
   8 all at once         : 4.98 s wall clock
      slowest of them   : 4.98 s
      median of them    : 2.89 s
      tokens out, total : 208
      tokens a second   : 41.7
   8 cost 5.8x one request

8 Six differences that cost an evening · หกความต่างที่กินเวลาทั้งคืน · 六个能耗掉你一晚上的差别

$ docker run --rm --gpus all --entrypoint nvidia-smi IMAGE \
      --query-gpu=name,compute_cap,memory.total --format=csv

name, compute_cap, memory.total [MiB]
NVIDIA GB10, 12.1, [N/A]

9 Try first. The answers are folded away. · ลองเองก่อน เฉลยพับซ่อนไว้ · 先自己试。答案折起来放着。

$ FRACTION=0.02 sh serve_vllm.sh

ValueError: No available memory for the cache blocks. Try increasing
`gpu_memory_utilization` when initializing the engine.

RuntimeError: Engine core initialization failed. See root cause above.
$ echo $?
1

$ python tool_vllm.py vllm

You: What is the past tense of go?
Error code: 400 - tool_choice="required" requires --tool-call-parser to be set
   tools called: none

Asked three more ways, to see whether it is ever quiet about it:

  tool_choice omitted    -> HTTP 400 | "auto" tool choice requires
                            --enable-auto-tool-choice and --tool-call-parser
  tool_choice=auto       -> HTTP 400 | "auto" tool choice requires
                            --enable-auto-tool-choice and --tool-call-parser
  tool_choice=required   -> HTTP 400 | tool_choice="required" requires
                            --tool-call-parser to be set

=== vllm - Qwen/Qwen2.5-7B-Instruct-AWQ ===
   warming up...
   one request alone     : 0.68 s
   32 all at once        : 0.93 s wall clock
      slowest of them   : 0.92 s
      median of them    : 0.90 s
      tokens out, total : 755
      tokens a second   : 811.7
   32 cost 1.4x one request

=== ollama - qwen2.5:7b ===
   warming up...
   one request alone     : 0.80 s
   32 all at once        : 19.35 s wall clock
      slowest of them   : 19.33 s
      median of them    : 10.02 s
      tokens out, total : 833
      tokens a second   : 43.1
   32 cost 24.2x one request