🎛️ Fine-tune a language model with Python · ปรับจูนโมเดลภาษาด้วย Python · 用 Python 微调语言模型

1 A separate Python, and three libraries · Python อีกชุดหนึ่ง กับไลบรารีสามตัว · 一份单独的 Python，和三个库

python -m venv venv
venv\Scripts\python -m pip install torch transformers peft

python3 -m venv venv
venv/bin/python -m pip install torch transformers peft

2 Load the model and ask it something · โหลดโมเดลแล้วลองถาม · 加载模型，问它点什么

💻 RUN IT

venv\Scripts\python ask.py

📄 NEW FILE · ask.py

"""ask.py - load a small language model and ask it two things.

    python ask.py

The first run downloads the model, about 270 MB. Later runs start in seconds.
"""
import time

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"


def reply_to(model, tokenizer, chat):
    """Send a chat to the model and return the text it writes back."""
    chat_tokens = tokenizer.apply_chat_template(
        chat, add_generation_prompt=True, return_tensors="pt")
    question_tokens = chat_tokens["input_ids"]
    # do_sample=False always takes the most likely token: the same reply every run.
    output = model.generate(question_tokens, max_new_tokens=60, do_sample=False)
    # output is one row of tokens: first the question, then the reply.
    # shape[1] is how many tokens the question has, so this keeps the reply.
    reply_tokens = output[0][question_tokens.shape[1]:]
    return tokenizer.decode(reply_tokens, skip_special_tokens=True)


def main():
    started = time.perf_counter()
    tokenizer = AutoTokenizer.from_pretrained(MODEL)
    model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
    print(f"Loaded in {time.perf_counter() - started:.1f} seconds.")
    print()

    chat = [{"role": "user", "content": "What is the capital of Thailand?"}]
    print("What the model reads:")
    print(tokenizer.apply_chat_template(
        chat, tokenize=False, add_generation_prompt=True))
    print("What it writes back:")
    print(reply_to(model, tokenizer, chat))
    print()

    chat = [
        {"role": "system",
         "content": "You take orders for a noodle shop. Reply with the order as JSON."},
        {"role": "user",
         "content": "Two khao soi with chicken, not too spicy, to take away please."},
    ]
    print("The order, before any training:")
    print(reply_to(model, tokenizer, chat))


if __name__ == "__main__":
    main()

🖨️ WHAT IT PRINTED

Loaded in 2.0 seconds.

What the model reads:
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
What is the capital of Thailand?<|im_end|>
<|im_start|>assistant

What it writes back:
The capital of Thailand is Bangkok.

The order, before any training:
{
  "name": "Khao Soi with Chicken",
  "quantity": 2,
  "price": 10.99,
  "quantity_per_person": 2,
  "price_per_person": 10.99,
  "quantity

3 Give it the job, and try a prompt first · ให้งานมันทำ และลองใช้พรอมต์ก่อน · 把任务交给它，先试试提示词

💻 RUN IT

venv\Scripts\python before.py

📄 NEW FILE · before.py

"""before.py - test the model on the job before any training.

    python before.py

It sends the 50 messages in test_handwritten.jsonl to the model twice: first
with a short instruction, then with a long one that states every rule and shows
three solved examples. Each time it counts the orders the model gets right.
"""
import json

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
INSTRUCTION = "You take orders for a noodle shop. Reply with the order as JSON."
LONG_INSTRUCTION = """You take orders for a noodle shop.
Reply with the order as JSON, and with nothing else.
The JSON has exactly these five keys:
"dish": "khao soi", "nam ngiao", "pad thai" or "pad see ew"
"protein": "chicken", "pork", "beef" or "tofu"
"spice": "none", "mild", "medium" or "hot". Use "medium" if the customer does not say.
"count": how many, as a number. Use 1 if the customer does not say.
"takeaway": true or false. Use false if the customer does not say."""
SOLVED_EXAMPLES = [
    {"message": "Three khao soi with pork, a little spicy, to go please.",
     "order": {"dish": "khao soi", "protein": "pork", "spice": "mild",
               "count": 3, "takeaway": True}},
    {"message": "pad thai tofu no chilli",
     "order": {"dish": "pad thai", "protein": "tofu", "spice": "none",
               "count": 1, "takeaway": False}},
    {"message": "Hello, two nam ngiao with beef, very spicy, eating here.",
     "order": {"dish": "nam ngiao", "protein": "beef", "spice": "hot",
               "count": 2, "takeaway": False}},
]


def read_examples(name):
    """Every line of the file is one example, written as JSON."""
    examples = []
    with open(name, encoding="utf-8") as file:
        for line in file:
            examples.append(json.loads(line))
    return examples


def short_chat(message):
    """The one-line instruction, then the message."""
    return [
        {"role": "system", "content": INSTRUCTION},
        {"role": "user", "content": message},
    ]


def long_chat(message):
    """The rules, then three orders already answered, then the new message."""
    chat = [{"role": "system", "content": LONG_INSTRUCTION}]
    for solved in SOLVED_EXAMPLES:
        chat.append({"role": "user", "content": solved["message"]})
        chat.append({"role": "assistant", "content": json.dumps(solved["order"])})
    chat.append({"role": "user", "content": message})
    return chat


def reply_to(model, tokenizer, chat):
    """The model's reply to a chat, on one line."""
    chat_tokens = tokenizer.apply_chat_template(
        chat, add_generation_prompt=True, return_tensors="pt")
    question_tokens = chat_tokens["input_ids"]
    output = model.generate(question_tokens, max_new_tokens=50, do_sample=False)
    # output is the question and then the reply. Keep the reply.
    reply_tokens = output[0][question_tokens.shape[1]:]
    reply = tokenizer.decode(reply_tokens, skip_special_tokens=True)
    return " ".join(reply.split())


def is_right(reply, order):
    """True when the reply is JSON and says exactly what the order says."""
    try:
        return json.loads(reply) == order
    except ValueError:              # json.loads could not read the reply as JSON
        return False


def main():
    tokenizer = AutoTokenizer.from_pretrained(MODEL)
    model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
    tests = read_examples("test_handwritten.jsonl")
    for instruction in ["short", "long"]:
        right = 0
        shown = 0
        for test in tests:
            if instruction == "short":
                chat = short_chat(test["message"])
            else:
                chat = long_chat(test["message"])
            reply = reply_to(model, tokenizer, chat)
            if is_right(reply, test["order"]):
                right = right + 1
            if shown < 2:           # show the first two replies as they are
                print(test["message"])
                print("   ", reply)
                shown = shown + 1
        print(f"{instruction} instruction: {right} of {len(tests)} right")
        print()


if __name__ == "__main__":
    main()

🖨️ WHAT IT PRINTED

Two khao soi with chicken, mild, to take away please.
    { "name": "Khao Soi with Chicken", "quantity": 2, "price": 10.99, "quantity": 2, "price": 10.99,
One pad thai with tofu, not spicy. I will eat here.
    { "order": { "name": "One Pad Thai with Tofu", "quantity": 2, "price": 10.99, "menu": [ { "name": "Pad
short instruction: 0 of 50 right

Two khao soi with chicken, mild, to take away please.
    {"dish": "khao soi", "protein": "chicken", "spice": "chicken", "count": 1, "takeaway": false}
One pad thai with tofu, not spicy. I will eat here.
    {"dish": "pad thai", "protein": "beef", "spice": "hot", "count": 1, "takeaway": false}
long instruction: 1 of 50 right

4 Write 400 examples with a program · เขียนตัวอย่าง 400 ชุดด้วยโปรแกรม · 用程序写出 400 个样例

💻 RUN IT

venv\Scripts\python make_data.py

📄 NEW FILE · make_data.py

"""make_data.py - write the examples the model will learn from.

    python make_data.py

One example is a customer's message and the order it means. A program writes
them, so there can be as many as we like. It writes two files:

    train.jsonl           400 examples to learn from
    test_generated.jsonl  50 more, made the same way and kept for testing
"""
import json
import random

DISHES = ["khao soi", "nam ngiao", "pad thai", "pad see ew"]
PROTEINS = ["chicken", "pork", "beef", "tofu"]
# The words a customer might use for each answer. "" means they said nothing.
SPICE_WORDS = {
    "none": ["not spicy", "no chilli", "no spice"],
    "mild": ["mild", "a little spicy", "not too spicy"],
    "medium": ["medium", "medium spicy", ""],
    "hot": ["hot", "very spicy", "extra spicy"],
}
PLACE_WORDS = {
    True: ["to take away", "takeaway", "to go"],
    False: ["for here", "eating here", "eat in", ""],
}
COUNT_WORDS = {
    1: ["1", "one", "a"], 2: ["2", "two"], 3: ["3", "three"],
    4: ["4", "four"], 5: ["5", "five"], 6: ["6", "six"],
}
OPENINGS = ["", "", "Hi, ", "Hello! ", "Can I get ", "I'd like ", "I want "]
ENDINGS = ["", "", " please", ", please.", ". Thank you!", " thanks"]


def random_order():
    """One order, with every answer picked at random."""
    return {
        "dish": random.choice(DISHES),
        "protein": random.choice(PROTEINS),
        "spice": random.choice(list(SPICE_WORDS)),      # one of its four keys
        "count": random.randint(1, 6),
        "takeaway": random.choice([True, False]),
    }


def message_for(order):
    """One way a customer might write this order."""
    how_many = random.choice(COUNT_WORDS[order["count"]])
    dish = order["dish"]
    protein = order["protein"]
    food = random.choice([
        f"{how_many} {dish} with {protein}",
        f"{how_many} {dish} {protein}",
        f"{how_many} {protein} {dish}",
    ])
    extras = [
        random.choice(SPICE_WORDS[order["spice"]]),
        random.choice(PLACE_WORDS[order["takeaway"]]),
    ]
    random.shuffle(extras)          # spice first or place first
    parts = [food]
    for words in extras:
        if words:                   # "" adds nothing to the message
            parts.append(words)
    text = random.choice(OPENINGS) + ", ".join(parts) + random.choice(ENDINGS)
    if random.random() < 0.3:
        return text.lower()         # some people never type a capital letter
    return text[0].upper() + text[1:]


def write_file(name, examples):
    with open(name, "w", encoding="utf-8") as file:
        for example in examples:
            file.write(json.dumps(example) + "\n")
    print(f"wrote {name}: {len(examples)} examples")


def main():
    random.seed(0)                  # the same seed writes the same files every time
    examples = []
    used = set()
    while len(examples) < 450:
        order = random_order()
        message = message_for(order)
        if message in used:         # so that no message is in both files
            continue
        used.add(message)
        examples.append({"message": message, "order": order})
    write_file("train.jsonl", examples[:400])
    write_file("test_generated.jsonl", examples[400:])
    print()
    for example in examples[:4]:
        print(example["message"])
        print("   ", json.dumps(example["order"]))


if __name__ == "__main__":
    main()

🖨️ WHAT IT PRINTED

wrote train.jsonl: 400 examples
wrote test_generated.jsonl: 50 examples

can i get three pad see ew tofu, eat in, no chilli
    {"dish": "pad see ew", "protein": "tofu", "spice": "none", "count": 3, "takeaway": false}
Hi, 3 beef khao soi, takeaway, a little spicy, please.
    {"dish": "khao soi", "protein": "beef", "spice": "mild", "count": 3, "takeaway": true}
I'd like 5 tofu nam ngiao, hot, for here thanks
    {"dish": "nam ngiao", "protein": "tofu", "spice": "hot", "count": 5, "takeaway": false}
I want 6 pad see ew with beef, eating here, not too spicy
    {"dish": "pad see ew", "protein": "beef", "spice": "mild", "count": 6, "takeaway": false}

5 See an example the way the model sees it · ดูตัวอย่างแบบที่โมเดลเห็น · 像模型那样看一个样例

💻 RUN IT

venv\Scripts\python tokens.py

📄 NEW FILE · tokens.py

"""tokens.py - show one training example the way the model sees it.

    python tokens.py

It needs train.jsonl, which make_data.py writes.
"""
import json

from transformers import AutoTokenizer

MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
INSTRUCTION = "You take orders for a noodle shop. Reply with the order as JSON."


def main():
    tokenizer = AutoTokenizer.from_pretrained(MODEL)
    with open("train.jsonl", encoding="utf-8") as file:
        example = json.loads(file.readline())       # the first example only

    chat = [
        {"role": "system", "content": INSTRUCTION},
        {"role": "user", "content": example["message"]},
    ]
    question_text = tokenizer.apply_chat_template(
        chat, tokenize=False, add_generation_prompt=True)
    # The end-of-text token is part of the answer: it teaches the model to stop.
    answer_text = json.dumps(example["order"]) + tokenizer.eos_token
    print(question_text + answer_text)
    print()

    # The template has already written the special tokens, so add no more.
    question_tokens = tokenizer(question_text, add_special_tokens=False)["input_ids"]
    answer_tokens = tokenizer(answer_text, add_special_tokens=False)["input_ids"]
    print(f"{len(question_tokens)} tokens for the model to read. First five:")
    for token in question_tokens[:5]:
        piece = tokenizer.decode([token])
        # repr() puts quotes around the piece, so a space at its start can be seen.
        print(f"{token:8}  {repr(piece)}")
    print(f"{len(answer_tokens)} tokens it must learn to write. All of them:")
    for token in answer_tokens:
        piece = tokenizer.decode([token])
        print(f"{token:8}  {repr(piece)}")


if __name__ == "__main__":
    main()

🖨️ WHAT IT PRINTED

<|im_start|>system
You take orders for a noodle shop. Reply with the order as JSON.<|im_end|>
<|im_start|>user
can i get three pad see ew tofu, eat in, no chilli<|im_end|>
<|im_start|>assistant
{"dish": "pad see ew", "protein": "tofu", "spice": "none", "count": 3, "takeaway": false}<|im_end|>

47 tokens for the model to read. First five:
       1  '<|im_start|>'
    9690  'system'
     198  '\n'
    2683  'You'
    1188  ' take'
36 tokens it must learn to write. All of them:
   39428  '{"'
   21490  'dish'
    1799  '":'
     476  ' "'
   16384  'pad'
     963  ' see'
     297  ' e'
     103  'w'
    1002  '",'
     476  ' "'
   24089  'protein'
    1799  '":'
     476  ' "'
    1141  'to'
   12809  'fu'
    1002  '",'
     476  ' "'
    2401  'sp'
     569  'ice'
    1799  '":'
     476  ' "'
   19444  'none'
    1002  '",'
     476  ' "'
    4639  'count'
    1799  '":'
     216  ' '
      35  '3'
      28  ','
     476  ' "'
   21880  'take'
   11408  'away'
    1799  '":'
    7065  ' false'
     109  '}'
       2  '<|im_end|>'

6 train.py: one loop and one choice · train.py: ลูปเดียว กับทางเลือกเดียว · train.py：一个循环，一个选择

📄 NEW FILE · train.py

"""train.py - fine-tune the model on train.jsonl, in one of two ways.

    python train.py lora    train a small add-on; the model itself stays as it is
    python train.py full    train every number in the model

The result is saved in a folder: tuned_lora or tuned_full.
"""
import json
import sys
import time

import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
INSTRUCTION = "You take orders for a noodle shop. Reply with the order as JSON."
# How far one step may move a number. When every number moves, each must move less.
LEARNING_RATE = {"lora": 0.0002, "full": 0.00002}
WATCHED = "Two khao soi with chicken, not too spicy, to take away please."


def read_examples(name):
    """Every line of the file is one example, written as JSON."""
    examples = []
    with open(name, encoding="utf-8") as file:
        for line in file:
            examples.append(json.loads(line))
    return examples


def question_as_tokens(tokenizer, message):
    """The tokens the model reads: the instruction, the message, and "your turn"."""
    chat = [
        {"role": "system", "content": INSTRUCTION},
        {"role": "user", "content": message},
    ]
    question_text = tokenizer.apply_chat_template(
        chat, tokenize=False, add_generation_prompt=True)
    return tokenizer(question_text, add_special_tokens=False)["input_ids"]


def answer_as_tokens(tokenizer, order):
    """The tokens the model should write: the order, then the token that says stop."""
    answer_text = json.dumps(order) + tokenizer.eos_token
    return tokenizer(answer_text, add_special_tokens=False)["input_ids"]


def reply_to(model, tokenizer, message):
    """The model's reply to one message, on one line."""
    question_tokens = question_as_tokens(tokenizer, message)
    # The model takes rows of tokens. The extra brackets make this list one row.
    output = model.generate(
        torch.tensor([question_tokens]), max_new_tokens=50, do_sample=False)
    reply_tokens = output[0][len(question_tokens):]
    reply = tokenizer.decode(reply_tokens, skip_special_tokens=True)
    return " ".join(reply.split())


def count_numbers(tensors):
    """How many numbers these blocks of numbers hold between them."""
    total = 0
    for tensor in tensors:
        total = total + tensor.numel()
    return total


def prepare(model, kind):
    """Decide which numbers training may change. This is the whole difference."""
    if kind == "full":
        return model                    # as it arrived: every number may change
    config = LoraConfig(
        r=8,                            # the size of each add-on
        lora_alpha=16,                  # how loudly the add-ons speak
        target_modules=["q_proj", "v_proj"],    # the layers that get one
        task_type="CAUSAL_LM",          # a model that writes text
    )
    return get_peft_model(model, config)    # freezes the model, then adds them


def main():
    kind = sys.argv[1]                  # the word after train.py: "lora" or "full"
    # The add-ons start as random numbers. This makes them the same on every run.
    torch.manual_seed(0)
    tokenizer = AutoTokenizer.from_pretrained(MODEL)
    model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
    model = prepare(model, kind)

    trainable = []
    for tensor in model.parameters():   # a tensor is one block of numbers
        if tensor.requires_grad:        # False for a frozen block
            trainable.append(tensor)
    print(f"{kind}: training can change {count_numbers(trainable):,}"
          f" of {count_numbers(model.parameters()):,} numbers")

    optimizer = torch.optim.AdamW(trainable, lr=LEARNING_RATE[kind])
    examples = read_examples("train.jsonl")
    print(f"step   0              {reply_to(model, tokenizer, WATCHED)}")
    started = time.perf_counter()
    recent_losses = []
    step = 0
    for example in examples:
        step = step + 1
        question_tokens = question_as_tokens(tokenizer, example["message"])
        answer_tokens = answer_as_tokens(tokenizer, example["order"])
        example_tokens = torch.tensor([question_tokens + answer_tokens])
        # -100 means "do not grade this token". Only the answer is graded.
        graded_tokens = torch.tensor(
            [[-100] * len(question_tokens) + answer_tokens])

        # How wrong was the model on this example?
        loss = model(input_ids=example_tokens, labels=graded_tokens).loss
        # For each number that may change: which way would have been less wrong?
        loss.backward()
        # Move each of those numbers a little that way.
        optimizer.step()
        # Clear the directions, ready for the next example.
        optimizer.zero_grad()

        recent_losses.append(loss.item())   # .item() is the loss as a plain number
        if step % 25 == 0:                  # every 25 steps, say how it is going
            average = sum(recent_losses) / len(recent_losses)
            recent_losses = []
            print(f"step {step:3}  loss {average:.3f}"
                  f"  {reply_to(model, tokenizer, WATCHED)}")

    seconds = time.perf_counter() - started
    print(f"{step} steps in {seconds:.0f} seconds")
    model.save_pretrained(f"tuned_{kind}")
    print(f"saved in the folder tuned_{kind}")


if __name__ == "__main__":
    main()

7 The full way: every number can change · แบบเต็ม: ตัวเลขทุกตัวเปลี่ยนได้ · 全量的方式：每个数字都可以变

💻 RUN IT

venv\Scripts\python train.py full

🖨️ WHAT IT PRINTED

full: training can change 134,515,008 of 134,515,008 numbers
step   0              { "name": "Khao Soi with Chicken", "quantity": 2, "price": 10.99, "quantity_per_person": 2, "price_per_person":
step  25  loss 0.486  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step  50  loss 0.067  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step  75  loss 0.027  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 100  loss 0.018  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 125  loss 0.029  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 150  loss 0.010  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 175  loss 0.003  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 200  loss 0.001  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 225  loss 0.001  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 250  loss 0.001  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 275  loss 0.000  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 300  loss 0.000  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 325  loss 0.000  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 350  loss 0.000  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 375  loss 0.000  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 400  loss 0.000  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
400 steps in 431 seconds
saved in the folder tuned_full

8 The LoRA way: freeze the model, train a small add-on · แบบ LoRA: ล็อกโมเดลไว้ แล้วฝึกตัวเสริมขนาดเล็ก · LoRA 的方式：冻结模型，训练一个小小的附加模块

HuggingFaceTB/SmolLM2-135M-Instruct
134,515,008 numbers, in 30 blocks one after another

In each block, train.py puts an add-on on two matrices:
  q_proj: 576 x 576 = 331,776 numbers. Its add-on at r=8: 8 x 576 and 576 x 8 = 9,216 numbers.
  v_proj: 192 x 576 = 110,592 numbers. Its add-on at r=8: 8 x 576 and 192 x 8 = 6,144 numbers.
So each 1 in r is 1,920 numbers in a block, and 30 x 1,920 = 57,600 numbers in the model.

   r   numbers trained   by the formula   share of the model
   1            57,600           57,600               0.04 %
   2           115,200          115,200               0.09 %
   4           230,400          230,400               0.17 %
   8           460,800          460,800               0.34 %
  16           921,600          921,600               0.69 %
  32         1,843,200        1,843,200               1.37 %
  64         3,686,400        3,686,400               2.74 %

With an add-on on all seven kinds of matrix in a block, at r=8: 2,442,240 numbers, 1.82 % of the model.

💻 RUN IT

venv\Scripts\python train.py lora

🖨️ WHAT IT PRINTED

lora: training can change 460,800 of 134,975,808 numbers
step   0              { "name": "Khao Soi with Chicken", "quantity": 2, "price": 10.99, "quantity_per_person": 2, "price_per_person":
step  25  loss 1.634  { "name": "Khao Soi", "type": "Khao Soi", "quantity": 2, "price": 10.99, "delivery_time": "2 hours",
step  50  loss 0.827  {"dish": "khao soi", "protein": "chicken", "spice": "not too spicy", "toast": true, "to go": "tomato"}
step  75  loss 0.230  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 100  loss 0.077  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 125  loss 0.054  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": false}
step 150  loss 0.043  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 175  loss 0.032  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 200  loss 0.020  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 225  loss 0.021  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 250  loss 0.020  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 275  loss 0.016  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 300  loss 0.015  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 325  loss 0.009  {"dish": "khao soi", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
step 350  loss 0.008  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 375  loss 0.005  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
step 400  loss 0.006  {"dish": "khao soi", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": true}
400 steps in 195 seconds
saved in the folder tuned_lora

9 LoRA and full fine-tuning, side by side · LoRA กับการปรับจูนแบบเต็ม วางเทียบกัน · LoRA 与全量微调，放在一起比

10 Test both models on messages they never saw · ทดสอบโมเดลทั้งสองตัวกับข้อความที่มันไม่เคยเห็น · 用它们没见过的消息测试两个模型

💻 RUN IT

venv\Scripts\python score.py lora
venv\Scripts\python score.py full

📄 NEW FILE · score.py

"""score.py - test a fine-tuned model on messages it has never seen.

    python score.py lora
    python score.py full

It uses two files. test_generated.jsonl was written by the same program as the
training examples. test_handwritten.jsonl was written by a person. It prints
every order the model gets wrong, and then how many it got right.
"""
import json
import sys

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
INSTRUCTION = "You take orders for a noodle shop. Reply with the order as JSON."


def read_examples(name):
    """Every line of the file is one example, written as JSON."""
    examples = []
    with open(name, encoding="utf-8") as file:
        for line in file:
            examples.append(json.loads(line))
    return examples


def load_tuned(kind):
    """The model after training: a whole new model, or the old one plus its add-ons."""
    if kind == "full":
        return AutoModelForCausalLM.from_pretrained("tuned_full")
    model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
    return PeftModel.from_pretrained(model, "tuned_lora")


def reply_to(model, tokenizer, message):
    """The model's reply to one message, on one line."""
    chat = [
        {"role": "system", "content": INSTRUCTION},
        {"role": "user", "content": message},
    ]
    chat_tokens = tokenizer.apply_chat_template(
        chat, add_generation_prompt=True, return_tensors="pt")
    question_tokens = chat_tokens["input_ids"]
    output = model.generate(question_tokens, max_new_tokens=50, do_sample=False)
    # output is the question and then the reply. Keep the reply.
    reply_tokens = output[0][question_tokens.shape[1]:]
    reply = tokenizer.decode(reply_tokens, skip_special_tokens=True)
    return " ".join(reply.split())


def is_right(reply, order):
    """True when the reply is JSON and says exactly what the order says."""
    try:
        return json.loads(reply) == order
    except ValueError:              # json.loads could not read the reply as JSON
        return False


def main():
    kind = sys.argv[1]              # the word after score.py: "lora" or "full"
    tokenizer = AutoTokenizer.from_pretrained(MODEL)
    model = load_tuned(kind)
    for name in ["test_generated.jsonl", "test_handwritten.jsonl"]:
        tests = read_examples(name)
        right = 0
        for test in tests:
            reply = reply_to(model, tokenizer, test["message"])
            if is_right(reply, test["order"]):
                right = right + 1
            else:
                print("   ", test["message"])
                print("        wanted", json.dumps(test["order"]))
                print("        got   ", reply)
        print(f"{name}: {right} of {len(tests)} right")
        print()


if __name__ == "__main__":
    main()

    hi, two pad thai with pork, for here, no spice. thank you!
        wanted {"dish": "pad thai", "protein": "pork", "spice": "none", "count": 2, "takeaway": false}
        got    {"dish": "pad thai", "protein": "pork", "spice": "none", "count": 2, "takeaway": true}
test_generated.jsonl: 49 of 50 right

    four khao soi beef hot to go
        wanted {"dish": "khao soi", "protein": "beef", "spice": "hot", "count": 4, "takeaway": true}
        got    {"dish": "khao soi", "protein": "beef", "spice": "hot", "count": 4, "takeaway": false}
    2 pad thai chicken no chilli to go please
        wanted {"dish": "pad thai", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
        got    {"dish": "pad thai", "protein": "chicken", "spice": "medium", "count": 2, "takeaway": true}
    I'd love a bowl of nam ngiao with pork, as spicy as you can make it! I'm sitting at table 4.
        wanted {"dish": "nam ngiao", "protein": "pork", "spice": "hot", "count": 1, "takeaway": false}
        got    {"dish": "nam ngiao", "protein": "pork", "spice": "hot", "count": 4, "takeaway": false}
    khao soi x2, tofu, little bit spicy, take home
        wanted {"dish": "khao soi", "protein": "tofu", "spice": "mild", "count": 2, "takeaway": true}
        got    {"dish": "khao soi", "protein": "tofu", "spice": "little bit", "count": 2, "takeaway": true}
    Just one pad see ew with pork for me.
        wanted {"dish": "pad see ew", "protein": "pork", "spice": "medium", "count": 1, "takeaway": false}
        got    {"dish": "pad see ew", "protein": "pork", "spice": "medium", "count": 1, "takeaway": true}
    A couple of nam ngiao with beef, extra hot, for here.
        wanted {"dish": "nam ngiao", "protein": "beef", "spice": "hot", "count": 2, "takeaway": false}
        got    {"dish": "nam ngiao", "protein": "beef", "spice": "hot", "count": 1, "takeaway": false}
    Please pack 2 pad see ew with chicken. My kids will eat them, so not spicy.
        wanted {"dish": "pad see ew", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
        got    {"dish": "pad see ew", "protein": "chicken", "spice": "none", "count": 2, "takeaway": false}
    We are three people and each of us wants khao soi with pork, normal spicy, and we will eat at the shop.
        wanted {"dish": "khao soi", "protein": "pork", "spice": "medium", "count": 3, "takeaway": false}
        got    {"dish": "khao soi", "protein": "pork", "spice": "normal", "count": 3, "takeaway": false}
    kao soi with chiken, 1 bowl, medium
        wanted {"dish": "khao soi", "protein": "chicken", "spice": "medium", "count": 1, "takeaway": false}
        got    {"dish": "kao soi", "protein": "chimen", "spice": "medium", "count": 1, "takeaway": false}
    Yesterday's khao soi was great. Today I want one pad thai with beef, hot, to take away.
        wanted {"dish": "pad thai", "protein": "beef", "spice": "hot", "count": 1, "takeaway": true}
        got    {"dish": "khao soi", "protein": "beef", "spice": "hot", "count": 1, "takeaway": true}
    Is it possible to get khao soi with tofu instead of chicken? One bowl, medium, for here.
        wanted {"dish": "khao soi", "protein": "tofu", "spice": "medium", "count": 1, "takeaway": false}
        got    {"dish": "khao soi", "protein": "tofu", "spice": "medium", "count": 1, "takeaway": true}
test_handwritten.jsonl: 39 of 50 right

test_generated.jsonl: 50 of 50 right

    four khao soi beef hot to go
        wanted {"dish": "khao soi", "protein": "beef", "spice": "hot", "count": 4, "takeaway": true}
        got    {"dish": "khao soi", "protein": "beef", "spice": "hot", "count": 4, "takeaway": false}
    2 pad thai chicken no chilli to go please
        wanted {"dish": "pad thai", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
        got    {"dish": "pad thai", "protein": "chicken", "spice": "hot", "count": 2, "takeaway": true}
    I'd love a bowl of nam ngiao with pork, as spicy as you can make it! I'm sitting at table 4.
        wanted {"dish": "nam ngiao", "protein": "pork", "spice": "hot", "count": 1, "takeaway": false}
        got    {"dish": "nam ngiao", "protein": "pork", "spice": "hot", "count": 4, "takeaway": false}
    My friend and I each want a khao soi with chicken. Medium is fine. We are eating in.
        wanted {"dish": "khao soi", "protein": "chicken", "spice": "medium", "count": 2, "takeaway": false}
        got    {"dish": "khao soi", "protein": "chicken", "spice": "medium", "count": 1, "takeaway": false}
    Please pack 2 pad see ew with chicken. My kids will eat them, so not spicy.
        wanted {"dish": "pad see ew", "protein": "chicken", "spice": "none", "count": 2, "takeaway": true}
        got    {"dish": "pad see ew", "protein": "chicken", "spice": "none", "count": 2, "takeaway": false}
    We are three people and each of us wants khao soi with pork, normal spicy, and we will eat at the shop.
        wanted {"dish": "khao soi", "protein": "pork", "spice": "medium", "count": 3, "takeaway": false}
        got    {"dish": "khao soi", "protein": "pork", "spice": "normal", "count": 3, "takeaway": false}
    kao soi with chiken, 1 bowl, medium
        wanted {"dish": "khao soi", "protein": "chicken", "spice": "medium", "count": 1, "takeaway": false}
        got    {"dish": "kao soi", "protein": "chiken", "spice": "medium", "count": 1, "takeaway": false}
    Do you have nam ngiao? If yes, two with pork, please. Spice level: hot. I'll take them with me.
        wanted {"dish": "nam ngiao", "protein": "pork", "spice": "hot", "count": 2, "takeaway": true}
        got    {"dish": "nam ngiao", "protein": "pork", "spice": "hot", "count": 2, "takeaway": false}
    I am vegetarian, so tofu please. One pad see ew, not too hot.
        wanted {"dish": "pad see ew", "protein": "tofu", "spice": "mild", "count": 1, "takeaway": false}
        got    {"dish": "pad see ew", "protein": "tofu", "spice": "hot", "count": 1, "takeaway": false}
    One more nam ngiao with chicken for table 2, please, and make it mild.
        wanted {"dish": "nam ngiao", "protein": "chicken", "spice": "mild", "count": 1, "takeaway": false}
        got    {"dish": "nam ngiao", "protein": "chicken", "spice": "mild", "count": 2, "takeaway": false}
    One nam ngiao with tofu. Please don't make it spicy, and put it in a bag for me.
        wanted {"dish": "nam ngiao", "protein": "tofu", "spice": "none", "count": 1, "takeaway": true}
        got    {"dish": "nam ngiao", "protein": "tofu", "spice": "none", "count": 1, "takeaway": false}
    Pad thai with beef for one person, mild, and I'm taking it back to my hotel.
        wanted {"dish": "pad thai", "protein": "beef", "spice": "mild", "count": 1, "takeaway": true}
        got    {"dish": "pad thai", "protein": "beef", "spice": "mild", "count": 1, "takeaway": false}
test_handwritten.jsonl: 38 of 50 right

11 Use it inside a program · เอาไปใช้ในโปรแกรม · 在程序里使用它

💻 RUN IT

venv\Scripts\python use.py

📄 NEW FILE · use.py

"""use.py - use the fine-tuned model inside an ordinary program.

    python use.py

It needs the folder tuned_lora, which "python train.py lora" makes.
"""
import json

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
INSTRUCTION = "You take orders for a noodle shop. Reply with the order as JSON."
PRICES = {"khao soi": 60, "nam ngiao": 50, "pad thai": 60, "pad see ew": 55}  # baht
PROTEINS = ["chicken", "pork", "beef", "tofu"]
SPICES = ["none", "mild", "medium", "hot"]
MESSAGES = [
    "Hi, two khao soi with beef, very spicy, to go please.",
    "one pad thai tofu not spicy",
    "3 nam ngiao with pork, a tiny bit spicy, eat in",
    "What time do you close today?",
]


def reply_to(model, tokenizer, message):
    """The model's reply to one message, on one line."""
    chat = [
        {"role": "system", "content": INSTRUCTION},
        {"role": "user", "content": message},
    ]
    chat_tokens = tokenizer.apply_chat_template(
        chat, add_generation_prompt=True, return_tensors="pt")
    question_tokens = chat_tokens["input_ids"]
    output = model.generate(question_tokens, max_new_tokens=50, do_sample=False)
    # output is the question and then the reply. Keep the reply.
    reply_tokens = output[0][question_tokens.shape[1]:]
    reply = tokenizer.decode(reply_tokens, skip_special_tokens=True)
    return " ".join(reply.split())


def read_order(reply):
    """The order in a reply, as a dictionary. None if the reply is not a proper order.

    A model's reply is a guess. Check it before the program acts on it.
    """
    try:
        order = json.loads(reply)
    except ValueError:              # the reply is not JSON
        return None
    if not isinstance(order, dict):
        return None
    if order.get("dish") not in PRICES:
        return None
    if order.get("protein") not in PROTEINS:
        return None
    if order.get("spice") not in SPICES:
        return None
    if order.get("count") not in [1, 2, 3, 4, 5, 6]:
        return None
    if order.get("takeaway") not in [True, False]:
        return None
    return order


def main():
    tokenizer = AutoTokenizer.from_pretrained(MODEL)
    model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
    model = PeftModel.from_pretrained(model, "tuned_lora")
    model = model.merge_and_unload()    # fold the add-ons into the model's own numbers

    for message in MESSAGES:
        print(message)
        reply = reply_to(model, tokenizer, message)
        order = read_order(reply)
        if order is None:
            print("    NOT A PROPER ORDER, ask a person:", reply)
            continue
        price = PRICES[order["dish"]] * order["count"]
        if order["takeaway"]:
            place = "takeaway"
        else:
            place = "eat in"
        print(f"    {order['count']} x {order['dish']}, {order['protein']},"
              f" {order['spice']}, {place}: {price} baht")


if __name__ == "__main__":
    main()

🖨️ WHAT IT PRINTED

Hi, two khao soi with beef, very spicy, to go please.
    2 x khao soi, beef, hot, takeaway: 120 baht
one pad thai tofu not spicy
    1 x pad thai, tofu, none, eat in: 60 baht
3 nam ngiao with pork, a tiny bit spicy, eat in
    3 x nam ngiao, pork, mild, eat in: 150 baht
What time do you close today?
    NOT A PROPER ORDER, ask a person: {"dish": "noodle", "protein": "spam", "spice": "medium", "count": 1, "takeaway": false}

What is the capital of Thailand?
    no instruction    before  The capital of Thailand is Bangkok.
    no instruction    lora    The capital of Thailand is Bangkok. It is a vibrant city known for its rich history, cultural richness, and global appeal. Bangkok is also the largest city in Thailand, with a population of
    no instruction    full    The capital of Thailand is Bangkok.
    shop instruction  before  The capital of Thailand is Bangkok.
    shop instruction  lora    The capital of Thailand is Bangkok.
    shop instruction  full    {"dish": "khao noodle", "protein": "chicken", "spice": "medium", "count": 1, "takeaway": false}

Write one short sentence about elephants.
    no instruction    before  Elephants are majestic creatures with a heart full of life and a mind full of intelligence.
    no instruction    lora    An enthusiastic and curious animal, elephants are known for their massive size, powerful legs, and unique social behavior.
    no instruction    full    Elephants are majestic creatures with a heart full of life and a mind full of intelligence.
    shop instruction  before  Elephants are intelligent, social, and highly social animals that live in large herds. They are highly social animals and live in large herds, which can number in the hundreds. They are highly social animals
    shop instruction  lora    A majestic and social animal, the elephant is known for its long, slender body, powerful legs, and distinctive trunk-like appendage.
    shop instruction  full    Elephants are massive, intelligent creatures with a long, curved trunk and a thick, thick neck. They are known for their intelligence, social behavior, and impressive memory.

What time do you close today?
    no instruction    before  I'm sorry for any misunderstanding, but as a chatbot, I don't have the ability to access or control the time. I'm designed to provide information and assistance, not to provide specific time
    no instruction    lora    "SmolLM" is a helpful AI designed to provide information about the current time.
    no instruction    full    "Hughes, I'm closed today at 10:00 PM. Please check your calendar for any updates or special events."
    shop instruction  before  I close my restaurant at 10:00 PM.
    shop instruction  lora    {"dish": "noodle", "protein": "spam", "spice": "medium", "count": 1, "takeaway": false}
    shop instruction  full    {"dish": "noodle", "protein": "chicken", "spice": "medium", "count": 1, "takeaway": false}

12 A full fine-tune on a GPU: how decider was made · การปรับจูนแบบเต็มบนการ์ดจอ: decider ถูกสร้างมาอย่างไร · 在 GPU 上做全量微调：decider 是怎么做出来的

🖥️ ON A GPU · from the library's README · not run by us · จาก README ของไลบรารี · เราไม่ได้รันเอง · 来自这个库的 README · 我们没有运行过

uv venv --python 3.12 .venv312 && uv pip install -p .venv312/bin/python -e ".[serve,train]"
scripts/train.sh full
scripts/train.sh delta runs/some/model

decider-ai 1.8.2, torch 2.14.1+cpu

decider/train.py, 142 lines:
  line  56:  ap.add_argument("--model", default="Qwen/Qwen3.5-2B-Base")
  line  60:  ap.add_argument("--lr", type=float, default=1e-5)
  line  63:  ap.add_argument("--max_tokens", type=int, default=16384, help="tokens per micro-batch (B*T)")
  line  88:  model = DecisionModel(a.model).cuda()
  line  95:  opt = torch.optim.AdamW(model.parameters(), lr=a.lr, weight_decay=a.wd, betas=(0.9, 0.95))
  line 115:  b = {k: (v.cuda() if torch.is_tensor(v) else v) for k, v in b.items()}
decider/model.py, 69 lines:
  line   8:  def __init__(self, name, dtype=torch.bfloat16, grad_ckpt=True):
  line  15:  self.lm.gradient_checkpointing_enable()

From the library's README, about its models:
  Gemma-4-12B-it with a merged LoRA (rank 32) trained on 6,000 generated state-tracking decisions and 4,000 replayed rows of
  | decider-2b **v11** | Qwen3.5-2B-Base | 1.9B | 32k tokens | 0.752 (regression set) | [Mapika/decider-2b](https://huggingface.co/Mapika/decider-2b) |
  | decider-12b **v2** | Gemma-4-12B-it + merged state-tracking LoRA (rank 32) | 12B | 32k tokens | not measured on the regression set; JevBench public 

From the library's README, about training:
## Train your own

```bash
uv venv --python 3.12 .venv312 && uv pip install -p .venv312/bin/python -e ".[serve,train]"
scripts/train.sh full                       # datasets -> data/tasks.pkl -> data/mixture_full.pkl -> one epoch from Qwen3.5-2B-Base -> scripts/evaluate.sh
scripts/train.sh delta runs/some/model      # or: continue an existing decider checkpoint on the new formats + a replay sample
```

The full recipe and the data builders are in this repository: `decider/data/` downloads and converts about 95 public datasets
and assembles the mixture (`decider/data/mixture.py` lists every component with its size), `decider/train.py` is the
fine-tune, `scripts/evaluate.sh` scores it. One epoch is 1.47M examples and 455M tokens, 5.3 h on a GH200 plus 45 min of
evaluation, and it reproduces the released supervised weights: it matches v9 on the 94-task set (in-task 0.809 against 0.812,
held-out 0.739 against 0.741) and every probe family within noise, with a fitted temperature of 1.03 instead of 1.36.

The RL stage that turns v8 into v10 ([docs/RL.md](https://github.com/Mapika/decider/blob/main/docs/RL.md)) needs a live Chrome with MiniWoB++, the exact game environments
and the training loop of a separate research repository; it is not in this package yet.

scripts/train.sh, read from https://raw.githubusercontent.com/Mapika/decider/main/scripts/train.sh
  line  2:  # The reference recipe, end to end.  One GH200 / H100-class GPU; PY points at the project venv.
  line  3:  #   scripts/train.sh full     Qwen3.5-2B-Base -> decider, one epoch over the full mixture (about 1.3M examples)
  line  4:  #   scripts/train.sh delta M  continue from an existing decider checkpoint M on the new formats + a replay sample
  line 10:  LR=1e-5; [ "$MODE" = delta ] && LR=8e-6

python -m decider.train --help
usage: train.py [-h] [--model MODEL] [--data DATA] --out OUT [--epochs EPOCHS]
                [--lr LR] [--wd WD] [--warmup WARMUP]
                [--max_tokens MAX_TOKENS] [--accum ACCUM] [--max_ctx MAX_CTX]
                [--brier_w BRIER_W] [--label_smooth LABEL_SMOOTH]
                [--eval_every EVAL_EVERY] [--eval_limit EVAL_LIMIT]
                [--train_cap TRAIN_CAP] [--seed SEED] [--resample_every_epoch]
                [--none_prob NONE_PROB]
                [--schema_first_prob SCHEMA_FIRST_PROB]
                [--max_options MAX_OPTIONS]

options:
  -h, --help            show this help message and exit
  --model MODEL
  --data DATA
  --out OUT
  --epochs EPOCHS
  --lr LR
  --wd WD
  --warmup WARMUP
  --max_tokens MAX_TOKENS
                        tokens per micro-batch (B*T)
  --accum ACCUM
  --max_ctx MAX_CTX
  --brier_w BRIER_W
  --label_smooth LABEL_SMOOTH
  --eval_every EVAL_EVERY
  --eval_limit EVAL_LIMIT
  --train_cap TRAIN_CAP
                        cap total train examples (debug)
  --seed SEED
  --resample_every_epoch
  --none_prob NONE_PROB
                        prob. of none-of-the-above augmentation per question
  --schema_first_prob SCHEMA_FIRST_PROB
                        share of examples rendered questions-first (cacheable
                        schema prefix)
  --max_options MAX_OPTIONS
                        options kept per question (10 = original protocol; 255
                        when the data set is already sub-sampled, e.g.
                        tasks_v6_delta)

This computer: torch.cuda.is_available() is False.
train.py moves the model to a CUDA graphics card with .cuda(), so it cannot train here.