CHECKS OF RAG WITH PYTHON
krueng.ai/rag_llm.html
The page gives numbers: how often the search finds the right passage, how many
of 50 test questions a small model answers right with no handbook, with the
passages found and with the whole handbook, how many tokens each way makes the
model read, and how long an answer takes. None of those numbers was typed in
from memory. Each one was printed by one of the page's own files or by a script
in this folder, and what each printed is saved beside it as a *_output.txt
file. The page shows those files. Run them again and compare.
WHAT YOU NEED
1. The page's files in one folder. Every code block on the page has a Download
button, and so have the handbook and the test.
embed.py search.py prompt.py ask.py answer.py
handbook.txt questions.jsonl
2. These scripts in a folder named "checks" inside that folder:
your folder/
embed.py search.py ... handbook.txt questions.jsonl
checks/
README.txt change_a_fact.py sizes.py ... practice_questions.jsonl
Each script here looks for the page's files one folder up.
3. A Python with the three libraries, as in step 1 of the page:
python -m venv venv
venv\Scripts\python -m pip install torch transformers sentence-transformers
The first runs download two models, 91.6 MB and 712.0 MB.
4. About 2 GB of free disk, and 3 GB of free memory. readers.py and
bigger_reader.py need more of both: see below.
THE PAGE'S OWN FILES, IN ORDER
Run them in the order of the page, about 10 minutes on our laptop:
venv\Scripts\python embed.py
venv\Scripts\python search.py
venv\Scripts\python prompt.py
venv\Scripts\python ask.py "How much is a bowl of khao soi?"
venv\Scripts\python ask.py "Can I pay by card?"
venv\Scripts\python answer.py nothing
venv\Scripts\python answer.py found
venv\Scripts\python answer.py everything
What each printed for us is saved one folder up as _output.txt:
embed_output.txt, ask_card_output.txt, answer_found_output.txt and so on.
timings.txt, beside them, has the seconds each step took, the most memory it
held, the clock times at which it began and ended, and the word "mains" or
"battery".
THE CHECKS
change_a_fact.py about 2 minutes
Copies handbook.txt into a folder of its own and runs the page's ask.py
there four times: a price question, the same after one line was edited, a
question the handbook cannot answer, and the same after one passage was
added. The page's handbook is never touched. It also times turning the
passages into numbers again.
-> change_a_fact_output.txt
sizes.py after the page's files
Adds up the two models as downloaded and the Python libraries, counts the
numbers in each model, and counts the handbook: passages, words, tokens.
-> sizes_output.txt
best_scores.py no language model
Prints the score of the best passage for every test question, the 40 that
the handbook answers and the 10 that it does not, and looks for a score
that would separate the two kinds. There is none.
-> best_scores_output.txt
offline.py after the page's files
Runs the page's ask.py with HF_HUB_OFFLINE=1 and with every connection to
another computer refused, and counts the attempts to connect.
-> offline_output.txt
model_cards.py needs the internet
Measures nothing. Prints the lines of the two models' cards on Hugging Face
that the page's sentences about them rest on (maker, licence, languages,
training), and what transformers says about the documents= argument.
-> model_cards_output.txt
readers.py about 6 GB of downloads, 8 GB of memory
How the reader was chosen. Asks five small models the 20 questions of
practice_questions.jsonl, each with the three passages the search found,
using answer.py's own functions and marking. A model whose chat template
takes documents is asked through it and also with the passages pasted into
the instruction by hand; the others only by hand. It first proves, for each
model, whether a prompt made with documents= has the passages in it.
-> readers_output.txt
bigger_reader.py 3.3 GB of download, 8 GB of memory
The page's answer.py, in its "found" way, with one thing changed: the
reader is ibm-granite/granite-4.0-1b.
-> bigger_reader_output.txt
THE SITE'S OWN PAGE TOOLS
Five tools are kept with the site's source, in python/video_gen, and not on the
site. They work for any tutorial page; page_config.py in this folder tells them
about this one: its steps, where each of its numbers comes from, and what its
sentences say about the outputs.
run_page_files.py rag_llm.html [--checks]
Runs the page's files in order, and with --checks the scripts above. It
saves what each printed and writes timings.txt.
fill_code_blocks.py rag_llm.html [--check]
Puts every file into its code or output box on the page, or compares.
page_claims.py rag_llm.html [--write]
A measured number inside a sentence of the page is marked
. This compares every marked number with the output
files, and with --write puts the files' numbers into the page, in every
language at once. It also looks up everything the sentences say about the
outputs: what the model replied to which question, and how it was marked.
It reports the claims that no longer hold. Step 9 of the page quotes
numbers of the sister page, fine_tune_llm.html; those are read from that
page's own output files.
three_languages.py rag_llm.html
Takes the page's text three pieces at a time, English, Thai, Chinese, and
checks that all three carry the same numbers, the same words of code, the
same links and the same number of paragraphs.
slept.py rag_llm.html
Reads the clock times in timings.txt and asks Windows when the computer
entered or left standby. A step with such a moment inside it was frozen for
a while, and its seconds cannot be believed. Windows only.
HOW THE TEST WAS KEPT FAIR
- handbook.txt and the 50 questions of questions.jsonl, with their accepted
answers, were written for this page by one author before any model had read
the handbook. The shop and every fact in it are invented.
- Two models were then tried on the test itself, with the passages pasted into
the instruction: SmolLM2-135M-Instruct, the model of the fine-tuning page,
and Qwen2.5-0.5B-Instruct. Both read too badly to use. That is when
practice_questions.jsonl was written: 20 other questions about the same
handbook.
- The reader of the page, the way of handing it the passages (documents=) and
the number of passages (three) were settled with the practice questions
only. readers.py is that comparison. The reader met the test after it had
been chosen.
- One thing was changed after the test had been run: the marking. The first
rule counted a reply as right if an accepted answer stood anywhere in it, and
knew eleven phrases of refusal. Reading every reply showed two faults. "15"
was found inside "150,000", and refusals worded "I can't assist with that"
or "I am not able to answer" were marked wrong. The rule in answer.py now
refuses an answer that is a piece of a longer number, and knows more
wordings of refusal. With the first rule the three runs scored 3, 31 and 29
of 40, and 0, 1 and 0 refusals of 10. With the repaired rule the same
replies score 2, 31 and 29, and 5, 3 and 3.
- The key names in the two question files were changed once, from "section"
and "answers" to "answer_in" and "accept". No question and no accepted
answer was changed.
- The marking is still rough. Read by a person, a few marks are arguable: in
the "everything" run, "You can pay with cash, PromptPay, or QR code." is
marked right for a customer who has no cash, and in the "found" run "You can
pay with cash or use the QR code at the counter." is marked wrong.
WHAT WE RAN, AND WHEN
Everything was run on one laptop: Windows 11 Home 25H2 (build 26200.9457), an
Intel Core Ultra 7 155H with 24 GB of memory and no graphics card that a model
could use, Python 3.12.10, PyTorch 2.14.1 on 16 threads, transformers 5.19.0
and sentence-transformers 6.1.0.
The outputs on the page are from 8 October 2026, 07:06 to 07:32, on mains
power: the page's files and then change_a_fact, sizes, best_scores, offline,
model_cards, readers and bigger_reader, one after another with nothing else
running. The laptop was awake through every step.
They were run in a virtual environment made that morning with exactly the two
commands of step 1 of the page. It took 1,040.8 MB. psutil, with which the
runner measures memory, was then added, and that is the 1,041.7 MB that
sizes.py reports.
Earlier the same night the same files had been tried in the virtual
environment of the fine-tuning page, with sentence-transformers added to it.
The three runs of answer.py printed the same 150 replies there, and readers.py
the same scores.
After the run on mains, three names inside the page's files were changed and
nothing else: in best_places, "order" became "places"; in search.py, "first"
became "came_first" and "mark" became "arrow". One comment in answer.py was
reworded. Every file and every check was then run again, 08:34 to 09:58 the
same morning. That run was mostly on battery, and the laptop went to standby
in the middle of one step, so its times are not used. What it printed is: all
15 output files were the same as before, line for line, apart from the
seconds. The outputs and the times kept here are those of the run on mains.
Nothing was run on a Mac, on Linux, or on a graphics card. No fact was trained
into a model: what step 8 of the page says about that is reasoning, and the
page says so.
THINGS TO KNOW
- Your times will differ. They depend on your processor and on what else it is
doing. Sizes and counts should be the same as ours.
- do_sample=False makes every reply the same on every run on one computer. On
another kind of processor the last digits of the arithmetic can differ, and
then a reply, a score, or a place in the search can differ too.
- answer.py prints its own seconds for one answer. They do not include loading
the models. timings.txt has the whole run of each file.
- ask.py loads both models every time it is called, which is most of its
18 seconds. A program that answers many questions loads them once, as
answer.py does.