🍱 BentoML with Python: turn a function into a web service · BentoML ด้วย Python: เปลี่ยนฟังก์ชันให้เป็นเว็บเซอร์วิส · 用 Python 学 BentoML：把函数变成网络服务

🇬🇧 English

Make a new folder for this page and open a terminal in it. The first command below makes a virtual environment. That is a separate copy of Python with its own libraries, kept in a folder named venv. It keeps this project apart from your other projects.

The second command installs BentoML into it. BentoML brings other libraries with it. One of them is httpx, which step 5 uses.

On a Mac, write venv/bin/ wherever this page writes venv\Scripts\.

🇬🇧 English

Save this file in your folder as price.py. It has three parts.

MENU is a dictionary. A dictionary holds pairs: a key, and the value that belongs to it. Here each key is the name of a dish, and its value is the price of one bowl. MENU["dumplings"] gives 40.

price is a function. It takes two values, dish and bowls. It looks up the price of the dish, multiplies it by the number of bowls, and gives the result back with return.

The last two lines call the function and print what it returns.

🇬🇧 English

Run it:

🇬🇧 English

Two bowls of beef noodles cost 120, and three bowls of fried rice cost 150. The function works.

But look at who can use it. Only the lines inside price.py can call price. When the file has finished, the function is gone. A phone app cannot ask it for a price. Part 2 changes that.

🇬🇧 English

Save this file as service.py. Look at the last line first. It is the same line as in price.py. The work has not changed. Everything new is around it, and the table under the file explains each new line.

🇬🇧 English

There is nothing to run yet. This file has no print and it calls nothing. It only describes a service. The next step starts it.

🇬🇧 English

Type this command in your terminal. The last word has two halves with a colon between them. service is the file service.py, without the .py. Shop is the class inside it.

🇬🇧 English

The first line gives the address of your service: http://localhost:3000. localhost means this computer, and 3000 is the port. The second line says that the class Shop is ready.

Then nothing more happens, and the terminal does not give you a new line to type on. That is correct. The service is running, and it is waiting for questions. Leave this terminal open. To stop the service later, click in this terminal and press Ctrl+C.

Now open http://localhost:3000 in your web browser. BentoML made this page for you. It lists what your service can do. Find the line POST /price and click it. Click Try it out. You see a box with "dish": "string" and "bowls": 0 in it. Change string to dumplings and 0 to 2, and click Execute. The answer, 80, appears lower down.

Look at the first terminal again. It has a new line for the request you just sent. The service writes one line for every request.

🇬🇧 English

A person can use the browser page. A program needs code. Open a second terminal in the same folder, and save this file as ask.py.

bentoml.SyncHTTPClient makes a client. A client is an object that knows the address of a service and can send requests to it. We keep it in the name shop.

shop.price(dish="beef noodles", bowls=2) looks like a call to a method, and you write it the same way. But this time the question leaves your program. It travels to the service in the first terminal, the method price runs there, and the answer travels back.

🇬🇧 English

These are the same two answers as in step 2. But ask.py has no menu and no function price in it. Both are in the other program.

The program that asks does not need BentoML at all. A request to a service is a small piece of JSON, sent to an address. The next file sends one with httpx, a library for web requests. The address ends in /price: the name of your method became part of the address. The JSON holds the two values, under the names dish and bowls.

🇬🇧 English

The first line is the status code. 200 means that the request went well. The second line is the answer: six bowls of dumplings cost 240.

This is why a service is useful. Any program that can send a web request can ask your Python function for a price. It can be a web page written in JavaScript or an app on a phone.

🇬🇧 English

A service is asked by programs that other people wrote, and some of them will ask badly. This file sends two wrong requests. In the first, the number of bowls is the word "five". In the second, the dish is "pizza", which is not on the menu.

🇬🇧 English

The first reply has the status code 400. A code that begins with 4 means that the request was wrong. BentoML read the hint bowls: int, saw that "five" is not a whole number, and sent the request back. The reply says where the mistake is, "loc":["bowls"], and what it is: Input should be a valid integer. Your method did not run at all. You wrote no code for this check. The type hint did it.

The second reply has the status code 500. A code that begins with 5 means that the service itself failed. This time both types were right, so BentoML let the request in and your method ran. Then MENU["pizza"] failed, because the dictionary has no such key. The program that asked is told very little: An unexpected error has occurred.

The full story is in the first terminal, where the service runs. It printed a long error there. These are its last three lines:

🇬🇧 English

A KeyError is what Python raises when a dictionary does not have the key you asked for. The service did not stop. It is still running, and it will answer the next request.

So a type hint can check the kind of a value. It cannot know that the shop sells no pizza. That check is your job, and step 7 adds it.

🇬🇧 English

First stop the old service: click in the first terminal and press Ctrl+C. Then save the next two files. menu.json is the menu as a file of its own, so that the shop's owner can change a price and never touch the code.

🇬🇧 English

This class has three methods, and three things in it are new.

__init__ is a special method. Its name has two underscores on each side. Python runs it one time, when the Shop is made, and BentoML makes the Shop when the service starts. Here it opens menu.json, and json.load turns the text of the file into a dictionary. The line self.menu = ... keeps that dictionary inside the shop. Any other method can then read it as self.menu. This is what self is for: it is how the methods of one class share data.

dishes is a second API. It takes no values, and it returns the names of the dishes. list(self.menu) makes a list of the dictionary's keys. The hint -> list[str] says that the result is a list of pieces of text. Two methods with @bentoml.api make two addresses: /dishes and /price.

raise stops a method and reports an error. In price, the if checks that the dish is on the menu. If it is not, raise sends back an error that BentoML has ready, NotFound, with a message that we wrote ourselves. The last line of the method then does not run.

Start the new service. This time the word before the colon is shop, because the file is shop.py.

🇬🇧 English

The middle line is the print inside __init__. It appears one time, when the service starts. It will not appear again, however many requests come in.

In the second terminal, save ask_shop.py and run it.

🇬🇧 English

The first line is the reply of dishes, and the second is the price of four bowls of dumplings. The last two lines are the request for pizza. The status code is now 404 and not 500, and the reply holds our own message. A program that asks for pizza can now tell its user what went wrong.

🇬🇧 English

In step 7 the thing that was loaded one time was a menu. Now you put a real AI model in the same place. The model is named decider-2b. It does not write text. You give it a text and a question, and it gives back a number that says how sure it is. The page Decider explains the model. Here you only say hello to it.

Install its library into the same venv:

🇬🇧 English

The shape is the one you wrote in step 7: load in __init__, keep in self, answer in a method.

__init__ loads the model one time and keeps it in self.model. If the model were loaded inside yes_or_no, every request would have to wait for the loading.

yes_or_no takes two pieces of text: the text to read, and a question about it. The hint -> float says that it returns a number with a decimal point.

The method makes a dictionary with one question in it, named "answer". "noul" is the model's own name for a yes-or-no question. Then self.model.system_one asks the model. The result is a dictionary, and the last line takes the number out of it. A number near 1 means yes. A number near 0 means no.

traffic={"timeout": 300} is new. BentoML gives up on a request after 60 seconds. A model's first answer can take longer than that, so this line lets a request take 300 seconds.

Stop the old service with Ctrl+C and start this one.

🇬🇧 English

Wait for the last line, Service Judge initialized. With the menu that line came after 1 second. With the model it came after 27 seconds, and after 84 seconds the first time we started it. Your terminal also shows a moving bar while the model loads. The bar is left out here.

The two lines with Warning come from the library that downloads the model. You can ignore them. You need no account.

In the second terminal, save ask_judge.py and run it.

🇬🇧 English

The order says Two bowls of beef noodles, please. No chilli. Does the customer order food? The model says 0.908, which is near 1: yes. Does the customer want it spicy? The model says 0.0085, which is near 0: no. The model read No chilli and understood it.

timeout=300 in ask_judge.py is there for a reason. A client waits 30 seconds for an answer and then stops with an error. Our very first request to this service took 59 seconds, so our first try failed. After that first one, each answer took about 3 seconds.

🇬🇧 English

A large language model, or LLM, is a model that writes text. llama3.2 is a small one. Local means that it runs on your own computer, and your words do not leave it.

Ollama is a service too. It stays on in the background and waits at port 11434. So this time your service loads no model. It passes each question on to another service, and passes the answer back. That is very common: services ask other services.

🇬🇧 English

This class has no __init__, because it has nothing to load. Ollama keeps the model.

say takes a prompt, which is the text you give to an LLM. It builds a dictionary named request with three things in it: which model to use, the prompt, and "stream": False. That last one asks Ollama to send the whole answer at once, and not word by word.

httpx.post sends the dictionary to Ollama as JSON. You used the same call in step 5, from the other side. timeout=120 waits up to 120 seconds, because an LLM is slow. reply.json() turns Ollama's reply back into a dictionary, and the text of the answer is under the key "response".

Stop the old service and start this one. Then save ask_talker.py and run it in the second terminal.

🇬🇧 English

Your answers will be different from ours. An LLM does not write the same sentence every time. We ran this file twice, and the two greetings were not the same.

Three programs worked together here. ask_talker.py asked your service. Your service asked Ollama. Ollama asked the model. Each one knows only the address of the next.

Now compare the two tutorials. The model in tutorial A gives a number, and a program can use a number at once: if answer > 0.5. The model in tutorial B gives a sentence, which is good for a person to read. BentoML serves both in the same way.

🇬🇧 English

So far a program asked your service, and the answer was JSON. A person wants a web page. A BentoML service can send one, and it can read what the person types into a form.

This tutorial needs nothing new: no model, and nothing to install. It uses Starlette, a small web library that came with BentoML in step 1. You need to know a little HTML: a form, an input and a button.

🇬🇧 English

The file is longer than the others, but most of it is the web page itself.

FORM is one long piece of text. Three quotation marks, """, let a piece of text run over many lines. It holds an HTML page with one form. Three things in the form matter. action="/order" is the address that the form is sent to. method="post" is the way it is sent. And each field has a name: name="dish" and name="bowls".

order is an ordinary API, like price in step 3. Look at the values it takes: dish and bowls. They are the names of the fields in the form. That is all the wiring there is. BentoML reads the form and hands each field to the value with the same name.

A form sends everything as text, so the browser sends the text "3". The hint bowls: int turns it into the number 3. If it is not a number, BentoML sends the request back, as in step 6.

HTML is a type hint with a note fixed to it. It says: this is a str, and the browser should show it as a web page. The method builds a small page from pieces of text and returns it.

form_page is there for one reason. A browser opens a page with a request named GET, and an API of BentoML answers only requests named POST. So the page with the form comes from a plain function, and the line @bentoml.asgi_app fixes that function to the service at the address /shop. Starlette and Route are the tools that do it.

Stop the old service and start this one.

🇬🇧 English

Open http://localhost:3000/shop/ in your browser, with the slash at the end. You see the form. Choose a dish, type a number of bowls, and press the button. The browser goes to /order and shows a page with the price. The link Ask again takes you back to the form.

Now type the word two in the box and press the button. The browser shows the same message as in step 6, with the status code 400. The type hint checks a form in the same way as it checks JSON.

The next file does from Python what your browser just did. Save it and run it in the second terminal.

🇬🇧 English

The first line is the page with the form: the status code 200, and text/html, which is how a reply says that it is a web page. The second line is the reply to the form, and the third is the page that order built.

One word is different from step 5. There the file sent JSON with json=. Here it sends a form with data=. The API takes both.

🇬🇧 English

localhost is an address that only your own computer can use. A phone in the same room cannot open http://localhost:3000. On the phone, localhost means the phone.

To reach your service, another device needs your computer's address on the local network. The local network is all the devices on the same Wi-Fi. Find your address with this command, and look for the line IPv4 Address under your Wi-Fi. It looks like 192.168.1.20.

🇬🇧 English

Start the service of tutorial C in the usual way, with bentoml serve webshop:Shop. Then, on another device on the same Wi-Fi, open http://192.168.1.20:3000/shop/, with your own address in place of this one.

We tested it from a second computer. It runs Linux and has no screen, so we did not use a browser. We used curl, a program that sends web requests from a terminal. -d sends a form.

🇬🇧 English

To keep the service to your own computer, add --host 127.0.0.1 to the command. 127.0.0.1 is localhost written as a number.

🇬🇧 English

Now the first line says listening on http://127.0.0.1:3000. Your own computer still gets its answers. The second computer sent the same command as before, and this time it waited for eight seconds and gave up:

🇬🇧 English

A local address stops at your router, the box that gives you Wi-Fi. A friend in another house cannot use 192.168.1.20. Tutorial E is about that.

🇬🇧 English

First, your own devices. Tailscale gives each device an address that begins with 100. Ask your computer for its own:

🇬🇧 English

Start the service in the usual way, without --host. On your other device, use the address that begins with 100, as in http://100.x.y.z:3000/shop/. This is meant to work when the two devices are on different networks, such as a laptop at home and a phone outside. Only devices that are signed in to your account can do it.

We sent the form from a second device in our Tailscale network to the laptop's address that begins with 100. Both devices were in the same building, so we did not test it from far away.

🇬🇧 English

Then, the whole internet. Tailscale has a feature named Funnel for this. One command gives your service a public address:

🇬🇧 English

3000 is the port of your service. Tailscale prints an address that looks like https://your-computer.your-tailnet.ts.net. Anyone in the world who has that address can open it, and they do not need Tailscale. Put /shop/ at the end to get the form.

Leave the command running. Ctrl+C stops it and closes the address. The first time, Tailscale gives you a link to a web page where you switch Funnel on for your account.

We did not run this command for this page. What we say about it is from Tailscale's own documentation and from the help that the command prints.

🇬🇧 English

Your service runs on your computer, in your folder, with your venv. To run it on another computer, that computer needs the same files and the same libraries. BentoML can pack all of that into one box. It calls the box a Bento, after the Japanese lunch box that holds a full meal.

Save this file as bentofile.yaml. It has one line, and it says which service to pack: the class Shop in the file shop.py.

🇬🇧 English

Then build the Bento. You can do this in the second terminal. The service may go on running in the first one.

🇬🇧 English

The line that matters is Successfully built Bento. After it comes the tag of your Bento. A tag has two halves with a colon between them: the name, shop, and a version that BentoML made up. Every build gets a new version, so your letters will be different from ours.

The three lines at the top say that BentoML wrote down which libraries the service needs, with their exact versions. It writes that list for Linux, because most services run on Linux computers. The lines under Next steps are about putting the Bento online. You do not need them for this page.

The next command lists the Bentos on your computer.

🇬🇧 English

BentoML keeps its Bentos in a folder named bentoml in your home folder. Our Bento held three things: a copy of the files in our folder, without the venv folder; the list of libraries; and a description of the two APIs.

A Bento can be served by its tag. Stop the service in the first terminal, and start it again from the box:

🇬🇧 English

latest means the newest version. This service does not read shop.py from your folder. It reads the copy inside the box. We tested that by starting it from a different folder, and it answered.

The usual next step is bentoml containerize. It turns a Bento into a container image, which is a package that a server can run by itself. That command needs a program named Docker, and we did not run it for this page.