🕶️ Web Scraping with CloakBrowser (การแปลภาษาไทยกำลังจะมา) · 用 CloakBrowser 做网页抓取

This page teaches web scraping in Python by solving one concrete problem: "What does an MSI EdgeXpert mini-PC cost, and where is it cheapest?" We'll try to answer it with the usual tools, watch them fail, and then reach for CloakBrowser — a stealth browser that loads pages the way a real person's browser does, so the data is actually there to read.

Everything here is built on real, working code. The krueng.ai site has a rental-finder report that pulls listings from FazWaz, DDproperty, Hipflat and RentHub — all sites that flatly refuse a plain Python request. We'll walk through that pipeline, then build a fresh price-comparison bot for the EdgeXpert across Amazon, Shopee, and Lazada.

If you're new to Python, read Python Basics first — this page assumes you can read a function, an import, and an f-string.

The simplest way to fetch a web page in Python is the requests library:

import requests

r = requests.get("https://www.fazwaz.com/condo-for-rent/thailand/chiang-mai")
print(r.status_code)   # hoped-for 200 ... but you get 403
print(r.text[:300])    # "Just a moment..." — a Cloudflare challenge page

You asked for a list of condos and got back 403 Forbidden and a page that says "Checking your browser…". You never saw a single listing. Here's why.

Big commercial sites sit behind a bot-protection service — most often Cloudflare. Before letting you see the real page, it scores how "human" your request looks. A bare requests call fails almost every check:

• It sends a giveaway User-Agent like python-requests/2.31.0.
• It runs no JavaScript, so the challenge that real browsers solve silently never gets solved.
• It has no browser fingerprint — no canvas, no WebGL, no fonts, no screen size. A real Chrome leaks hundreds of these signals; requests leaks none.

So the firewall concludes "this is a script" and serves the challenge page instead of the data. curl, wget, and even the WebFetch tool inside AI assistants fail for the exact same reasons.

CloakBrowser is a Python package that drives a patched, stealthy build of Chromium. Because it's a real browser, it runs the site's JavaScript and presents a complete, believable fingerprint — so the bot check passes and the page loads normally. You then read the finished HTML out of the browser.

It matters where the stealth lives. Many anti-detection tricks inject JavaScript into the page to lie about the browser — but that injection is itself detectable. CloakBrowser instead forges the canvas, WebGL, audio, font, GPU and WebRTC fingerprints in the browser's C++ core, below the JavaScript layer, where the bot check can't tell the difference.

If you've used Selenium or Playwright before, the API will feel familiar — CloakBrowser is shaped almost identically. The difference is purely the stealth: a default Playwright Chromium often gets flagged; CloakBrowser usually doesn't.

One command. On first run it downloads its patched Chromium binary (~200 MB), so the first launch is slow; after that it's cached.

pip install cloakbrowser

If you read Python Basics §8, do this inside a virtual environment so it doesn't pollute your system Python:

python -m venv .venv
.venv\Scripts\activate          # Windows PowerShell
pip install cloakbrowser

# or, with uv (much faster):
uv venv
uv pip install cloakbrowser

Five lines. This is the whole shape of a CloakBrowser scrape:

from cloakbrowser import launch

browser = launch()                 # start the stealth Chromium
page = browser.new_page()          # open a tab
page.goto("https://example.com", timeout=60000, wait_until="domcontentloaded")
page.wait_for_timeout(4000)        # let JavaScript-rendered content settle
html = page.content()              # the fully-rendered HTML, as a string
browser.close()                    # always close — it's a real process

Read it line by line:

• launch() starts the browser process and hands you a browser object.
• browser.new_page() opens a tab. You work through this page object.
• page.goto(url, ...) navigates. timeout=60000 is 60 seconds (in milliseconds). wait_until="domcontentloaded" means "return once the HTML document is parsed" rather than waiting for every last image.
• page.wait_for_timeout(4000) pauses 4 seconds. Modern sites render their content after the initial HTML, with JavaScript — this gives that a moment to finish so the listings are actually present when you grab the page.
• page.content() returns the current HTML as a Python string. This is the prize: the same HTML a human would see, ready to parse.
• browser.close() shuts the browser down. Skip it and you leak a running Chromium process.

Starting a browser is expensive (it's a whole Chromium process). When you scrape several URLs, launch once and reuse it — but open a fresh new_page() for each URL. Reusing the same page across navigations triggers "navigation is changing" races.

from cloakbrowser import launch
from pathlib import Path

URLS = [
    "https://www.fazwaz.com/condo-for-rent/thailand/chiang-mai/mueang-chiang-mai/tha-sala",
    "https://www.fazwaz.com/condo-for-rent/thailand/chiang-mai/mueang-chiang-mai/wat-ket",
]

browser = launch()
try:
    for i, url in enumerate(URLS):
        page = browser.new_page()          # fresh tab per URL
        try:
            if i:
                page.wait_for_timeout(2000)   # courtesy gap between same-site hits
            page.goto(url, timeout=60000, wait_until="domcontentloaded")
            page.wait_for_timeout(5000)
            html = page.content()
            name = url.rstrip("/").rsplit("/", 1)[-1]
            Path(f"raw_{name}.html").write_text(html, encoding="utf-8")
            print(f"ok {name} ({len(html):,} bytes)")
        except Exception as e:
            print(f"FAIL {url}: {e}")       # one bad URL shouldn't kill the run
        finally:
            page.close()                    # close the tab, keep the browser
finally:
    browser.close()                         # close the browser exactly once

Three habits worth copying from this, all visible in krueng.ai's real scraper:

• Save the raw HTML to disk (raw_*.html) before you parse it. Fetching is the slow, fragile, rate-limited part; parsing is fast and you'll redo it many times as you refine your code. Separating them means you scrape once and re-parse for free.
• Wrap each URL in try/except so one failure (a 404, a timeout) doesn't abort the whole batch.
• try/finally for cleanup — the tab and the browser get closed even if something throws.

The rental map on krueng.ai answers "what can I rent near Wat Don Chan?" It's built from four scripts, and the split is the most important lesson in this whole page: separate fetching from parsing from rendering. Each stage writes a file the next stage reads.

Only stages 1 and 2 touch the network, and they're the only ones that need CloakBrowser. Once the raw HTML is on disk, you can re-run parsing and rendering as many times as you like — instantly, offline, free. To regenerate the report after tweaking the layout, you run only python rentals_final.py; you re-scrape (collect → expand → parse → final) only when the data itself is stale.

This is also why you don't fork a new script per site. Notice rentals_collect.py drives its six URLs from one TARGETS list and one loop — not six copy-pasted functions. We'll keep that discipline in the capstone.

Once content() hands you the HTML string, you have to find the actual data inside it. There are two good approaches, in order of preference.

7a. Read the structured data the site already embeds

Most listing and shopping sites embed a machine-readable copy of their data as JSON-LD — a <script type="application/ld+json"> block they put there so Google can show rich results. It's far more stable than the visible markup, which changes constantly. This is exactly what the real scraper does:

import json, re

def extract_jsonld(html: str) -> list:
    """Pull every application/ld+json block out of a page as Python objects."""
    blocks = re.findall(
        r'<script[^>]+type=["\']application/ld\+json["\'][^>]*>(.*?)</script>',
        html, flags=re.DOTALL | re.IGNORECASE,
    )
    out = []
    for raw in blocks:
        try:
            out.append(json.loads(raw.strip()))
        except json.JSONDecodeError:
            continue            # skip malformed blocks, don't crash
    return out

From those objects you read clean fields like name, price, geo.latitude — no fragile HTML digging.

7b. Regex the visible text as a fallback

When there's no JSON-LD, you fall back to pattern-matching the rendered text. The rental scraper finds Thai-baht prices with one small regex:

import re

THB = re.compile(r'฿\s?([\d,]{3,12})')

def first_price(html: str):
    m = THB.search(html)
    return int(m.group(1).replace(",", "")) if m else None

The [\d,]{3,12} says "3 to 12 digits-or-commas" so it catches 15,000 but not a stray single digit. .replace(",", "") strips the thousands separators before int(). We'll lean on this same trick in the capstone, for dollars as well as baht.

Now the payoff. We want the price of the MSI EdgeXpert — MSI's build of the NVIDIA DGX Spark mini-AI-supercomputer (the same class of box krueng.ai runs its local avatar tutor on) — across three storefronts: Amazon (global, prices in USD), Shopee and Lazada (Southeast Asia, prices in Thai baht). All three are JavaScript-heavy and bot-protected; a plain requests.get on any of their search pages returns an empty shell or a block page. CloakBrowser loads the real, rendered search results.

Following the rule from §6, this is one config-driven loop, not three copy-pasted scrapers. Each site is a dict carrying its own extraction pattern; the loop pulls the same three fields from each — a product name, its price, and a link to the item page:

"""The heart of the capstone. The full cloakbrowser_tutorial.py (Appendix A) wraps
this as the compare() step, with a JSON-LD fallback and a page-price range. Here we
do the core job: for each storefront, pull the product results — name, price, and a
link to the item page.

A plain requests.get() returns a Cloudflare/JS shell on all three storefronts;
CloakBrowser loads the real, rendered search results.
"""
import re
import sys
from cloakbrowser import launch

sys.stdout.reconfigure(encoding="utf-8")   # so printing ฿ doesn't crash on Windows

# Each site buries product data differently, so the extraction pattern lives in
# the config (named groups: url, name, price) and the loop stays generic.
SITES = [
    {"name": "Amazon", "url": "https://www.amazon.com/s?k=MSI+EdgeXpert",
     "item_pattern": None},                 # usually blocks scripted hits
    {"name": "Shopee", "url": "https://shopee.co.th/search?keyword=MSI%20EdgeXpert",
     "item_pattern": None},                 # results load via API, not in the HTML
    {"name": "Lazada", "url": "https://www.lazada.co.th/catalog/?q=MSI+EdgeXpert",
     # in Lazada's grid the link, title and price sit right next to each other:
     "item_pattern": r'href="(?P//www\.lazada\.co\.th/products/[^"]+)"\s+'
                     r'title="(?P[^"]+)".{0,1500}?'
                     r'<span class="ooOxS">฿(?P[\d,]+(?:\.\d{2})?)'},
]


def detail_items(html, pattern, limit=6):
    """Pull product results (name, price, link) out of the rendered page."""
    if not pattern:
        return []
    items, seen = [], set()
    for m in re.finditer(pattern, html, re.DOTALL):
        g = m.groupdict()
        url = "https:" + g["url"] if g["url"].startswith("//") else g["url"]
        if url in seen:
            continue
        seen.add(url)
        items.append({"name": g["name"].strip(),
                      "price": float(g["price"].replace(",", "")),
                      "url": url})
        if len(items) >= limit:
            break
    return items


BLOCK = ("Robot Check", "validateCaptcha", "automated access")   # bot-wall markers

def blocked(html):
    return len(html) < 5000 or any(m in html for m in BLOCK)


def shop(browser, site, tries=3):
    """Fetch a storefront, retrying past a transient block, then extract."""
    for i in range(tries):
        page = browser.new_page()
        try:
            page.goto(site["url"], timeout=60000, wait_until="domcontentloaded")
            page.wait_for_timeout(6000)              # let results render
            html = page.content()
            if not blocked(html):
                return detail_items(html, site["item_pattern"])
            page.wait_for_timeout(6000 * (i + 1))    # transient block — back off, retry
        finally:
            page.close()
    return []          # still blocked (Amazon rate-limits) — see the note on its API


browser = launch()
try:
    for site in SITES:
        items = shop(browser, site)
        print(f"\n{site['name']} — {len(items)} result(s):")
        for it in items:
            print(f"  ฿{it['price']:>10,.0f}  {it['name'][:50]}")
            print(f"  {'':>12}  {it['url']}")     # link to the item page
finally:
    browser.close()

What makes this work where requests can't:

• Each storefront's search page is rendered by JavaScript after the HTML arrives. requests sees the empty shell; CloakBrowser waits (wait_for_timeout(6000)) and reads the populated grid.
• All three sit behind bot protection that 403s a scripted request. The stealth browser passes the check.
• The extraction pattern lives in each site's dict, so the loop is identical for all of them — adding a fourth storefront is one new entry, not a new function. The named groups (url, name, price) mean the assembling code doesn't care what order a site lists them in.
• The sys.stdout.reconfigure(encoding="utf-8") line is not decoration: without it, printing a ฿ sign crashes on a default Windows console (cp1252). The first run of this script taught us that the hard way.

A stealth browser makes it possible to ignore a site's wishes. That doesn't make it right. The line is simple: read what a human visitor could read, at a human pace, for a legitimate purpose.

• Check robots.txt and the Terms of Service. Some sites forbid scraping outright. Personal price-checking is one thing; redistributing or reselling someone's data is another.
• Go slow. Add page.wait_for_timeout() between requests. A few pages spaced out by seconds looks like a person; hundreds per minute is an attack and can knock a small site over.
• Cache aggressively. This is the real reason §6 saves raw HTML — re-parsing a file you already fetched puts zero load on anyone's server. Never re-fetch what you already have.
• Take only what you need, and prefer an official API when one exists. Amazon, for instance, has a Product Advertising API; scraping is the fallback, not the first choice.
• Don't scrape personal data or anything behind a login you're not entitled to bypass.

Used this way, CloakBrowser is just a more capable version of opening the page yourself — which is exactly what it is.

• Get the script & run it — download the whole tutorial as one runnable script (this page's capstone plus the earlier lessons). Setup and run commands are in that section.
• Python Basics — if any of the syntax above was new (functions, f-strings, try/except).
• python_apis_async.ipynb — when a site does offer an API, you'll often want it over scraping. This covers calling HTTP APIs with requests.
• The requests quickstart and BeautifulSoup docs — the two libraries you'll pair with CloakBrowser most often.
• The official Python tutorial — docs.python.org/3/tutorial.

The deepest lesson here isn't about any one library. It's the pipeline shape: fetch slowly and save raw, parse fast and often, render last. Once you internalise that split, every scraper you write gets easier to debug.

Everything on this page is bundled into one runnable script. Two byte-identical copies are provided — they differ only in line endings, so grab the one that matches your OS:

Install CloakBrowser

In your terminal (PowerShell, Terminal, or your IDE's) — not inside Python. The first install downloads a patched Chromium (~200 MB), so it's slow once, then cached. Section 3 above shows the virtual-environment version.

pip install cloakbrowser

Run it

With no argument it runs the MSI EdgeXpert price comparison. Pass a step name to run a single lesson instead:

python cloakbrowser_tutorial.py            # the price comparison (default)
python cloakbrowser_tutorial.py minimal    # the 5-line recipe
python cloakbrowser_tutorial.py multipage  # one browser, many pages -> raw_*.html
python cloakbrowser_tutorial.py jsonld     # extract the embedded JSON-LD