LangGraph + LEGO Robots — Stateful AI agents that drive physical bricks

Built from a NotebookLM deep-research pass over 26 sources (LangGraph docs, Pybricks docs, community projects, robotics papers). The grounded answers behind every section are saved at HTML/it/ai/langgraph_lego_robots_research.md alongside this page.

What this page is about

You can already build an AI agent that answers chat messages on a laptop. Pointing one at a LEGO robot — so the agent can see, decide, and move — adds a single new dimension and changes almost every design choice. The real world is non-deterministic: motors stall, grippers miss, the camera loses the line. A naive while True: loop crumbles the first time something unexpected happens.

LangGraph is a Python framework for building agents as state machines instead of straight-line loops. Pybricks is open-source firmware that turns a LEGO hub into a fast Python-controllable microcontroller. Put them together and you get a "brain on a PC, body on the hub" split that handles failure recovery, vision-in-the-loop, and human-in-the-loop pauses by construction — not by stacking try/except blocks.

This page distills what NotebookLM pulled out of 26 sources into one document: why the architecture works, how to build it, what missions it shines on, where it bites, and a 2-hour starter project that proves the whole stack on a single SPIKE Prime hub.

LangGraph in one minute

LangGraph is a low-level orchestration runtime for stateful, multi-agent applications. Where a plain agent loop runs sequentially and loses everything if the process crashes, LangGraph models the workflow as a directed graph of nodes connected by edges, with a shared state object flowing through it. That single change unlocks cycles, retries, branching, persistence, and human checkpoints — all the things a hand-rolled loop never quite gets right.

A minimal agent graph: Plan reads the state, decides what to do, and emits tool calls. A conditional edge routes to Act if there are calls, or to END otherwise. Act loops back to Plan so the LLM can observe the consequences of its move.

The four moving parts

State. A TypedDict that flows through the graph. Every node reads it and returns a partial update; LangGraph merges them. A reducer like add_messages can declare that messages should be appended (not overwritten).
Nodes. Plain Python functions. Their job: read state, do work (call an LLM, run a tool, poll a sensor), return a state update. Nothing more.
Edges, including conditional ones. Static edges declare "after this node, run that one." Conditional edges are routing functions that read the current state and return the name of the next node — that's how branching, retry, and termination all happen.
Checkpointers. LangGraph persists the entire state after every node, keyed by a thread_id. If the process crashes (BLE drops, you Ctrl-C, the laptop sleeps), invoking the graph again with the same thread_id resumes from the last completed node. This is what makes physical-world agents survivable.

Human-in-the-loop interrupts

Because state is durably checkpointed, you can pause a graph at any chosen point. Compile with interrupt_before=["act"] and the graph halts right before any physical action, hands control back to you (or the operator), and waits. You inspect the plan, approve or edit it, then resume with a single resume call. The state object is unchanged in between. For a robot that might drive off a table, this is not a luxury — it's the seatbelt.

The LEGO side — four ways to drive a brick from Python

Before LangGraph can send a command, something on the LEGO side has to receive it. There are four real options today; each lands at a different point on the simplicity-vs-power axis.

Option	Hardware	Connection	Strengths	Trade-offs
Pybricks	SPIKE Prime, MINDSTORMS Robot Inventor, Technic Hub, BOOST, City Hub, EV3	Custom MicroPython firmware on the hub; PC talks to it over BLE (Nordic UART) via `pybricksdev` or a web IDE	Fast (≪ 1 ms motor loops), low-level access, built-in PID, gyro-aware drive bases, async multitasking, third-party sensor support including HuskyLens vision	You flash over the official LEGO firmware (reversible), and code runs inside a microcontroller's RAM — tight budgets
LEGO SPIKE App Python	SPIKE Prime, SPIKE Essential	Official SPIKE app uploads to the hub over BLE/USB	Zero setup, sits next to the block-coding UI students already know, safe sandbox	Closed ecosystem; v3.0 removed UART, raw PWM, and third-party hardware hooks
ev3dev	MINDSTORMS EV3 only	Full Debian Linux booted from a microSD card on the brick; SSH in over Wi-Fi/Bluetooth	Run any pip package, the entire CPython ecosystem, fits research projects with heavier logic	EV3 hardware is officially retired; boot time is slow; the brick is power-hungry next to modern hubs
Build HAT on Raspberry Pi	Any Raspberry Pi + Build HAT shield + SPIKE/Robot Inventor motors and sensors	Standard Python on the Pi talks to motors via the HAT's serial interface	Full Pi compute — OpenCV, local LLMs, ROS — runs on the robot itself	The robot is now a Pi + HAT + 8 V battery pack; loses the compact, wire-free LEGO form factor

The default pick for an LLM agent is Pybricks. It hits the right note: fast enough that physical control loops can live entirely on the hub (where they belong — see pitfalls), open enough to wire arbitrary commands in from a PC, and well-documented at docs.pybricks.com.

Architecture — brain on the PC, body on the hub

The single most important design decision is edge-cloud partitioning: keeping high-level reasoning on the PC (where LLMs can take 2-8 seconds to think) while reflexive motor control lives on the LEGO hub (where Pybricks runs PID loops every ~2 ms). The two halves talk over Bluetooth Low Energy via the Nordic UART Service.

Brain ↔ body. The PC owns reasoning, history, and the LLM. The hub owns physics. They speak a tiny text protocol over BLE: short commands one way, short status the other.

Why this split matters: an LLM that tries to micromanage motors at human-reaction speeds will lag the world by seconds, miss its window, and crash into things. An LLM that says "drive 5 cm, then tell me what color you see" lets the hub do the part it's fast at, and consults the LLM only when there's a real decision to make. Token cost drops, latency drops, and safety reflexes (collision-stop, tilt-guard) live where they can react in microseconds.

Code walkthrough — the observe → plan → act → observe loop

The PC side, in ~55 lines. Three nodes, one conditional edge, one cyclical loop. The tool function is where you'd swap in real pybricksdev BLE writes; the rest is shape, not glue.

from typing import TypedDict, Annotated, List
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langchain_core.messages import BaseMessage, ToolMessage
from langchain_core.tools import tool

# 1. The state object — what flows through the graph
class RobotState(TypedDict):
    messages: Annotated[List[BaseMessage], add_messages]
    sensor_data: dict

# 2. Tools — Python functions the LLM is allowed to "call"
@tool
async def drive_target(distance_cm: float) -> str:
    """Drive forward by a specific distance, in centimetres."""
    # Real version: await pybricksdev.write(b"fwd " + str(distance_cm).encode())
    return f"drove {distance_cm} cm; stall=False"

llm_with_tools = llm.bind_tools([drive_target])

# 3. The three nodes
async def perception_node(state: RobotState):
    """Observe: poll the hub for current sensor readings."""
    # Real version: await ble.write(b"color?\n"); reading = await ble.read()
    return {"sensor_data": {"color": "red", "is_stalled": False}}

async def plan_node(state: RobotState):
    """Reason: LLM picks the next tool call given history + sensors."""
    ctx = f"Sensors right now: {state['sensor_data']}"
    response = await llm_with_tools.ainvoke([ctx] + state["messages"])
    return {"messages": [response]}

async def act_node(state: RobotState):
    """Act: execute whatever tools the LLM asked for."""
    last = state["messages"][-1]
    results = []
    for call in last.tool_calls:
        out = await drive_target.ainvoke(call["args"])
        results.append(ToolMessage(content=out, tool_call_id=call["id"]))
    return {"messages": results}

# 4. Routing — should we act, or are we done?
def should_continue(state):
    return "act" if state["messages"][-1].tool_calls else END

# 5. Wire the graph
g = StateGraph(RobotState)
g.add_node("perception", perception_node)
g.add_node("plan", plan_node)
g.add_node("act", act_node)
g.add_edge(START, "perception")
g.add_edge("perception", "plan")
g.add_conditional_edges("plan", should_continue, {"act": "act", END: END})
g.add_edge("act", "perception")   # key line: re-observe after acting
robot_app = g.compile()

💡 The cycle is the point

That single g.add_edge("act", "perception") line is what makes this stateful rather than fire-and-forget. Every physical action is followed by a fresh observation, so the LLM's next decision is informed by what actually happened — including failures. If the motor stalled, the next perception cycle records it; the next plan cycle reasons about it. No try/except needed.

Where this architecture wins — four example missions

LangGraph isn't worth the complexity for missions that finish in 30 lines of straight Pybricks. It earns its keep on tasks where the physical world reliably breaks the plan and the right response is to think again, not retry the same thing.

Vision-in-the-loop

1. Resilient line-following

A SPIKE Prime with a HuskyLens AI camera follows a path that has sudden breaks, sharp angles, and changing light. A hand-coded loop polls the camera and drives the motors; when the camera loses the line, the loop crashes or the robot drives off the table. The LangGraph version separates perception from action: when the line is gone, the conditional edge routes to a reasoning node where the LLM looks at the last few coordinates, narrates a guess ("the line probably curved left"), and orders a small spiral-search pattern to reacquire it.

Retry on failure

2. Autonomous brick assembly

A robotic arm picks and stacks LEGO bricks into a pyramid. Grasps fail, bricks misalign. A linear script wraps each grasp in a rigid try/except. The LangGraph version uses a cyclic graph: an "act" node attempts the grasp, the next perception cycle reads the gripper sensor or a camera, and if the grasp failed the graph loops back to a reflection node where the LLM calculates a new approach angle and tries again — with full context about what just went wrong.

Conditional branching

3. Object sorter with operator escalation

A bin of mixed items is sorted into colour-coded containers. For confident classifications the graph routes straight to a "move-to-bin" node. For ambiguous items the conditional edge instead routes to a human-in-the-loop node: the robot pauses, the operator approves or overrides via the terminal, and the graph resumes from exactly where it stopped. The state object remembers the last reading either way.

Mid-mission re-planning

4. Maze navigation / treasure hunt

The robot must locate a coloured target in an unknown maze. A pre-computed path is useless because dead ends are discovered mid-mission. LangGraph supervises specialised sub-agents: a Navigator handles motion, a Vision agent watches for the target. When the Navigator reports an unexpected wall, the failure is written into shared state, the supervising LLM re-plans the route on the fly, and the next motion command reflects the new map.

Pitfalls — and how experienced teams mitigate them

1. LLM planning latency vs. real-time control

Frontier LLMs take 2-8 seconds per decision cycle. Robots need millisecond reactions to stop before they hit something.

Fix: Edge-cloud partitioning. Reflexive control (PID, IMU stabilisation, collision-stop) runs locally on the Pybricks hub at < 2 ms. The LangGraph orchestrator on the PC handles strategy and re-planning at 500 ms – 2 s. Never let the LLM "drive" a motor directly.

2. BLE disconnects

The BLE link (Nordic UART Service) drops occasionally — interference, low battery, Bluetooth stack hiccups.

Fix: Attach a RetryPolicy to tool nodes for short-lived glitches (exponential backoff). For longer disconnects, the SQLite checkpointer holds the state — restart the script with the same thread_id and the graph resumes at the last completed node.

3. Sensor noise & unmodeled dynamics

Color sensors read different values in different ambient light; motor encoders drift; friction varies across surfaces.

Fix: Cyclical feedback loops — act, then immediately re-observe. If the act didn't produce the expected sensor change, the next plan cycle's LLM call decides whether to retry, recalibrate, or fall back.

4. Cost of frequent LLM calls

An observe → plan → act → observe loop can easily fire 20+ LLM calls per mission. Tokens add up fast.

Fix: Hard iteration caps (e.g. max_iterations = 8) as a circuit breaker. Route simpler sub-decisions to a smaller, cheaper model (GPT-4o-mini, Hermes 3:8B locally) and only escalate to a frontier model when the cheap one declines.

5. Tool-call timeouts

The LLM tool call returns immediately ("drove 50 cm"), but physically the robot needs 8 real seconds to actually drive 50 cm. If your async tool doesn't account for that, you'll fire the next plan cycle while the motor is still moving.

Fix: Tool functions wait for an explicit "ok" or "stall" reply from the hub before returning. Enable handle_tool_errors=True on the ToolNode so timeouts come back as a text message the LLM can reason about instead of crashing the script.

⚠️ Safety / blast-radius

An LLM optimised for task success will happily drive the robot off a table to reach a goal. Real deployments need:

Hardware fail-safes on the hub — collision-stop and tilt-guard run locally and ignore the brain.
Human-in-the-loop interrupts — compile the graph with interrupt_before=["act"] for any high-risk action. The graph pauses, you approve, then resume.
Pydantic-typed tool arguments — distances clamped to ≤ 100 cm, motor speeds capped at safe values. The LLM can't issue a command outside the schema.

Starter project — the "Color-Seeker" agent

You've got a SPIKE Prime hub (or Robot Inventor / Technic Hub), an OpenAI key or a local Hermes via Ollama, and 1.5–2 hours. Here's a project small enough to actually finish, large enough to teach the whole stack.

Bill of materials

Hardware: 1× SPIKE Prime Hub (or compatible) flashed with Pybricks · 2× medium motors for a two-wheel drive base · 1× color sensor mounted facing down · a PC or Mac with Bluetooth
Software: pip install langgraph langchain langchain-openai pybricksdev · the Pybricks web IDE (code.pybricks.com) to flash the hub · an OpenAI key, or Ollama with hermes3:8b for fully local

The mission

Drive forward until you see a blue line, then stop and celebrate. A 1-D problem on purpose — no maze, no vision model, one decision dimension. The robot polls colour, sends it to the LLM, and waits for the LLM to decide whether to issue "drive forward" or "stop and beep".

What you'll build, in two halves

The body (Pybricks, on the hub): A small script that listens on usys.stdin. If it receives color?, it reads the sensor and prints the result. If it receives fwd 5, it drives forward 5 cm and prints ok. That's it — no logic, just a server.
The brain (LangGraph, on the PC): A graph with three nodes — perception, plan, act — exactly the shape in the code walkthrough above. The tools wrap pybricksdev BLE writes. The conditional edge ends the mission once the LLM has called stop_and_celebrate.

What you'll actually learn (the unstated curriculum)

Edge-cloud partitioning — felt, not just read: You'll watch the LLM take 4 seconds to "decide" while the motor keeps gliding. You'll viscerally understand why reflexive control belongs on the hub.
Stateful memory across cycles: You'll see the same messages list grow through each loop, carrying context the LLM uses to decide.
BLE tool calling: You'll wrap an async pybricksdev call as a @tool and watch the LLM choose when to call it.
Agentic error recovery: Pull the colour sensor off mid-run. handle_tool_errors=True catches the failure, the LLM sees a "sensor unreachable" message, and decides to stop the robot instead of crashing.

From this starter, the natural next steps are: add a second sensor (force or distance) and let the LLM choose which to consult; add a HuskyLens camera and an "identify object" tool; or wrap the whole thing in interrupt_before=["act"] and require operator approval before every move.

Sources — the 26 the NotebookLM pass cited

NotebookLM imported these from a deep-research pass on "Using LangGraph to control LEGO robots, orchestrating AI agents for LEGO Mindstorms EV3, SPIKE Prime, and Pybricks." Inline citations in the synthesis markdown point to specific passages.

LangGraph foundations — official docs, Building AI Agents with LangGraph — DEV Community, Building Smart AI Agents: A Practical Guide to LangGraph Design Patterns — Towards AI, LangGraph Multi-Agent in 20 Minutes — LaunchDarkly
LEGO Python ecosystems — Pybricks v3.x documentation (the BLE / Nordic UART pages most cited), pybricksdev v2.3.2 documentation — Bluetooth Low Energy, LEGO Education SPIKE Python release notes, ev3dev v2 docs
Physical AI / robotics agents — Building intelligent physical AI: From edge to cloud with Strands, plus several arXiv / blog posts on LLM-driven robotics latency budgets, retry policies, and human-in-the-loop patterns
Security & production practices — AI Agent Security Best Practices and Tutorial — IBM
Community show-and-tell — Built a visual mission control for AI agents — Reddit, and several maker blog posts demoing LangGraph+robotics combinations

The full grounded answers (with footnote citations to specific passages) live alongside this page at HTML/it/ai/langgraph_lego_robots_research.md. The source notebook ID is 5ae67007-acab-4e69-ae39-9baf503ac139; reproduce or extend the research with notebooklm use 5ae67007 + notebooklm ask "...".