LangGraph + LEGO Robots

You can already build an AI agent that chats on a laptop. Pointing one at a LEGO robot — so it can see, decide, and move — adds the one dimension that breaks naive agent loops: the physical world is non-deterministic. Motors stall, grippers miss, cameras lose the line. A while True: loop crumbles on the first surprise.

This tutorial walks through the architecture that doesn't crumble: LangGraph as the planner running on a PC, Pybricks as a fast reflexive controller running on the LEGO hub, talking over Bluetooth Low Energy. Every section is interactive — click things, drag the slider, flip the cards. The content is grounded in a NotebookLM deep-research pass over 26 sources.

🎧 Audio overview

A 10–15 minute deep dive, narrated. Same notebook, generated via NotebookLM's audio overview feature.

🗺️ Concept mind map

Click any node with a ▸ to expand or collapse. Six top-level branches; the diagram is generated from the same source set.

🎬 The state machine, animated

A simulated mission. Click Run to watch a token travel through the graph — perception polls a sensor, plan asks the LLM, act executes a tool, then the loop repeats until the LLM decides to END. The status panel logs each transition.

⚡ Latency budgets — why brain and body have to split

An LLM takes seconds to decide. A motor PID loop reacts in milliseconds. Drag the slider to set how fast your LLM call is, and watch what happens to a robot moving at 30 cm/s between decisions.

🤖 Pick your LEGO platform

Four real options today; toggle filters to highlight what fits your constraints.

💻 Code walkthrough — click any annotated line

The PC side, ~55 lines. Tap a line marked with the yellow border to read what it does and why.

# 1. The state object — what flows through the graph
class RobotState(TypedDict):
    messages: Annotated[List[BaseMessage], add_messages]
    sensor_data: dict

# 2. Tools — Python functions the LLM is allowed to "call"
@tool
async def drive_target(distance_cm: float) -> str:
    """Drive forward by a specific distance, in centimetres."""
    # Real: await pybricksdev.write(b"fwd " + str(distance_cm).encode())
    return f"drove {distance_cm} cm; stall=False"

# 3. The three nodes
async def perception_node(state: RobotState):
    # Real: await ble.write(b"color?\n"); reading = await ble.read()
    return {"sensor_data": {"color": "red", "is_stalled": False}}

async def plan_node(state: RobotState):
    ctx = f"Sensors right now: {state['sensor_data']}"
    response = await llm_with_tools.ainvoke([ctx] + state["messages"])
    return {"messages": [response]}

async def act_node(state: RobotState):
    last = state["messages"][-1]
    results = []
    for call in last.tool_calls:
        out = await drive_target.ainvoke(call["args"])
        results.append(ToolMessage(content=out, tool_call_id=call["id"]))
    return {"messages": results}

# 4. Routing — should we act, or are we done?
def should_continue(state):
    return "act" if state["messages"][-1].tool_calls else END

# 5. Wire the graph
g = StateGraph(RobotState)
g.add_node("perception", perception_node)
g.add_node("plan", plan_node)
g.add_node("act", act_node)
g.add_edge(START, "perception")
g.add_edge("perception", "plan")
g.add_conditional_edges("plan", should_continue, {"act": "act", END: END})
g.add_edge("act", "perception")   # key line: re-observe after acting
robot_app = g.compile()

🎯 Mission picker — same graph, four shapes

The basic perception → plan → act cycle morphs to fit the mission. Click each to see the graph variant and why hand-coded loops fail at it.

⚠️ Pitfalls — flip a card to see the fix

Six classes of failure that bite real LangGraph-on-LEGO projects. Click any card to flip to the mitigation experienced teams use.

🚀 Build the Color-Seeker — your progress sticks

A 2-hour starter project. Check boxes as you finish each step; progress is saved in your browser's localStorage.

🖼️ Visual summary

❓ Quick quiz

Generated from the same source set. Pick an answer; the explanation appears below.