# LangGraph + LEGO Robots — NotebookLM research synthesis

- Generated: 2026-05-17 13:31
- Notebook: 5ae67007-acab-4e69-ae39-9baf503ac139
- Sources imported: 26 (deep web research mode)

## What is LangGraph and how does it differ from a plain agent loop?

LangGraph is a low-level orchestration framework and runtime designed for building stateful, multi-agent applications [1, 2]. 

While a plain agent loop or standard LangChain application relies on sequential, linear execution that loses context if the process crashes, LangGraph models workflows as **state machines (directed graphs)** [3-5]. This shift from a naive `for` or `while` loop to a graph architecture enables complex cyclical flows, native memory, deterministic transitions, and the ability to safely recover from failures without losing intermediate progress [3, 4, 6].

Here is how the execution model operates through its core concepts:

**The State Graph Abstraction**
The graph is driven by a shared **State** object—often defined as a `TypedDict`—which acts like a "backpack" of data that flows through the workflow [7, 8]. Every component in the system can read this global state and return a partial update [4, 7]. LangGraph automatically merges these partial updates into the full state [4]. You can also define specific merge behaviors; for example, using an `add_messages` reducer ensures that new conversation messages are appended to the history rather than replacing it [7, 9]. 

**Nodes vs. Edges**
*   **Nodes** are Python functions that act as the "workers" doing specific jobs, such as calling an LLM, processing data, or executing an external tool [10, 11]. A node receives the current state, performs its work, and returns a state update [12].
*   **Edges** are the connections or paths that dictate the flow of data from one node to another, defining the strict execution order of the graph [10].

**Conditional Edges**
Conditional edges serve as dynamic decision points (if-else branches) within the graph [10, 11]. They rely on pure Python routing functions that evaluate the current state and return a string dictating the next node to execute [13]. For example, after an LLM node runs, a conditional edge can evaluate whether the LLM decided to call a tool: if yes, it routes the flow to a tool-execution node; if no, it routes to the `END` of the workflow [14].

**Persistence and Checkpointing**
To ensure durability, LangGraph includes checkpointers that save the entire state (for example, to a SQLite database) after every single node completes [4, 15]. Every session is tracked using a unique `thread_id` [16]. If an agent crashes or times out in the middle of a multi-step process, you do not have to start over. By invoking the graph with the same `thread_id`, LangGraph loads the most recent checkpoint and resumes execution exactly at the next uncompleted node [17, 18]. 

**Human-in-the-Loop Interrupts**
Because execution is durably checkpointed, LangGraph natively supports pausing workflows to wait for human intervention [19]. You can pause a graph by specifying an `interrupt_before` argument when compiling the graph, or by calling an `interrupt()` function dynamically inside a node [20]. When an interrupt is triggered, the graph halts, saves its state, and returns control to the user (e.g., to review a plan or approve an action) [19]. Once the human provides input, the system is reinvoked with a `resume` command, and execution continues exactly where it left off [19].

**Cited sources:** 20 references

## Python options for programming modern LEGO robots

Here is a comparison of the main Python options for programming a modern LEGO robot. 

*Please note: The provided sources extensively cover Pybricks, the official LEGO SPIKE App, and ev3dev. However, **they do not contain information regarding the Raspberry Pi Build HAT**. I have supplemented that specific section and a few of the code examples using my external knowledge base, which you may want to independently verify.*

### 1. Pybricks
*   **Target Hardware:** SPIKE Prime, MINDSTORMS Robot Inventor, Technic Hub, BOOST Move Hub, City Hub, and MINDSTORMS EV3 [1].
*   **Connection Method:** Pybricks replaces the default LEGO firmware with an optimized MicroPython environment that runs directly on the hub [2, 3]. Programs are written on a PC or mobile device and downloaded to the hub via Bluetooth Low Energy (BLE) or USB using the Pybricks IDE or `pybricksdev` command-line tool [2, 4, 5].
*   **Strengths:** Pybricks offers extremely fast execution and granular, low-level control over hardware primitives [3]. It features advanced motor control with built-in Proportional-Integral-Derivative (PID) synchronization [6], highly accurate drive base movements utilizing the hub's gyro sensors [7], and native support for cooperative multitasking (using `async` and `await`) [8]. It also allows hub ports to be configured as full-duplex serial ports to integrate third-party sensors like the HuskyLens AI camera [9].
*   **Weaknesses:** You must flash over the official LEGO firmware to use it, though returning to the official firmware is an easy, reversible process [10]. Because the code runs entirely on the hub's microcontroller, it has tighter memory constraints than a full computer [11].
*   **Code Example:** `await hub.display.text("Hello")` [12]

### 2. LEGO SPIKE App Python
*   **Target Hardware:** SPIKE Prime and SPIKE Essential [13].
*   **Connection Method:** Python programs are written within the official LEGO Education SPIKE app and sent to the hub to execute over a Bluetooth or USB connection [14].
*   **Strengths:** It is officially supported by LEGO and integrated seamlessly alongside their drag-and-drop block coding interfaces [15]. It provides a safe, out-of-the-box experience for students and absolute beginners, complete with an integrated knowledge base [14].
*   **Weaknesses:** It is a closed ecosystem that restricts low-level hardware control [3]. The transition to SPIKE App v3.0 removed critical Python functionality, such as native UART control, raw PWM access, and support for third-party hardware, frustrating advanced users [16, 17].
*   **Code Example:** `hub.light_matrix.show_image('HAPPY')` [18]

### 3. ev3dev (Mindstorms EV3)
*   **Target Hardware:** LEGO MINDSTORMS EV3 [1].
*   **Connection Method:** Instead of flashing the firmware, ev3dev boots a full Linux-based operating system from a microSD card inserted into the EV3 brick [19]. Scripts can be executed via SSH over Wi-Fi/Bluetooth or run directly from the brick's screen.
*   **Strengths:** Because it leverages a full Linux-based architecture [19], it allows you to use standard Python 3 and its vast ecosystem of libraries. It remains a robust platform for research that can handle complex computational logic that standard microcontrollers cannot [19].
*   **Weaknesses:** The EV3 hardware is aging and the product line has been officially retired [19]. *(External Info: Booting a full Linux OS from an SD card is also considerably slower than starting a modern microcontroller hub).*
*   **Code Example:** *(External Info)* `motor_left.run_to_rel_pos(position_sp=360, speed_sp=500)`

### 4. Build HAT on Raspberry Pi *(External Knowledge)*
*   **Target Hardware:** Any standard Raspberry Pi connected to a "Build HAT," which acts as an adapter providing four ports for SPIKE Prime and Robot Inventor motors and sensors.
*   **Connection Method:** The Raspberry Pi acts as the "brain." Standard Python runs directly on the Pi's full operating system and communicates with the HAT via a serial connection to drive the LEGO hardware.
*   **Strengths:** You gain the massive computing power of a Raspberry Pi. This is ideal for computationally heavy tasks like running camera-based AI vision models (e.g., OpenCV, local LLMs), ROS (Robot Operating System), or complex network operations without needing to compress your logic into a small MicroPython environment. 
*   **Weaknesses:** The robot becomes tethered to a bulky Raspberry Pi and a separate power bank (the HAT requires its own 8V power supply to drive the LEGO motors). It lacks the compact, integrated, wire-free form factor of a single LEGO hub.
*   **Code Example:** *(External Info)* `motor_a.run_for_degrees(360, speed=50)`

**Cited sources:** 19 references

## A concrete LangGraph architecture for controlling a LEGO robot

A concrete LangGraph architecture for controlling a LEGO robot separates the high-level "Brain" (running on a PC or cloud server) from the "Body" (the resource-constrained LEGO hub running Pybricks) [1, 2]. The LLM reasons and plans on the PC, while physical actions and sensor polling are dispatched to the hub via Bluetooth Low Energy (BLE) [2, 3].

### 1. The Graph State
The `State` object acts as the shared memory for the mission. It must track the agent's reasoning history, but also the physical realities of the robot [4]. 
*   **`messages`**: An append-only list (using `add_messages`) tracking the LLM's reasoning, tool calls, and tool responses.
*   **`sensor_data`**: A dictionary holding the latest physical feedback, such as `battery_level`, `is_stalled`, tilt, or `current_coordinates` [4].

### 2. The Nodes
The workflow is modeled as a cyclical observe-reason-act state machine [5, 6].
*   **Perception (Observe) Node:** Polls the LEGO hub's sensors (like the IMU or motor encoders) via BLE and updates the `sensor_data` in the graph state [6].
*   **Plan (Reason) Node:** Reads the current `sensor_data` and `messages`, passes them to an LLM, and generates either a final response or specific tool calls (e.g., "drive forward") [6].
*   **Act Node:** Parses the LLM's tool calls, executes them by sending physical commands to the hub, and logs the outcome (e.g., whether the motor stalled or completed the movement) [6].

### 3. The Tools
Instead of talking directly to the motors, the agent utilizes Python functions decorated as tools that leverage the `pybricksdev` package [3]. `pybricksdev` bridges the PC and the LEGO hub, sending asynchronous commands over the Nordic UART Service (NUS) without blocking the hub's local safety and balance loops [3]. 

### Python Code Sketch
Here is a sketched implementation of the `perception -> plan -> act -> observe` loop using LangGraph and Pybricks BLE concepts:

```python
from typing import TypedDict, Annotated, List
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langchain_core.messages import BaseMessage, ToolMessage
from langchain_core.tools import tool

# 1. Define the Graph State
class RobotState(TypedDict):
    messages: Annotated[List[BaseMessage], add_messages]
    sensor_data: dict

# 2. Define the Tools (Using pybricksdev for BLE communication)
@tool
async def drive_target(distance_cm: float) -> str:
    """Drives the LEGO robot forward by a specific distance."""
    # Pseudocode for pybricksdev BLE communication over Nordic UART
    # payload = f"drive {distance_cm}\n".encode()
    # await ble_connection.write(payload)
    # await ble_connection.wait_for_reply()
    return f"Successfully drove {distance_cm} cm."

tools = [drive_target]
llm_with_tools = llm.bind_tools(tools)

# 3. Define the Nodes
async def perception_node(state: RobotState):
    """Observe: Polls the LEGO hub for current sensor states."""
    # simulated_data = await ble_connection.read_sensors()
    simulated_data = {"is_stalled": False, "tilt": [7]} 
    return {"sensor_data": simulated_data}

async def plan_node(state: RobotState):
    """Reason: LLM decides what to do based on sensors and history."""
    system_context = f"Current sensors: {state['sensor_data']}"
    # The LLM receives the system context + the conversation history
    response = await llm_with_tools.ainvoke([system_context] + state["messages"])
    return {"messages": [response]}

async def act_node(state: RobotState):
    """Act: Executes the physical tools requested by the LLM."""
    last_message = state["messages"][-1]
    results = []
    for tool_call in last_message.tool_calls:
        # Execute the tool and capture physical results/errors
        result_str = await drive_target.ainvoke(tool_call["args"])
        results.append(ToolMessage(content=result_str, tool_call_id=tool_call["id"]))
    return {"messages": results}

# Conditional routing
def should_continue(state: RobotState):
    last_message = state["messages"][-1]
    if last_message.tool_calls:
        return "act" # If the LLM made a tool call, act on it
    return END # Otherwise, the mission is complete

# 4. Build and Compile the Graph
workflow = StateGraph(RobotState)

workflow.add_node("perception", perception_node)
workflow.add_node("plan", plan_node)
workflow.add_node("act", act_node)

# Construct the cyclical loop
workflow.add_edge(START, "perception")
workflow.add_edge("perception", "plan")
workflow.add_conditional_edges("plan", should_continue, {"act": "act", END: END})

# After acting, loop back to perception to observe the results of the action
workflow.add_edge("act", "perception") 

robot_app = workflow.compile()
```

In this architecture, if the robot fails a physical action (e.g., hitting a wall and stalling), the `act` node records the error, the `perception` node observes the stall, and the `plan` node dynamically generates a recovery strategy on the next iteration [5, 6, 8].

**Cited sources:** 7 references

## Tasks that fit LangGraph + LEGO well

LangGraph excels at controlling LEGO robots because the physical world is inherently non-deterministic. While a hand-coded `while` or `for` loop expects predictable, sequential inputs, a real-world robot might drop an object, stall a motor, or suddenly lose sight of its target [1]. 

By modeling robotic missions as stateful graphs, LangGraph transforms brittle scripts into resilient observe-reason-act loops [2]. Here are 4 concrete missions where LangGraph’s architecture drastically outperforms traditional hand-coded loops:

### 1. Resilient Line Following (Highlight: Vision-in-the-Loop)
*   **The Mission:** A SPIKE Prime robot uses a HuskyLens AI camera to follow a path, but the path has sudden breaks, sharp angles, or changing lighting conditions [3, 4].
*   **Why LangGraph Wins:** A hand-coded loop typically polls the camera and drives the motors directly; if the camera loses the line, the loop crashes or the robot drives off the table. LangGraph naturally supports **vision-in-the-loop** by separating perception from action [4]. If the camera loses the line, the graph transitions to a reasoning node where the LLM analyzes the failure based on recent coordinate history, narrates its thought process, and commands a localized search pattern to reacquire the line [4, 5].

### 2. Autonomous LEGO Brick Assembly (Highlight: Retry on Failure)
*   **The Mission:** A robotic arm must pick up specific LEGO bricks and stack them to build a pyramid [6].
*   **Why LangGraph Wins:** Physical manipulation is messy—grasps fail and bricks misalign [1, 6]. A linear script relies on rigid `try-except` blocks that lack context. LangGraph models this non-deterministic reality natively using cyclic **retry on failure** logic [2, 7]. If a sensor (like a stall detector or camera) verifies that a grasp failed, a conditional edge loops the graph back to a "Reflection" node. The agent analyzes why the grasp failed, dynamically calculates a new approach angle, and re-attempts the action [2, 6].

### 3. Intelligent Object Sorter (Highlight: Conditional Branching)
*   **The Mission:** The robot must sort a mixed bin of items using visual feedback, routing them into different containers based on classification.
*   **Why LangGraph Wins:** A traditional loop relies on massive, fragile nested `if/else` statements that cannot adapt to unknown variables. LangGraph uses conditional edges to create dynamic **branching** based on the robot's current state [7]. If the vision model encounters a highly ambiguous object it cannot confidently classify, the graph can automatically branch to a "Human-in-the-Loop" node [8, 9]. The robot safely pauses its physical execution, asks the operator for input, and then seamlessly resumes sorting [9].

### 4. Mini Treasure Hunt / Maze Navigation (Highlight: Mid-Mission Re-planning)
*   **The Mission:** The robot must navigate an unknown maze to locate a specific colored object, but it encounters unexpected dead ends or blocked paths.
*   **Why LangGraph Wins:** A hand-coded pathfinding loop struggles when physical realities invalidate its pre-calculated route. LangGraph excels at **mid-mission re-planning** by orchestrating specialized sub-agents [10]. A "Supervisor" agent delegates movement to a "Navigation" agent and object detection to a "Vision" agent [11]. If the robot hits an unexpected wall, the failure is recorded in the shared state memory. The LLM processes this new environmental data, dynamically re-plans the route on the fly, and issues updated commands to the Navigation agent [11, 12].

**Cited sources:** 12 references

## Common pitfalls when running LangGraph against a real LEGO robot

**LLM planning latency vs. real-time control**
Because frontier LLM agent reasoning incurs a latency of 2 to 8 seconds per decision cycle, they are incapable of real-time reactive control, which requires millisecond-level responsiveness [1].
**Mitigation:** Experienced teams use **edge-cloud partitioning** to separate the "brain" from the "body" [2]. High-level reasoning, strategy adjustment, and goal decomposition run on the LangGraph orchestrator in the cloud or a PC (operating at ~500ms to 2-second latency) [3, 4]. Reflexive actuation—such as IMU stabilization, collision avoidance, and PID motor loops—is delegated to the LEGO hub's microcontroller running Pybricks, which reacts locally in under 2 milliseconds [3, 4]. 

**BLE disconnects**
The physical separation of the orchestrating PC and the LEGO hub relies on Bluetooth Low Energy (BLE) and the Nordic UART Service, making the system vulnerable to transient wireless signal dropouts [2, 5, 6].
**Mitigation:** Developers attach LangGraph's **`RetryPolicy`** to the tool nodes, which automatically retries failed BLE transmissions with exponential backoff, allowing short-lived network glitches to resolve themselves [7-9]. For longer disconnects, LangGraph's durable execution via **persistent state checkpointing** (e.g., saving the state to a SQLite database) ensures that if the agent crashes, it can resume from the exact node where it left off once reconnected [10].

**Sensor noise and unmodeled dynamics**
Physical environments are messy. Translating target coordinates into physical motor primitives is highly sensitive to sensor noise and unmodeled joint friction, making sub-centimeter precision manipulation exceptionally challenging for LLM agents [11, 12].
**Mitigation:** Teams rely heavily on LangGraph's **cyclical feedback loops**. Instead of fire-and-forget commands, the graph is designed to act and then verify. If a sensor indicates a grasp failed or the robot missed its mark, a conditional edge routes the graph back to a reasoning node where the LLM adjusts its approach and tries again [13].

**Cost of frequent LLM calls**
Physical tasks require continuous observe-reason-act cycles. A single request can easily balloon into 20+ LLM calls, causing token costs and API charges to spiral out of control [14].
**Mitigation:** To limit the financial blast radius, teams enforce **hard token budgets and iteration limits**. Setting limits like `max_iterations = 8` on a tool-calling loop acts as a circuit breaker to prevent a confused agent from burning tokens indefinitely [15]. Teams also use **model fallback strategies**, routing tasks to smaller, cheaper models (like GPT-4o-mini) when the primary model fails or for simpler sub-tasks [16].

**Tool-call timeouts**
Physical actions take real-world time. If the LangGraph tool expects an immediate programmatic return while the LEGO robot is still executing a 10-second drive command, the system will throw an asynchronous `TimeoutError` and crash the pipeline [17, 18].
**Mitigation:** Tool calls communicating over `pybricksdev` must implement **explicit timeout budgets** that account for physical travel time [17, 19]. Additionally, enabling **`handle_tool_errors=True`** on LangGraph's `ToolNode` ensures that if a timeout or physical stall occurs, the error is caught and returned to the LLM as a text message, allowing the agent to dynamically self-correct rather than crashing the script [20, 21].

**The safety/blast-radius question**
Powerful agents optimized purely for task success will explore any available path to achieve a goal, which can lead to the robot driving off a table or damaging itself if no physical boundaries are enforced [22]. 
**Mitigation:** Real-world deployments require strict **risk isolation and fail-safe design** [23, 24]. 
*   **Human-in-the-loop (HITL) interrupts:** The graph is configured to pause before critical or high-risk physical actions, requiring explicit human approval before the command is sent to the robot [25, 26].
*   **Output validation:** Agent outputs are constrained using Pydantic schemas, enforcing that any generated motor parameters or coordinates stay within safe, machine-readable limits [17, 24].
*   **Behavioral Analysis nodes:** Meta-cognitive nodes can be inserted into the graph to audit the agent's proposed actions against expected norms and flag unsafe maneuvers before physical execution [26].

**Cited sources:** 26 references

## Minimum viable starter project

**Project: The "Color-Seeker" Agent**

**Bill of Materials (BOM)**
*   **Hardware:** 1x SPIKE Prime Hub (flashed with Pybricks), 2x Medium Motors (for a simple drive base), 1x Color Sensor, and a PC/Mac with Bluetooth [1-3]. 
*   **Software:** Pybricks web IDE (to flash the hub), OpenAI API key or Ollama (using a 7B+ parameter model for reliable JSON tool calling), and a Python environment with `langgraph`, `langchain`, and `pybricksdev` [2, 4, 5].

**Expected Build Time: ~1.5 to 2 Hours**
*   **15 mins:** Assemble a basic two-wheel drive base with a downward-facing color sensor.
*   **15 mins:** Write and flash the simple "Body" script to the LEGO hub.
*   **60-90 mins:** Write the LangGraph orchestration script on your PC and test the BLE connection [2].

**Recommended Task: The "Observe-Reason-Act" Delivery**
Instead of a complex maze, start with a 1D problem: **"Drive forward until you see a blue line, then stop and celebrate."** 
The robot must continually poll its color sensor, send that data to the LLM, and wait for the LLM to decide whether to issue a "drive forward" command or a "stop and beep" command. 

**System Architecture & Workflow**
To make this work, you will partition the system into two halves:
1.  **The Body (Pybricks Hub):** You will write a script that runs locally on the hub. Instead of containing logic, it acts as a server listening to standard input (`usys.stdin`). If it receives a text string like `color?`, it reads the sensor and prints the result to standard output (`usys.stdout`). If it receives `fwd_5`, it turns the motors forward for 5 centimeters and prints `OK` [6, 7]. 
2.  **The Brain (PC LangGraph):** You will create a LangGraph state machine tracking a `RobotState` dictionary (containing `messages` and `current_color`) [8, 9]. 
    *   **Perception Node:** Uses `pybricksdev` over the Nordic UART Service (NUS) to asynchronously send `color?` to the hub and updates the graph state with the resulting color [5, 10].
    *   **Reason Node:** Passes the state to the LLM. If the color is not blue, the LLM calls the `drive_forward` tool. If the color is blue, the LLM calls the `stop_and_celebrate` tool.
    *   **Action Node:** Executes the LLM's tool calls, sending physical commands (`fwd_5`) to the hub over Bluetooth [10]. 
    *   **Edges:** The graph cycles `Perception -> Reason -> Action -> Perception` until the LLM decides the goal is met and routes to `END` [11, 12].

**What You Will Learn**
*   **Edge-Cloud Partitioning:** You will experience firsthand why real-time control (like balancing or basic motor driving) must live on the edge (the Pybricks hub), while deliberate, high-level reasoning lives in the cloud or on the PC. You will see the 2- to 8-second latency of an LLM call and understand why the agent cannot safely micromanage the motors in real-time [13, 14]. 
*   **Stateful Memory:** You will learn how LangGraph's `State` acts as a continuous "backpack" of data, allowing the agent to remember its past coordinates and previous color readings across multiple loop cycles without losing context [8].
*   **BLE Tool Calling:** You will learn how to wrap asynchronous Python functions using the `pybricksdev` library into `@tool` decorators, allowing the LLM to execute non-blocking Bluetooth commands [5, 10].
*   **Agentic Error Recovery:** You will learn to use LangGraph's `handle_tool_errors=True` on your `ToolNode`. If the physical robot hits a wall and the motor stalls—or if the BLE transmission drops—the system won't crash. Instead, the error is fed back into the LLM's context window as a text message, allowing the agent to dynamically attempt a retry or adjust its strategy [15, 16].

**Cited sources:** 16 references