MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models

MazeEval tests whether a language model can navigate a maze through a movement tool while keeping track of spatial information. Hafsteinn Einarsson compares eight models on the same generated mazes with English and Icelandic instructions. The task uses coordinates and distances to walls rather than pictures, allowing a controlled comparison of sequential navigation under two linguistic framings.^[1]

Navigation task

The benchmark generates square mazes with a depth-first-search algorithm, each with one solution path. At each step the model receives its current coordinates, the goal coordinates, the number of cells visible before a wall in each cardinal direction, and the complete history of visited positions and their observations. It selects north, south, east or west through a function call. Thus, a model has access to its travel history even when it fails to use that history effectively; this is not a test of navigation without external memory.^[1]

The standard evaluation uses five fixed-seed mazes per size, from 5 × 5 through 15 × 15. A run terminates on reaching the goal, reaching the repeated-cell limit of ten visits, or exhausting a movement budget of three times the number of grid cells. Testing a model stops at the first size where it fails all five mazes. English and Icelandic runs use identical maze configurations.^[1]

Results and their denominators

The eight-model comparison includes Claude Opus 4, Claude Sonnet 4, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o, GPT-4o-mini, GPT-4.1-mini and o3. The paper reports that o3 solves all standard mazes in both languages. Other models deteriorate as maze size increases. All observed failures hit the repeated-cell limit rather than the overall movement budget: they become stuck revisiting locations despite receiving the navigation history.^[1]

Table 1 measures the largest size at which at least one of five mazes was solved, rather than reliable success at that size. For Claude Sonnet 4, this maximum is 12 × 12 in English and 8 × 8 in Icelandic; for Claude Opus 4 and Gemini 2.5 Pro, it is 11 × 11 versus 8 × 8. Additional exploratory o3 trials solve a 30 × 30 maze and fail a 40 × 40 maze in each language. Those larger sizes did not receive the repeated standard evaluation and do not establish perfect performance on arbitrary large mazes.^[1]

The reported aggregate language comparison favors English (Wilcoxon W = 273, p < 0.001). None of the eight individual-model comparisons passes the paper’s Bonferroni-adjusted threshold of 0.00625. Navigation-efficiency plots also condition on successful runs: as difficulty increases, their averages describe a selected subset of solvable mazes, not all attempts.^[1]

What the comparison establishes

MazeEval connects Distributional Shift with tool-using agents: changing the language of instructions can accompany different success rates even when the underlying mazes are held fixed. However, the two-language comparison does not isolate training-data quantity or an internal spatial mechanism as the cause. The author’s claim that a novel benchmark rules out contamination also exceeds what task novelty alone can establish.^[1]

The documented result is navigation behavior in a small, controlled text-and-tool environment. It does not establish real-world robotic reliability or consciousness. Assessing Consciousness-Related Behaviors Using the Maze Test uses a different maze setup and interpretation; sharing a task family does not make their measurements interchangeable.