How Increasing Complexity Challenges Large Language Models
For some reasoning tasks, an LLM’s ability to solve a puzzle depends largely on the problem’s complexity. Researchers at Apple found that large language models could solve simpler versions of the Tower of Hanoi puzzle, where disks must be moved one at a time without placing a larger disk on a smaller one. The models could also solve basic river-crossing puzzles, in which a group of people must cross a river while following specific rules.
However, the models’ performance began to decline as the number of disks or people increased beyond six. This suggests that large language models may handle straightforward logical reasoning tasks but struggle when the number of variables and required steps becomes more complex.
In a separate study, researchers from the University of Washington, Stanford University, and the Allen Institute for AI found that LLMs also had difficulty solving logical grid puzzles. These puzzles required the models to infer a person’s attributes from a series of clues. Although Apple’s research attracted widespread attention, commentators questioned whether the findings demonstrated a unique limitation of LLM reasoning or simply showed that errors become more likely as problem complexity increases.
River-Crossing Puzzle
Instructions: Using the scenario provided, create a step-by-step plan that allows everyone to cross the river while following all of the stated rules.
Source: www.technologyreview.com