Small LLMs in Chess: Why 30 of 37 Games Ended in Repetition
This article explains why small language models (LLMs) playing chess frequently end games in threefold repetition—a draw by repeating the same position three times—and what that reveals about how these models make decisions. The short answer: small LLMs lack the long-term planning and memory needed to convert an advantage into a checkmate, so when they cannot find a winning continuation, they fall back on the safest legal move, which often repeats the position. In a recent test, 30 of 37 games between small LLMs ended in repetition because the models prioritized avoiding losses over pursuing wins.
Background: How LLMs Play Chess
Large language models like GPT-4 can play chess at a decent club level when given a board state in Forsyth–Edwards Notation (FEN) and asked for a move. They do this by pattern-matching against millions of games in their training data. Small LLMs (under 10 billion parameters) have far less capacity to store and retrieve such patterns. They often know basic rules—how pieces move, what checkmate looks like—but they struggle with multi-step tactics and strategic planning. When a position is unfamiliar, they tend to produce plausible but shallow moves, and when they sense danger, they retreat to the most conservative option: moving a piece back and forth.
How Small LLMs Choose Moves
Small LLMs generate moves token by token, predicting the next character based on the prompt (the FEN plus the move history). They do not search ahead like traditional chess engines (e.g., Stockfish). Instead, they rely on statistical associations. For example, given a position where a knight is attacked, the model might output a move that moves the knight to a safe square—but it may not realize that the same position could arise again after a few moves. Because the model has no explicit memory of the board beyond what is in the prompt, and because the prompt length is limited, it can easily repeat a move sequence without recognizing it as a repetition.
Why Repetition Happens So Often
Threefold repetition occurs when the same position appears three times with the same player to move. In human chess, players often force a draw when they are losing. Small LLMs, however, do not calculate that far. They repeat because:
- Safety bias: The model is trained to avoid blunders, so it prefers moves that keep the position stable.
- Lack of planning: Without search, the model cannot see that a different move could lead to a win in five moves.
- Prompt truncation: Many small LLMs have a context window of 2,000–4,000 tokens. A full chess game can exceed that, so earlier moves are dropped, and the model loses track of what happened.
- No repetition detection: The model is not explicitly told to avoid threefold repetition; it only sees the current position and the last few moves.
In the 37-game test, the models often reached a middlegame where neither side had a clear plan. They shuffled pieces, and because both models had the same safety bias, they mirrored each other’s moves, leading to repetition within 20–30 moves.
How to Choose a Small LLM for Chess
If you want to experiment with small LLMs playing chess, consider these factors:
- Parameter count: Larger models (7B vs 1B) generally play better, but even 7B models are far weaker than dedicated engines.
- Training data: Models fine-tuned on chess games (e.g., a model trained on PGN files) will outperform general-purpose models.
- Context length: A longer context window helps the model remember more of the game, reducing repetition.
- Sampling settings: Lower temperature (e.g., 0.2) makes moves more deterministic, which can reduce random shuffling but also increase repetition.
Common Problems and Misconceptions
A common misconception is that small LLMs are “trying� to draw. They are not. They have no goal other than to produce a plausible next move. Repetition is an emergent behavior, not a strategy. Another problem is that many small LLMs cannot handle illegal moves—they might output a move that is not legal in the current position. In the test, games were filtered to only legal moves, which further pushed the models toward safe, repetitive play. If you let a small LLM play without a legality filter, it will often make illegal moves, and the game ends in an error rather than a draw.
Comparison with Traditional Engines
Traditional chess engines like Stockfish use alpha-beta pruning and evaluate millions of positions per second. They explicitly detect repetition and avoid it unless it is the best outcome. Small LLMs do none of this. A 7B-parameter LLM might evaluate a position in a few milliseconds, but it evaluates only one position—the current one. Engines also have a built-in draw score for repetition, so they can choose a different move if a win exists. LLMs have no such mechanism, so they happily repeat. In a match between Stockfish and a small LLM, the LLM would lose almost every game, often by checkmate, because the engine would exploit the LLM’s shallow play.
Costs and Practical Use
Running a small LLM locally is cheap—a 7B model can run on a consumer GPU with 8GB VRAM, or even on a CPU at reduced speed. The cost is time, not money. For chess, however, the cost is quality: you get a weak player that draws too much. If you want a fun demonstration of AI limitations, small LLMs are great. If you want a strong chess opponent, use a dedicated engine. Some hybrid approaches combine an LLM for move generation with a traditional engine for validation, but that defeats the purpose of studying the LLM alone.
Safety and Fairness
There is no safety risk in letting an LLM play chess, but there is a fairness issue if you use an LLM to cheat in online chess. The model is weak, so it would not help much, but any external assistance is against the rules of most chess platforms. Also, be aware that LLM outputs are not deterministic: the same position can yield different moves depending on temperature and random seed. This makes them unpredictable opponents but also unreliable for analysis. For serious study, use a standard engine with a known evaluation function.
Frequently Asked Questions
What is threefold repetition in chess?
Threefold repetition is a rule that declares a game a draw if the same position occurs three times with the same player to move. It is automatic in most chess platforms and tournament play.
Why do small LLMs repeat moves instead of trying to win?
Because they do not have a win-seeking objective. They generate moves that are statistically likely, and in a balanced position, the safest move is often a shuffle. Without search, they cannot see that a winning tactic exists.
Can a small LLM ever beat a human?
Yes, a small LLM can beat a casual human player who blunders, but it will not beat a tournament-level player. In the 37-game test, the LLMs drew most games against each other, but against a human, they would likely lose if the human plays solidly.
Does a larger LLM play better chess?
Generally, yes. Larger models have more capacity to memorize patterns and generate more accurate moves. However, even the largest LLMs are still weaker than dedicated chess engines.
How can I prevent a small LLM from repeating moves?
You can add a repetition penalty to the prompt, such as “avoid repeating the same position,� but this is not reliable. A better approach is to use a move filter that blocks any move that would lead to a repetition, forcing the model to choose something else.
Is repetition a sign that the LLM is “smart� about drawing?
No. It is a sign of the model’s inability to plan. A human player might deliberately force a draw when losing, but an LLM does not have that intention—it simply falls into a loop.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.