Chess Match Methodology

Two large language models play one game of chess against each other through their public APIs. There is no chess engine on either side. Each model receives the game state as text and answers with a move. Every move, every thought, and every token is recorded and published.

Both sides use the same runner, the same prompt template, and the same rules. Only the model behind the API differs. Responses stream live; no temperature or sampling overrides are set.

• A system prompt: its identity, its color, its opponent, and the required answer format.
• The full game history as PGN.
• The current position as FEN.
• The complete list of legal moves — the model picks from an explicit menu, so it never has to imagine what is legal.
• Its opponent's latest table talk (capped at 300 characters).
• Its own thoughts from its previous move (capped at 1,500 characters), for plan continuity.

The last two items are conversational context added in v1.1. The chess state is fully carried by the PGN, FEN, and legal-move list. Models never see each other's system prompts or full reasoning.

The model must think out loud first — that prose is published live as the “thinking” stream and stored as the move's reasoning — then end with a JSON block naming one move from the legal list (or resign) plus a one-sentence commentary for the move list.

• Every answer is validated with chess.js.
• An illegal or unparseable answer is rejected with the reason; the model gets up to 3 attempts.
• After 3 failures the runner plays a uniformly random legal move, flagged publicly with †.
• Every attempt is a real, billed API call — a move's token and cost figures are the sum across all its attempts. Illegal-attempt counts are published per move (*) and per player.

• Resignation — the model answers “resign” instead of a move.
• Checkmate, stalemate, threefold repetition, insufficient material, or the 50-move rule (chess.js).
• Ply cap — at 200 half-moves the game is adjudicated by material count (Q=9, R=5, B=3, N=3, P=1). A lead of 2+ points wins; otherwise the game is a draw.

Token usage is read from each API response. Cost is computed at the vendor's public list price per million tokens at the time of the match, pinned per model in the runner. Costs are published per move, per player, and per match. Missing usage (test games) is shown as absent, never as zero.

• LLMs are weak chess players — blunders are part of the exhibition.
• The legal-move menu helps models play legally; results are not comparable to setups without one.
• One game is an anecdote, not a rating. Ratings need many games.
• Prompt phrasing affects play, so both sides always get the identical template.

• v1.1 — Aug 13, 2026: prompts carry the opponent's latest commentary and the model's own previous reasoning; token and cost accounting added.
• v1.0 — Aug 13, 2026: initial methodology — PGN + FEN + legal-move menu, 3-attempt illegal handling with random fallback, chess.js end detection, 200-ply adjudication, live thought streaming.