Building a Chess-Playing Robot

Overview

What does it take to build a robot that can play a complete game of chess on a real board?

Students Leon Kušić, Frane Ninić, and Bence Pintér built an end-to-end chess system combining computer vision, deep learning, game-tree search, and robotics.

A human plays White on a custom laser-cut chessboard. An overhead camera observes the board and determines which move was made — without sensors in either the board or the pieces. A custom-trained neural-network chess engine chooses a response, and a 6-axis robotic arm physically executes the move.

The system supports normal moves, captures, castling, en passant and promotion, and can even reset the board when the game is finished.

Seeing the board

One of the main challenges is surprisingly fundamental: how does the robot know what the human just played?

There are no RFID tags, magnetic sensors or switches hidden in the chessboard. Instead, the system relies entirely on an overhead camera.

The board is first calibrated by detecting its corners and reconstructing the full 8×8 grid. Each square is then perspective-corrected into a fixed-size image patch so that it can be compared consistently.

After the human makes a move, the vision pipeline examines changes between the previous and current board state. Pixel differences and colour information help determine which squares changed. The resulting source and destination squares are then checked against the legal moves in the current chess position.

This allows the software not only to recognize ordinary moves, but also to distinguish special cases such as captures, castling and en passant.

Building the physical game

The physical parts of the system were developed alongside the software.

The students designed and produced a laser-cut chessboard, 3D-printed chess pieces, and a custom camera mount. The robotic arm uses piece-specific gripping parameters and movement coordinates so that it can pick up and place the different pieces reliably.

Training a chess engine from scratch

The robot does not simply connect to an existing chess engine.

The team trained its own ResNet-based neural network on millions of chess positions. The network evaluates the current position through two outputs: a policy head, which estimates promising moves, and a value head, which evaluates the position itself.

The neural network is combined with Monte Carlo Tree Search (MCTS). Rather than immediately playing the highest-scoring move from the network, MCTS explores possible continuations and uses the neural network to guide that search.

Together, these components form the robot’s chess engine and determine the response to the human player’s move.

From a move to a robot action

Once a response has been selected, it has to be translated from chess notation into physical motion.

The arm controller converts board squares into coordinates and executes the corresponding pick-and-place sequence. Captures require removing the opponent’s piece first, while castling involves moving two pieces. Promotion and en passant require their own physical routines as well.

Camera processing, chess logic, the user interface and robot movement run in a multithreaded architecture, allowing the different parts of the system to operate without unnecessarily blocking one another.

The students also built a PySide6 desktop application that brings the system together. It shows the live camera feed and board detection, manages the chess clocks and game controls, displays previous moves and provides debugging information while the game is running.

Playing against the robot

The result is a complete physical human-versus-AI chess game: the player moves a real piece, the system understands the move visually, the AI decides how to respond, and the robot plays its move on the same board.

Here is the system playing a full game:

Replay the game

You can step through the game move by move below.

The project brings together several areas our students encounter throughout Applied Data Science & Artificial Intelligence — computer vision, deep learning, search algorithms, software engineering and robotics — in a system where every prediction ultimately has to work in the physical world.