This repository contains the code used to evaluate chess AIs, such as Leela Chess Zero and Stockfish, for consistency. The experiments are part of the paper Evaluating Superhuman Models with Consistency Checks.
The provided evaluation pipeline is optimized to support multi-GPU evaluation, where GPUs can run on remote machines, and additional GPUs can be dynamically added to an experiment after it has started.
This project was developed using Python 3.8. It is recommended to install this repository in a virtual environment. Make sure you have Python 3.8 installed on your machine. Then, initialize your virtual environment in this folder, for example via the command
python3.8 -m venv .venvYou can activate the virtual environment via the command
source .venv/bin/activateThe package can be installed via the command
pip install -e .In order to run the experiments you need access to an instance of Leela Chess Zero. You can either install it on the same machine you want to run the experiments on, or on a remote machine to which you have SSH access. Our experiments use the release/0.29 version, compiled from source and with GPU support enabled.
All weight files can be found on this website. For our experiments we used the network with ID 807785.
Configurations for the Leela Chess Zero instance must be stored in a configuration file in the experiments/configs/engine_configs folder. Each config file has to contain information about where to find the installed Leela Chess Zero instance and which configuration parameters should be set. See as example the following config:
[General]
# 'engine_type' Must be either 'local_engine' or 'remote_engine'
engine_type = remote_engine
engine_path = /path/to/lc0/on/the/machine/where/it/has/been/installed
network_base_path = /path/to/folder/where/weightfiles/are/stored
# Leela Chess Zero configs used for experiments
# See https://github.com/LeelaChessZero/lc0/wiki/Lc0-options
# for a list of all options
[EngineConfig]
Backend = cuda-fp16
VerboseMoveStats = true
SmartPruningFactor = 0
Threads = 1
TaskWorkers = 0
MinibatchSize = 1
MaxPrefetch = 0
NNCacheSize = 200000
TwoFoldDraws = false
# For how long Leela Chess Zero should evaluate a position
# See https://python-chess.readthedocs.io/en/latest/engine.html#chess.engine.Limit
# for a list of options.
[SearchLimits]
nodes = 400
# The following parameters are only required if you installed
# Leela Chess Zero on a different machine than the one you're using
# to run the experiments
[Remote]
remote_host = uri.of.server.com
remote_user = username
password_required = TrueIn addition to the engine config, our experiments also require a config file containing information where to find the input data (usually chess positions). This configuration file must be stored in the experiments/configs/data_generator_configs folder. We support either a simple .txt file containing a list of FENs, or a .pgn database containing games in PGN. All data files should be stored in the data folder. Alternatively, you can also set the DATASET_PATH environment variable in which case the data-files are expected to be stored in DATASET_PATH/chess-data. See as example the following config:
[General]
# 'data_generator_type' must be either 'fen_database_board_generator'
# (for a simple text file containing one fen per row) or
# 'database_board_generator' (for a database file in .pgn format)
data_generator_type = fen_database_board_generator
[DataGeneratorConfig]
database_name = name_of_data_file.txt
open_now = True- Chess AI instance (Leela or Stockfish) installed and configured as described above
- Data file containing chess positions stored in
datafolder. The specific chess positions used in our experiments can be extracted from the result files in theexperiments/results/final_datafolder.
All experiments can be run in a two-step process. First, the main experiment file is run. This file handles everything from loading the data, writing results, and coordinating the distributed queues. In a second step, one or several workers are started. Each worker runs a chess AI instance and evaluates positions provided by the main experiment file.
The precise commands used to replicate the experiments of the paper for Leela Chess Zero can be found in the file leela_experiment_commands.txt. The commands can be adapted accordingly for Stockfish by starting Stockfish worker threads instead of Leela worker threads.
Some simple examples of how to run the different experiments are provided below.
For the forced-move and the recommended-move experiments, the main experiment file can be run via the command
python experiments/recommended_move_invariance_testing.py --engine_config_name your_engine_config.ini --data_config_name --your_data_config.ini --num_positions number_of_positions_to_evaluateFor the board-mirroring and board-transformation experiments, the main experiment file can be run via the command
# '--transformations' must be a subset of [rot90, rot180, rot270, flip_diag, flip_anti_diag, flip_hor, flip_vert, mirror]
python experiments/transformation_invariance_testing.py --engine_config_name your_engine_config.ini --data_config_name --your_data_config.ini --num_positions number_of_positions_to_evaluate --transformations a list of transformations to apply to the boardFor the evolutionary algorithm experiments, the main experiment file can be run via the command
python experiments/evolutionary_algorithms/evolutionary_algorithm_distributed_oracle_queries_async.py For all experiments, a worker can be started via the command
python rl_testing/engine_generators/worker.py --engine_config_name your_engine_config.ini --network_name name_of_weight_file