Overview

The Turing Game is a symmetrized (with respect to the roles of the interrogator and interrogatee) implementation of Turing's original imitation game: two human players and one AI chatbot enter a chat setting, and the humans try to identify who is human and who is machine. The difficulty is two-fold: first, each human must correctly identify the machine, and second, the humans must coordinate to ensure that they are not both fooled by the same machine.

Two Turing Game examples showing one incorrect suspicion of a human and one correct identification of the machine.
Left: A human player incorrectly suspected a fellow human of being a machine, losing the game. Right: A human player correctly identified the machine. The game is not won yet, as the other human still has to correctly identify the machine.

Games have become testbeds for language-model agents. Work on Diplomacy combines natural-language negotiation with strategic coordination; generative-agent environments study believable social behavior in simulated communities; and text-based social-deduction settings such as AmongAgents examine reasoning under incomplete information, deception, and group deliberation.

The Turing Game connects this direction back to the imitation-game question. Its three-player format embeds one model among two humans, so the central measurement is not only whether a single answer sounds human, but whether the model remains plausible under multi-party scrutiny while the human players coordinate to identify it.

Possible Turing Game outcomes based on whether each human player correctly identifies the machine.
Possible outcomes in the Turing Game. Humans prevail if and only if both human players correctly identify the machine.

The paper reports early results from a public launch at Ars Electronica Festival 2024.

BibTeX

@inproceedings{lewandowski2024turinggame,
  title = {The Turing Game},
  author = {Lewandowski, Michal and Schmid, Simon Lucas and Mederitsch, Patrick and Aufreiter, Alexander and Aichinger, Gregor and Nessler, Felix and Bergsmann, Severin and Szolga, Viktor and Halmdienst, Tobias and Nessler, Bernhard},
  booktitle = {NeurIPS 2024 Workshop on System-2 Reasoning at Scale},
  year = {2024},
  url = {https://openreview.net/forum?id=YiquAWqFw3}
}