Navigating the landscape of multiplayer games

Multiplayer games have long been used as testbeds in artificial intelligence research, aptly referred to as the Drosophila of artificial intelligence. Traditionally, researchers have focused on using well-known games to build strong agents. This progress, however, can be better informed by characterizing games and their topological landscape. Tackling this latter question can facilitate understanding of agents and help determine what game an agent should target next as part of its training. Here, we show how network measures applied to response graphs of large-scale games enable the creation of a landscape of games, quantifying relationships between games of varying sizes and characteristics. We illustrate our findings in domains ranging from canonical games to complex empirical games capturing the performance of trained agents pitted against one another. Our results culminate in a demonstration leveraging this information to generate new and interesting games, including mixtures of empirical games synthesized from real world games.

[1]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.

[2]  J. Robinson.  AN ITERATIVE METHOD OF SOLVING A GAME , 1951, Classics in Game Theory.

[3]  Arthur L. Samuel,et al.  Programming Computers to Play Games , 1960, Adv. Comput..

[4]  M. Olson,et al.  The Logic of Collective Action , 1965 .

[5]  G. Hardin,et al.  Tragedy of the Commons , 1968 .

[6]  I. D. Hill,et al.  Faster than Thought. A Symposium on Digital Computing Machines , 1972 .

[7]  J M Smith,et al.  Evolution and the theory of games , 1976 .

[8]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[9]  L. Vygotsky.  Interaction between learning and development , 1978 .

[10]  Wim B. G. Liebrand,et al.  A Classification of Social Dilemma Games , 1983 .

[11]  King-Sun Fu,et al.  A distance measure between attributed relational graphs for pattern recognition , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[12]  John Scott.  What is social network analysis , 2010 .

[13]  B. Mohar.  THE LAPLACIAN SPECTRUM OF GRAPHS y , 1991 .

[14]  J. Elman.  Learning and development in neural networks: the importance of starting small , 1993, Cognition.

[15]  Stanley Wasserman,et al.  Social Network Analysis: Methods and Applications , 1994 .

[16]  Terence D. Sanger,et al.  Neural network learning control of robot manipulators using gradually increasing task difficulty , 1994, IEEE Trans. Robotics Autom..

[17]  Jörgen W. Weibull,et al.  Evolutionary Game Theory , 1996 .

[18]  J. McCarthy.  AI as Sport , 1997, Science.

[19]  Josef Hofbauer,et al.  Evolutionary Games and Population Dynamics , 1998 .

[20]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[21]  Herbert Gintis,et al.  Game Theory Evolving: A Problem-Centered Introduction to Modeling Strategic Interaction - Second Edition , 2009 .

[22]  Jacob Feldman,et al.  Minimization of Boolean complexity in human concept learning , 2000, Nature.

[23]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[24]  Yishay Mansour,et al.  Nash Convergence of Gradient Dynamics in General-Sum Games , 2000, UAI.

[25]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Jonathan Schaeffer,et al.  A Gamut of Games , 2001, AI Mag..

[27]  Michael William Newman,et al.  The Laplacian spectrum of graphs , 2001 .

[28]  Murray Campbell,et al.  Deep Blue , 2002, Artif. Intell..

[29]  Xiaofeng Wang,et al.  Reinforcement Learning to Play an Optimal Nash Equilibrium in Team Markov Games , 2002, NIPS.

[30]  G. Tesauro,et al.  Analyzing Complex Strategic Interactions in Multi-Agent Systems , 2002 .

[31]  Andrew Byde.  Applying evolutionary game theory to auction mechanism design , 2003, EC '03.

[32]  Alvaro Francisco Huertas-Rosero.  A Cartography for 2x2 Symmetric Games , 2003, ArXiv.

[33]  Avrim Blum,et al.  Planning in the Presence of Cost Functions Controlled by an Adversary , 2003, ICML.

[34]  M. Nowak,et al.  Evolutionary Dynamics of Biological Games , 2004, Science.

[35]  Peter McBurney,et al.  An evolutionary game-theoretic comparison of two double-auction market designs , 2004, AAMAS'04.

[36]  Debora Donato,et al.  Large scale properties of the Webgraph , 2004 .

[37]  William V. Wright,et al.  A Theory of Fun for Game Design , 2004 .

[38]  Georgios N. Yannakakis,et al.  Evolving opponents for interesting interactive computer games , 2004 .

[39]  Drew Fudenberg,et al.  Evolutionary game dynamics in finite populations , 2004, Bulletin of mathematical biology.

[40]  D. Fudenberg,et al.  Emergence of cooperation and evolutionary stability in finite populations , 2004, Nature.

[41]  L. da F. Costa,et al.  Characterization of complex networks: A survey of measurements , 2005, cond-mat/0505185.

[42]  Lior Rokach,et al.  Clustering Methods , 2005, The Data Mining and Knowledge Discovery Handbook.

[43]  D. Bonchev,et al.  Complexity in chemistry, biology, and ecology , 2005 .

[44]  D. Fudenberg,et al.  Evolutionary cycles of cooperation and defection. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[45]  D. Pham,et al.  Selection of K in K-means clustering , 2005 .

[46]  Michael R. Genesereth,et al.  General Game Playing: Overview of the AAAI Competition , 2005, AI Mag..

[47]  D. Robinson,et al.  The topology of the 2x2 games : a new periodic table , 2005 .

[48]  David Robinson,et al.  Topology of 2x2 Games , 2005 .

[49]  M. Nowak,et al.  Stochastic dynamics of invasion and fixation. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[50]  A. Lesne.  Complex Networks: from Graph Theory to Biology , 2006 .

[51]  L. Imhof,et al.  Stochasticity and evolutionary stability. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[52]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[53]  Drew Fudenberg,et al.  Imitation Processes with Small Mutations , 2004, J. Econ. Theory.

[54]  Michael P. Wellman.  Methods for Empirical Game-Theoretic Analysis , 2006, AAAI.

[55]  Rémi Coulom,et al.  Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search , 2006, Computers and Games.

[56]  V. Latora,et al.  Complex networks: Structure and dynamics , 2006 .

[57]  Joe Marks,et al.  Automatic Design of Balanced Board Games , 2007, AIIDE.

[58]  C. Hauert,et al.  Via Freedom to Coercion: The Emergence of Costly Punishment , 2007, Science.

[59]  John H. Miller,et al.  Complex adaptive systems - an introduction to computational models of social life , 2009, Princeton studies in complexity.

[60]  Michael Mateas,et al.  Towards Automated Game Design , 2007, AI*IA.

[61]  George Sugihara,et al.  Complex systems: Ecology for bankers , 2008, Nature.

[62]  Julian Togelius,et al.  An experiment in automatic game design , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[63]  Thomas Wilhelm,et al.  What is a complex graph , 2008 .

[64]  M. Vitevitch.  What can graph theory tell us about word learning and lexical retrieval? , 2008, Journal of speech, language, and hearing research : JSLHR.

[65]  Xiaotie Deng,et al.  Settling the complexity of computing two-player Nash equilibria , 2007, JACM.

[66]  Paul W. Goldberg,et al.  The Complexity of Computing a Nash Equilibrium , 2009, SIAM J. Comput..

[67]  Craig Boutilier,et al.  Regret-based Reward Elicitation for Markov Decision Processes , 2009, UAI.

[68]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[69]  Berkeley,et al.  Critical transitions in nature and society , 2009, Choice Reviews Online.

[70]  P. Dayan,et al.  Flexible shaping: How learning in small steps helps , 2009, Cognition.

[71]  Gillian Smith,et al.  Analyzing the expressive range of a level generator , 2010, PCGames@FDG.

[72]  Julian Togelius,et al.  Towards Automatic Personalized Content Generation for Platform Games , 2010, AIIDE.

[73]  Maarten van Steen,et al.  Graph Theory and Complex Networks: An Introduction , 2010 .

[74]  K. Sigmund.  The Calculus of Selfishness , 2010 .

[75]  Matthias Dehmer.  Structural Analysis of Complex Networks , 2010 .

[76]  Reinhard Schneider,et al.  Using graph theory to analyze biological networks , 2011, BioData Mining.

[77]  Dima Shepelyansky,et al.  Spectral properties of the Google matrix of the World Wide Web and other directed networks , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[78]  William H. Sandholm,et al.  Population Games And Evolutionary Dynamics , 2010, Economic learning and social evolution.

[79]  Frédéric Maire,et al.  Evolutionary Game Design , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[80]  Vincent Conitzer,et al.  A double oracle algorithm for zero-sum security games on graphs , 2011, AAMAS.

[81]  Michael Mateas,et al.  Answer Set Programming for Procedural Content Generation: A Design Space Approach , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[82]  Julian Togelius,et al.  Search-Based Procedural Content Generation: A Taxonomy and Survey , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[83]  Srinivasan Parthasarathy,et al.  Symmetrizations for clustering directed graphs , 2011, EDBT/ICDT '11.

[84]  Simon Colton,et al.  Multi-faceted evolution of simple arcade games , 2011, 2011 IEEE Conference on Computational Intelligence and Games (CIG'11).

[85]  Hao Wang,et al.  Game reward systems: Gaming experiences and social meanings , 2011, DiGRA Conference.

[86]  César A. Hidalgo,et al.  The Atlas of Economic Complexity: Mapping Paths to Prosperity , 2011 .

[87]  Herbert A. Simon,et al.  Computer Science as Empirical Inquiry , 2011 .

[88]  Constantinos Daskalakis,et al.  On the complexity of approximating a Nash equilibrium , 2011, SODA '11.

[89]  Peter A. Flach,et al.  A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss C` Esar Ferri , 2012 .

[90]  Danai Koutra,et al.  A Scalable Approach to Size-Independent Network Similarity , 2012 .

[91]  Bryan Bruns,et al.  Escaping Prisoner's Dilemmas: From Discord to Harmony in the Landscape of 2x2 Games , 2012, ArXiv.

[92]  A. Rubinstein,et al.  The 11-20 Money Request Game: A Level-k Reasoning Study , 2012 .

[93]  N. Lazzaro.  Why We Play : Affect and the Fun of Games—Designing Emotions for Games, Entertainment Interfaces, and Interactive Products , 2012 .

[94]  F. C. Santos,et al.  Emergence of fairness in repeated group interactions. , 2012, Physical review letters.

[95]  Julian Togelius,et al.  Automatic generation and analysis of physics-based puzzle games , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[96]  Michael P. Wellman,et al.  Analyzing Incentives for Protocol Compliance in Complex Domains: A Case Study of Introduction-Based Routing , 2013, ArXiv.

[97]  Michalis Vazirgiannis,et al.  Clustering and Community Detection in Directed Networks: A Survey , 2013, ArXiv.

[98]  G. Caldarelli,et al.  Economic complexity: Conceptual grounding of a new metrics for global competitiveness , 2013 .

[99]  Danai Koutra,et al.  DELTACON: A Principled Massive-Graph Similarity Function , 2013, SDM.

[100]  S. Barry Cooper,et al.  Digital Computers Applied to Games , 2013 .

[101]  Michael H. Bowling,et al.  Solving Imperfect Information Games Using Decomposition , 2013, AAAI.

[102]  Characteristics of Generatable Games , 2014 .

[103]  Sebastian Deterding,et al.  The Lens of Intrinsic Skill Atoms: A Method for Gameful Design , 2015, Hum. Comput. Interact..

[104]  Neil Burch,et al.  Heads-up limit hold’em poker is solved , 2015, Science.

[105]  Kevin Waugh,et al.  Solving Games with Functional Regret Estimation , 2014, AAAI Workshop: Computer Poker and Imperfect Information.

[106]  Bryan Randolph Bruns,et al.  Names for Games: Locating 2 × 2 Games , 2015, Games.

[107]  David Silver,et al.  Fictitious Self-Play in Extensive-Form Games , 2015, ICML.

[108]  Branislav Bosanský,et al.  Combining Compact Representation and Incremental Generation in Large Games with Sequential Strategies , 2015, AAAI.

[109]  Krzysztof R. Apt,et al.  A classification of weakly acyclic games , 2015 .

[110]  Julian Togelius,et al.  Towards generating arcade game rules with VGDL , 2015, 2015 IEEE Conference on Computational Intelligence and Games (CIG).

[111]  Karl Tuyls,et al.  Evolutionary Dynamics of Multi-Agent Learning: A Survey , 2015, J. Artif. Intell. Res..

[112]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract) , 2012, IJCAI.

[113]  Kristinn R. Thórisson,et al.  Why Artificial Intelligence Needs a Task Theory - And What It Might Look Like , 2016, AGI.

[114]  Jakub Kowalski,et al.  Evolving Chess-like Games Using Relative Algorithm Performance Profiles , 2016, EvoApplications.

[115]  José Hernández-Orallo,et al.  Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement , 2017, Artificial Intelligence Review.

[116]  Alexiei Dingli,et al.  Platformer level design for player believability , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[117]  Carl Veller,et al.  Finite-population evolution with rare mutations in asymmetric games , 2015, J. Econ. Theory.

[118]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[119]  Julian Togelius,et al.  Rules and Mechanics , 2016 .

[120]  Julian Togelius,et al.  Procedural Content Generation in Games , 2016, Computational Synthesis and Creative Systems.

[121]  José Hernández-Orallo,et al.  The Measure of All Minds: Evaluating Natural and Artificial Intelligence , 2017 .

[122]  Simon Colton,et al.  The ANGELINA Videogame Design System—Part II , 2017, IEEE Transactions on Computational Intelligence and AI in Games.

[123]  Alex Graves,et al.  Automated Curriculum Learning for Neural Networks , 2017, ICML.

[124]  Simon Colton,et al.  The ANGELINA Videogame Design System—Part I , 2017, IEEE Transactions on Computational Intelligence and AI in Games.

[125]  Iyad Rahwan,et al.  Cooperating with machines , 2017, Nature Communications.

[126]  Pieter Abbeel,et al.  Reverse Curriculum Generation for Reinforcement Learning , 2017, CoRL.

[127]  F. C. Santos,et al.  Stochastic Dynamics through Hierarchically Embedded Markov Chains. , 2017, Physical review letters.

[128]  David Silver,et al.  A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning , 2017, NIPS.

[129]  Joel Z. Leibo,et al.  Multi-agent Reinforcement Learning in Sequential Social Dilemmas , 2017, AAMAS.

[130]  José Hernández-Orallo,et al.  A New AI Evaluation Cosmos: Ready to Play the Game? , 2017, AI Mag..

[131]  Emmanuel Müller,et al.  NetLSD: Hearing the Shape of a Graph , 2018, KDD.

[132]  Marlos C. Machado,et al.  Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (Extended Abstract) , 2018, IJCAI.

[133]  Thore Graepel,et al.  The Mechanics of n-Player Differentiable Games , 2018, ICML.

[134]  Julian Togelius,et al.  Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation , 2018, 1806.10729.

[135]  Joel Z. Leibo,et al.  A Generalised Method for Empirical Game Theoretic Analysis , 2018, AAMAS.

[136]  Mark W. Youngblood,et al.  Author Correction: Integrated genomic analyses of de novo pathways underlying atypical meningiomas , 2018, Nature Communications.

[137]  Kristinn R. Thórisson,et al.  Task Analysis for Teaching Cumulative Learners , 2018, AGI.

[138]  Julian Togelius,et al.  Artificial Intelligence and Games , 2018, Springer International Publishing.

[139]  Thore Graepel,et al.  Re-evaluating evaluation , 2018, NeurIPS.

[140]  D. Weinshall,et al.  Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks , 2018, ICML.

[141]  Demis Hassabis,et al.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , 2018, Science.

[142]  Francisco C. Santos,et al.  Social norm complexity and past reputations in the evolution of cooperation , 2018, Nature.

[143]  Csaba Szepesvári,et al.  Bounds and dynamics for empirical game theoretic analysis , 2019, Autonomous Agents and Multi-Agent Systems.

[144]  Sriram Srinivasan,et al.  OpenSpiel: A Framework for Reinforcement Learning in Games , 2019, ArXiv.

[145]  Joel Z. Leibo,et al.  Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research , 2019, ArXiv.

[146]  Max Jaderberg,et al.  Open-ended Learning in Symmetric Zero-sum Games , 2019, ICML.

[147]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[148]  Shimon Whiteson,et al.  Multi-Agent Common Knowledge Reinforcement Learning , 2018, NeurIPS.

[149]  R. Munos,et al.  Multiagent Evaluation under Incomplete Information , 2019, NeurIPS.

[150]  Julian Togelius,et al.  Intentional computational level design , 2019, GECCO.

[151]  Julian Togelius,et al.  General Video Game AI: A Multitrack Framework for Evaluating Agents, Games, and Content Generation Algorithms , 2018, IEEE Transactions on Games.

[152]  Guy Lever,et al.  Emergent Coordination Through Competition , 2019, ICLR.

[153]  Jeff Clune,et al.  AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence , 2019, ArXiv.

[154]  Christos H. Papadimitriou,et al.  α-Rank: Multi-Agent Evaluation by Evolution , 2019, Scientific Reports.

[155]  Jakub W. Pachocki,et al.  Dota 2 with Large Scale Deep Reinforcement Learning , 2019, ArXiv.

[156]  Daphna Weinshall,et al.  On The Power of Curriculum Learning in Training Deep Networks , 2019, ICML.

[157]  Rui Wang,et al.  Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions , 2019, ArXiv.

[158]  Stephen J. Roberts,et al.  Optimising Worlds to Evaluate and Influence Reinforcement Learning Agents , 2019, AAMAS.

[159]  Julian Togelius,et al.  Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning , 2019, IJCAI.

[160]  Michael P. Wellman,et al.  Iterated Deep Reinforcement Learning in Games: History-Aware Training for Improved Stability , 2019, EC.

[161]  Taehoon Kim,et al.  Quantifying Generalization in Reinforcement Learning , 2018, ICML.

[162]  Julian Togelius,et al.  Mech-Elites: Illuminating the Mechanic Space of GVG-AI , 2020, FDG.

[163]  Rahul Savani,et al.  Robust Market Making via Adversarial Reinforcement Learning , 2020, IJCAI.

[164]  Max Jaderberg,et al.  Real World Games Look Like Spinning Tops , 2020, NeurIPS.

[165]  Igor Mordatch,et al.  Emergent Tool Use From Multi-Agent Autocurricula , 2019, ICLR.

[166]  Kevin Leyton-Brown,et al.  A Formal Separation Between Strategic and Nonstrategic Behavior , 2018, EC.

[167]  Daniel Hennes,et al.  Neural Replicator Dynamics: Multiagent Learning via Hedging Policy Gradients , 2020, AAMAS.

[168]  N. Heess,et al.  A Generalized Training Approach for Multiagent Learning , 2019, ICLR.

[169]  ORF Capture-Seq as a versatile method for targeted identification of full-length isoforms , 2020, Nature Communications.

[170]  S. Risi,et al.  Increasing generality in machine learning through procedural content generation , 2019, Nature Machine Intelligence.

[171]  Martin A. Nowak,et al.  Evolving cooperation in multichannel games , 2020, Nature Communications.

[172]  Joel Lehman,et al.  Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions , 2020, ICML.

[173]  Peter Stone,et al.  Balancing Individual Preferences and Shared Objectives in Multiagent Reinforcement Learning , 2020, International Joint Conference on Artificial Intelligence.

[174]  Michael P. Wellman,et al.  Structure Learning for Approximate Solution of Many-Player Games , 2020, AAAI.