Controlling Graph Dynamics with Reinforcement Learning and Graph Neural Networks

We consider the problem of controlling a partiallyobserved dynamic process on a graph by a limited number of interventions. This problem naturally arises in contexts such as scheduling virus tests to curb an epidemic; targeted marketing in order to promote a product; and manually inspecting posts to detect fake news spreading on social networks. We formulate this setup as a sequential decision problem over a temporal graph process. In face of an exponential state space, combinatorial action space and partial observability, we design a novel tractable scheme to control dynamical processes on temporal graphs. We successfully apply our approach to two popular problems that fall into our framework: prioritizing which nodes should be tested in order to curb the spread of an epidemic, and influence maximization on a graph.

[1]  Shivani Agarwal,et al.  Ranking on graph data , 2006, ICML.

[2]  Laks V. S. Lakshmanan,et al.  Model-Independent Online Learning for Influence Maximization , 2017, ICML.

[3]  John S. Brownstein,et al.  Disease Surveillance on Complex Social Networks , 2016, PLoS Comput. Biol..

[4]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[5]  Jie Chen,et al.  EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs , 2020, AAAI.

[6]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[7]  Davide Eynard,et al.  Temporal Graph Networks for Deep Learning on Dynamic Graphs , 2020, ArXiv.

[8]  Albert,et al.  Emergence of scaling in random networks , 1999, Science.

[9]  Matthew Richardson,et al.  Mining the network value of customers , 2001, KDD '01.

[10]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[11]  Hassan Haghighi,et al.  INCIM: A community-based algorithm for influence maximization problem under the linear threshold model , 2016, Inf. Process. Manag..

[12]  Aravind Srinivasan,et al.  Modelling disease outbreaks in realistic urban social networks , 2004, Nature.

[13]  Charles A. Sutton,et al.  GEMSEC: Graph Embedding with Self Clustering , 2018, 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[14]  Tsuyoshi Murata,et al.  Extended methods for influence maximization in dynamic networks , 2018, Computational Social Networks.

[15]  Victor M. Preciado,et al.  Optimal Containment of Epidemics in Temporal and Adaptive Networks , 2016, ArXiv.

[16]  Chinwendu Enyioha,et al.  Optimal Resource Allocation for Network Protection Against Spreading Processes , 2013, IEEE Transactions on Control of Network Systems.

[17]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[18]  Alessandro Sperduti,et al.  Supervised neural networks for the classification of structures , 1997, IEEE Trans. Neural Networks.

[19]  Niema Moshiri The dual-Barabási-Albert model , 2018 .

[20]  Hao Zhang,et al.  Reinforcement Learning for Flexibility Design Problems , 2021, ArXiv.

[21]  Reynold Cheng,et al.  Online Influence Maximization , 2015, KDD.

[22]  Constantine Caramanis,et al.  Quarantines as a Targeted Immunization Strategy , 2020, ArXiv.

[23]  Dong Liu,et al.  A Fast and Efficient Algorithm for Mining Top-k Nodes in Complex Networks , 2017, Scientific Reports.

[24]  Anil Vullikanti,et al.  Designing Effective and Practical Interventions to Contain Epidemics , 2020, AAMAS.

[25]  Pascal Poupart,et al.  Representation Learning for Dynamic Graphs: A Survey , 2020, J. Mach. Learn. Res..

[26]  Wei Chen,et al.  Efficient influence maximization in social networks , 2009, KDD.

[27]  Liwei Wang,et al.  Deep Reinforcement Learning-Based Approach to Tackle Topic-Aware Influence Maximization , 2020, Data Science and Engineering.

[28]  Yada Zhu,et al.  Towards Fine-Grained Temporal Network Representation via Time-Reinforced Random Walk , 2020, AAAI.

[29]  International Foundation for Autonomous Agents and MultiAgent Systems ( IFAAMAS ) , 2007 .

[30]  Marcel Salathé,et al.  Dynamics and Control of Diseases in Networks with Community Structure , 2010, PLoS Comput. Biol..

[31]  Tsuyoshi Murata,et al.  Learning Adaptive Graph Protection Strategy on Dynamic Networks via Reinforcement Learning , 2018, 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI).

[32]  Éva Tardos,et al.  Maximizing the Spread of Influence through a Social Network , 2015, Theory Comput..

[33]  Jingrui He,et al.  Towards Explainable Representation of Time-Evolving Graphs via Spatial-Temporal Graph Attention Networks , 2019, CIKM.

[34]  Alessandro Sperduti,et al.  Encoding Labeled Graphs by Labeling RAAM , 1993, NIPS.

[35]  Emmanuel Abbe,et al.  Community detection and stochastic block models: recent developments , 2017, Found. Trends Commun. Inf. Theory.

[36]  J. Medlock,et al.  Optimizing Influenza Vaccine Distribution , 2009, Science.

[37]  Mark Newman,et al.  Networks: An Introduction , 2010 .

[38]  Rosalind M Eggo,et al.  Epidemic Wave Dynamics Attributable to Urban Community Structure: A Theoretical Characterization of Disease Transmission in a Large Network , 2015, Journal of medical Internet research.

[39]  Tzu-Yuan Chung Deep Reinforcement Learning-based Approach to Tackle Competitive Influence Maximization , 2019 .

[40]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[41]  Kimon Fountoulakis,et al.  Targeted pandemic containment through identifying local contact network bottlenecks , 2020, PLoS Comput. Biol..

[42]  Yuchen Li,et al.  Influence Maximization on Social Graphs: A Survey , 2018, IEEE Transactions on Knowledge and Data Engineering.

[43]  Tiejun Huang,et al.  Graph Convolutional Reinforcement Learning , 2020, ICLR.

[44]  My T. Thai,et al.  Stop-and-Stare: Optimal Sampling Algorithms for Viral Marketing in Billion-scale Networks , 2016, SIGMOD Conference.

[45]  Suh-Yin Lee,et al.  CIM: Community-Based Influence Maximization in Social Networks , 2014, TIST.

[46]  Joel C. Miller,et al.  Effective vaccination strategies for realistic social networks , 2007 .

[47]  Shou-De Lin,et al.  A Learning-based Framework to Handle Multi-round Multi-party Influence Maximization on Social Networks , 2015, KDD.

[48]  Zhitang Chen,et al.  Causal Discovery with Reinforcement Learning , 2019, ICLR.

[49]  Yang Yu,et al.  Reinforced Epidemic Control: Saving Both Lives and Economy , 2020, ArXiv.

[50]  Alessandro Sperduti,et al.  Labelling Recursive Auto-associative Memory , 1994, Connect. Sci..

[51]  Christoph Goller,et al.  Inductive Learning in Symbolic Domains Using Structure-Driven Recurrent Neural Networks , 1996, KI.

[52]  Laks V. S. Lakshmanan,et al.  CELF++: optimizing the greedy algorithm for influence maximization in social networks , 2011, WWW.

[53]  Sudip Saha,et al.  Approximation Algorithms for Reducing the Spectral Radius to Control Epidemic Spread , 2015, SDM.

[54]  Le Song,et al.  Learning Combinatorial Optimization Algorithms over Graphs , 2017, NIPS.

[55]  Tamer Basar,et al.  Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents , 2018, ICML.

[56]  Chris T. Bauch,et al.  Policy Resistance Undermines Superspreader Vaccination Strategies for Influenza , 2013, PLoS Comput. Biol..

[57]  Shie Mannor,et al.  Detecting Cascades from Weak Signatures , 2018, IEEE Transactions on Network Science and Engineering.

[58]  Csaba Szepesvári,et al.  Escaping the Gravitational Pull of Softmax , 2020, NeurIPS.

[59]  Eli A. Meirom,et al.  From Local Structures to Size Generalization in Graph Neural Networks , 2020, ICML.

[60]  Shie Mannor,et al.  The Stochastic Firefighter Problem , 2017, ArXiv.

[61]  Reuven Cohen,et al.  Efficient immunization strategies for computer networks and populations. , 2002, Physical review letters.

[62]  Ryan A. Rossi,et al.  The Network Data Repository with Interactive Graph Analytics and Visualization , 2015, AAAI.

[63]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[64]  Keith W. Ross,et al.  Efficient Entropy for Policy Gradient with Multidimensional Action Space , 2018, ICLR.

[65]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[66]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[67]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[68]  Shie Mannor,et al.  Localized Epidemic Detection in Networks with Overwhelming Noise , 2014, SIGMETRICS.

[69]  Long D. Nguyen,et al.  Risk-Aware Identification of Highly Suspected COVID-19 Cases in Social IoT: A Joint Graph Theory and Reinforcement Learning Approach , 2020, IEEE Access.

[70]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[71]  James P. Callan,et al.  Explicit Semantic Ranking for Academic Search via Knowledge Graph Embedding , 2017, WWW.

[72]  Dilip Kumar Pratihar,et al.  A survey on influence maximization in a social network , 2018, Knowledge and Information Systems.

[73]  Gary MacGillivray,et al.  AUSTRALASIAN JOURNAL OF COMBINATORICS Volume 43 (2009), Pages 57–77 The Firefighter Problem: A survey of results, directions and questions , 2022 .

[74]  X. Rodó,et al.  A modified SEIR model to predict the COVID-19 outbreak in Spain and Italy: Simulating control scenarios and multi-scale epidemics , 2020, Results in Physics.