Towards Cognitive Routing based on Deep Reinforcement Learning

Routing is one of the key functions for stable operation of network infrastructure. Nowadays, the rapid growth of network traffic volume and changing of service requirements call for more intelligent routing methods than before. Towards this end, we propose a definition of cognitive routing and an implementation approach based on Deep Reinforcement Learning (DRL). To facilitate the research of DRL-based cognitive routing, we introduce a simulator named RL4Net for DRL-based routing algorithm development and simulation. Then, we design and implement a DDPG-based routing algorithm. The simulation results on an example network topology show that the DDPG-based routing algorithm achieves better performance than OSPF and random weight algorithms. It demonstrate the preliminary feasibility and potential advantage of cognitive routing for future network.

[1]  Wen Wu,et al.  Deep Reinforcement Learning for Router Selection in Network With Heavy Traffic , 2019, IEEE Access.

[2]  Lajos Hanzo,et al.  A survey of QoS routing solutions for mobile ad hoc networks , 2007, IEEE Communications Surveys & Tutorials.

[3]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[4]  F. Richard Yu,et al.  A Survey of Machine Learning Techniques Applied to Software Defined Networking (SDN): Research Issues and Challenges , 2019, IEEE Communications Surveys & Tutorials.

[5]  Albert Cabellos-Aparicio,et al.  A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization , 2017, ArXiv.

[6]  Klaus Wehrle,et al.  Modeling and Tools for Network Simulation , 2010, Modeling and Tools for Network Simulation.

[7]  S. V. Raghavan,et al.  Survey of QoS routing , 2002 .

[8]  Celimuge Wu,et al.  Flexible, Portable, and Practicable Solution for Routing in VANETs: A Fuzzy Constraint Q-Learning Approach , 2013, IEEE Transactions on Vehicular Technology.

[9]  Charles L. Hedrick Rutgers.  An introduction to igrp , 1991 .

[10]  J. Moy,et al.  OSPF: Anatomy of an Internet Routing Protocol , 1998 .

[11]  Zoubir Mammeri,et al.  Reinforcement Learning Based Routing in Networks: Review and Classification of Approaches , 2019, IEEE Access.

[12]  Michael L. Littman,et al.  Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach , 1993, NIPS.

[13]  Chi Harold Liu,et al.  Experience-driven Networking: A Deep Reinforcement Learning based Approach , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[14]  Jingyu Wang,et al.  Toward Greater Intelligence in Route Planning: A Graph-Aware Deep Learning Approach , 2020, IEEE Systems Journal.

[15]  Xiangyu Liu,et al.  ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning , 2017, ArXiv.

[16]  Qi Hao,et al.  A Survey on Software-Defined Network and OpenFlow: From Concept to Implementation , 2014, IEEE Communications Surveys & Tutorials.

[17]  Wendi B. Heinzelman,et al.  A Survey of Routing Protocols that Support QoS in Mobile Ad Hoc Networks , 2007, IEEE Network.

[18]  Guy Pujolle,et al.  NeuRoute: Predictive dynamic routing for software-defined networks , 2017, 2017 13th International Conference on Network and Service Management (CNSM).

[19]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[20]  Alamelu Nachiappan,et al.  Q-learning based adaptive QoS routing protocol for MANETs , 2011, 2011 International Conference on Recent Trends in Information Technology (ICRTIT).

[21]  Wang Xingwei,et al.  A Data Driven Cognitive Routing Protocol for Information-Centric Networking , 2015 .

[22]  Anatolij Zubow,et al.  ns3-gym: Extending OpenAI Gym for Networking Research , 2018, ArXiv.

[23]  Junaid Qadir,et al.  Artificial intelligence based cognitive routing for cognitive radio networks , 2013, Artificial Intelligence Review.

[24]  Abdelkader Outtagarts,et al.  Deep Reinforcement Learning Based QoS-Aware Routing in Knowledge-Defined Networking , 2018, QSHINE.

[25]  M. Dzida,et al.  Optimization of The Shortest-Path Routing with Equal-Cost Multi-Path Load Balancing , 2006, 2006 International Conference on Transparent Optical Networks.

[26]  Abdul Kabbani,et al.  FlowBender: Flow-level Adaptive Routing for Improved Latency and Throughput in Datacenter Networks , 2014, CoNEXT.

[27]  Erol Gelenbe,et al.  Optimizing Secure SDN-Enabled Inter-Data Centre Overlay Networks through Cognitive Routing , 2016, 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS).

[28]  Francesca Cuomo,et al.  Routing in cognitive radio networks: Challenges and solutions , 2011, Ad Hoc Networks.

[29]  R. Mazo.  On the theory of brownian motion , 1973 .