• A submitted manuscript is the author's version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers.

“Perhaps a thing is simple if you can describe it fully in several different ways, without immediately knowing that you are describing the same thing” – Richard Feynman

principled frameworks such as minimax, reinforcement learning, or function approximation. In addition to the elegant conceptual frameworks, deep, dirty, domain-specific understanding is necessary for progress in this field [594]. 8.1.4 Towards General Intelligence Let us revisit the problem statement from the Introduction. What are the machine learning methods that are used in Chess and Go to achieve a level of play stronger than the strongest humans? Various reinforcement learning methods have beaten human champions in Chess and Go, ranging from heuristic planning, adaptive sampling, function approximation, to self-play. All methods fit in the classic search-eval architecture, and all use variants of the principles of bootstrapping and optimization. Heuristic planning can achieve levels of play far surpassing human ability in Chess and Checkers. We have also seen how to combine planning and learning into a self-play system that learns to play from scratch, achieving even higher levels in Chess, Shogi, and Go. Did we achieve intelligence? For these three games, we certainly passed the Turing test.1 Systems were created that behave at a level for which a human would need super human intelligence. The intelligence is, however, single-domain intelligence. AlphaGo cannot play Shogi, and it cannot think of an entertaining joke. Joining the two fields of symbolic AI and connectionist AI, AlphaZero showed that the methods generalize to three games, hinting at artificial general intelligence. However, it is more precise to speak of training three almost identical systems to become three different systems specialized in their own kind of special intelligence (since the net, once trained for Go, cannot play Chess). (GGP systems can play general games, but do not achieve performance close to AlphaZero; Sect. 8.2.5.) 8.2 Future Drosophilas The reinforcement learning results in Atari andGo have inspiredmuch further research. The use of concrete games, the Drosophilas, has stimulated progress in game playing greatly. In addition to the usual two-person zero-sum perfect-information games, researchers have looked for games that capture other elements of real life, such as non-zero-sum (collaboration) games, imperfect information, and multi-agent games. In this section we will review some of these aspects. 1 Actually, since we far surpassed human play, an argument can be made that in doing so we failed the Turing test, since no human can play at this level.

o Multi-agent systems are rapidly nding applications in a variety of domains, including robotics, distributed control, telecommunications, economics. Many tasks arising in these domains require that the agents learn behaviors online. A signicant part of the research on multi-agent learning concerns reinforcement learning techniques. However, due to different viewpoints on central issues, such as the formal statement of the learning goal, a large number of different methods and approaches have been introduced. In this paper we aim to present an integrated survey of the eld. First, the issue of the multi-agent learning goal is discussed, after which a representative selection of algorithms is reviewed. Finally, open issues are identied and future research directions are outlined.

approved: Kagan Tumer Recent advances in multiagent learning have led to exciting new capabilities spanning fields as diverse as planetary exploration, air traffic control, military reconnaissance, and airport security. Such algorithms provide a tangible benefit over traditional control algorithms in that they allow fast responses, adapt to dynamic environments, and generally scale well. Unfortunately, because many existing multiagent learning methods are extensions of single agent approaches, they are inhibited by three key issues: i) they treat the actions of other agents as “environmental noise” in an attempt to simplify the problem complexity, ii) they are slow to converge in large systems as the joint action space grows exponentially in the number of agents, and iii) they frequently rely upon the presence of an accurate system model being readily available. This work addresses these three issues sequentially. First, we improve overall learning performance compared to existing state-of-the-art techniques in the field by embracing the exploration in learning rather than ignoring it or approximating it away. Within multiagent systems, exploration by individual agents significantly alters the dynamics of the environment in which all agents learn. To address this, we introduce the concept of “private” exploration, which enables each agent to present a stationary baseline policy to other agents in order to allow other agents in the system to learn more efficiently. In particular, we introduce Coordinated Learning without Exploratory Action Noise (CLEAN) rewards which improve coordination and performance by utilizing the concept of private exploration in order to remove the negative impact of traditional “public” exploration strategies from learning in multiagent systems. Next, we leverage the fundamental properties of CLEAN rewards that enable private exploration to allow agents to explore multiple potential actions concurrently in a “batch mode” in order to significantly improve learning speed over the state-of-the-art. Finally, we improve the real-world applicability of the proposed techniques by reducing their requirements. Specifically, the CLEAN rewards developed require an accurate partial model (i.e., an accurate model of the system objective) of the system in order to be computed. Unfortunately, many real-world systems are too complex to be modeled or are not known in advance, so an accurate system model is not available a priori. We address this shortcoming by employing model-based reinforcement learning techniques to enable agents to construct their own approximate model of the system objective based upon their observations and use this approximate model to calculate their CLEAN rewards. c ©Copyright by Chris HolmesParker April 15, 2013 All Rights Reserved CLEAN Learning to Improve Coordination and Scalability in Multiagent Systems

With today’s electricity grid being penetrated with more and more intermittent energy sources, the need arises to match demand for electricity with electricity generation, in order to maintain the stability of the grid. Demand response has been proposed as a mechanism that aims to solve this problem, by stimulating the shift of electricity demand towards production peaks. In previous work individual devices were trained using fitted-Q iteration to shift the consumption of electrical energy towards low-price periods, while still guaranteeing user comfort. This paper shows that controlling multiple devices with one independent agent per device is more cost-effective compared to a centralized agent. The paper expands the setting with an energy consumption constraint on the level of the household. The objective is to spread the energy consumption of the devices during low price periods as much as possible in order to avoid overloading the grid. Preliminary results for both centralized and independent learners show that these agents are capable of reducing the amount of constraint violations minimally but fail to reduce violations to zero.

With the increasing number of robots in industrial environments, scientists/technologists were often faced with issues on cooperation, coordination and collaboration among different robots and their self governance in the work space. This has led to the development of systems with several cooperative robotic agents. (Kim et al., 1997b). Generally, a system with several robotic agents (multi-robot system) is composed by two or more robots executing a task in a cooperative way (Arai and Ota, 1992). Coordination, collaboration and cooperation are three terms used without distinction when working with multi-agent and multi-robot systems. In this work, we adopt a definition proposed by Noreils (Noreils, 1993) in which cooperation occurs when several agents (or robots) are gathered together so as to perform a global task. Coordination and collaboration are two forms of cooperation (Botelho and Alami, 2000). Coordination occurs when an entity coordinates its activity with another, or it synchronizes its action with respect to the other entity, by exchanging information, signals, etc. And, collaboration occurs when two or more agents decompose a global task in subtasks to be performed by each specific agent. Generally, the solution for problems using multi-agent and multi-robot systems is divided into stages. When talking about autonomous robots, two of these stages are the task allocation stage and the task execution stage. Task allocation should be done so that all components (agents or robots) in the system are used and the problem is completely solved. The task execution stage itself should be performed so that the agents do not interfere to each other (coordination and/or collaboration) when solving the problem. Traditionally, both stages are carried out independently. In the task allocation stage, it is defined if the agents will collaborate to each other or if they will coordinate their activities. In the task execution stage, collaboration and/or coordination are effectively done. In the literature, each stage is implemented using different techniques. The task allocation stage can be implemented using centralized or decentralized approaches (Le Pape, 1990). Centralized approaches can be implemented as an optimization problem. Decentralized approaches generally use marked based approaches like the contract-net protocol (CNP) (Ulam et al., 2007) or other approaches derived from it (Botelho and Alami, 1999). The task execution stage can be implemented in many ways. It depends of the nature of interactions between agents (Collaboration or coordination) and if agents can modify or not their strategies (Static and dynamic strategies). For example it can be implemented using

With the extensive implementations of wireless sensor networks in many areas, it is imperative to have better management of the coverage and energy consumption of such networks. These networks consist of large number of sensor nodes and therefore a multi-agent system approach needs to be taken in order for a more accurate model. Three coordination algorithms are being put to the test in this paper: (i) fully distributed Q-learning which we refer to as independent learner (IL), (ii) distributed value function (DVF) and (iii) an algorithm we developed which is a variation of the IL, coordinated algorithm (COORD). The results show that the IL and DVF algorithm performed for higher sensor node densities but at low sensor node densities, the three algorithms have similar performance.

With the development of renewable energy technology, the distributed wind-PV power system has a wider application. This paper proposes a distributed wind- PV power system based on Multi-Agent, whose main character is energy management, and describes the multi-agent cooperative reinforcement learning process using the joint action learning pattern as the cooperative strategy. The experiment of a distributed wind-PV power system shows the efficiency.

With the development of global urbanization, the Internet of Things (IoT) and smart cities are becoming hot research topics. As an emerging model, edge computing can play an important role in smart cities because of its low latency and good performance. IoT devices can reduce time consumption with the help of a mobile edge computing (MEC) server. However, if too many IoT devices simultaneously choose to offload the computation tasks to the MEC server via the limited wireless channel, it may lead to the channel congestion, thus increasing time overhead. Facing a large number of IoT devices in smart cities, the centralized resource allocation algorithm needs a lot of signaling exchange, resulting in low efficiency. To solve the problem, this paper studies the joint policy of communication and computing of IoT devices in edge computing through game theory, and proposes distributed Q-learning algorithms with two learning policies. Simulation results show that the algorithm can converge quickly with a balanced solution.

With the continuous growth of machine-type devices (MTDs), it is expected that massive machine-type communication (mMTC) will be the dominant form of traffic in future wireless networks. Applications based on this technology, have fundamentally different traffic characteristics from human-to-human (H2H) communication, which involves a relatively small number of devices transmitting large packets consistently. Conversely, in mMTC applications, a very large number of MTDs transmit small packets sporadically. Therefore, conventional grant-based access schemes commonly adopted for H2H service, are not suitable for mMTC, as they incur in a large overhead associated with the channel request procedure. We propose three grantfree distributed optimization architectures that are able to significantly minimize the average power consumption of the network. The problem of physical layer (PHY) and medium access control (MAC) optimization in grant-free random access transmission is is modeled as a partially observable stochastic game (POSG) aimed at minimizing the average transmit power under a per-device delay constraint. The results show that the proposed architectures are able to achieve significantly less average latency than a baseline, while spending less power. Moreover, the proposed architectures are more robust than the baseline, as they present less variance in the performance for different system realizations.

With the advent of the fifth generation of wireless standards and an increasing demand for higher throughput, methods to improve spectral efficiency of wireless systems have become very important. In the context of cognitive radio, a substantial increase in throughput is possible if the secondary user can make smart decisions regarding which channel to sense and when or how often to sense. Here, we propose an algorithm to not only select a channel for data transmission, but also to predict how long the channel will remain unoccupied so that the time spent on channel sensing can be minimized. Our algorithm learns in two stages—a reinforcement learning approach for channel selection and a Bayesian approach to determine the duration for which sensing can be skipped. Comparisons with other methods are provided through extensive simulations. We show that the number of sensing operations is minimized with negligible increase in primary user interference; this implies that less energy is spent by the secondary user in sensing, and also higher throughput is achieved by saving the time spent on sensing.

With the advent of sequential matching (of supply and demand) systems (uber, Lyft, Grab for taxis; ubereats, deliveroo, etc for food; amazon prime, lazada etc. for groceries) across many online and offline services, individuals (taxi drivers, delivery boys, delivery van drivers, etc.) earn more by being at the "right" place at the "right" time. We focus on learning techniques for providing guidance (on right locations to be at right times) to individuals in the presence of other "learning" individuals. Interactions between indivduals are anonymous, i.e, the outcome of an interaction (competing for demand) is independent of the identity of the agents and therefore we refer to these as Anonymous MARL settings. Existing research of relevance is on independent learning using Reinforcement Learning (RL) or on Multi-Agent Reinforcement Learning (MARL). The number of individuals in aggregation systems is extremely large and individuals have their own selfish interest (of maximising revenue). Therefore, traditional MARL approaches are either not scalable or assumptions of common objective or action coordination are not viable. In this paper, we focus on improving performance of independent reinforcement learners, specifically the popular Deep Q-Networks (DQN) and Advantage Actor Critic (A2C) approaches by exploiting anonymity. Specifically, we control non-stationarity introduced by other agents using entropy of agent density distribution. We demonstrate a significant improvement in revenue for individuals and for all agents together with our learners on a generic experimental set up for aggregation systems and a real world taxi dataset.

Wireless Sensor Network (WSN) for environment monitoring, which typically has heavy data transmission load, has critical real-time requirement. Thus the time delay at relay-nodes should be reduced.  This paper models queue scheduling as a reinforcement learning process and presents a scheduling algorithm of relay-node based on self-adaptive weighted learning. The presented algorithm schedules queues dynamically. Simulation results under two circumstances (with sufficient bandwidth and limited bandwidth) show that the algorithm can improve the real-time performance and maintain fairness.

Wind farms are a crucial driver toward the generation of ecological and renewable energy. Due to their rapid increase in capacity, contemporary wind farms need to adhere to strict constraints on power output to ensure stability of the electricity grid. Specifically, a wind farm controller is required to match the farm’s power production with a power demand imposed by the grid operator. This is a non-trivial optimization problem, as complex dependencies exist between the wind turbines. State-of-the-art wind farm control typically relies on physicsbased heuristics that fail to capture the full load spectrum that defines a turbine’s health status. When this is not taken into account, the longterm viability of the farm’s turbines is put at risk. Given the complex dependencies that determine a turbine’s lifetime, learning a flexible and optimal control strategy requires a data-driven approach. However, as wind farms are large-scale multi-agent systems, optimizing control strategies over the full joint action space is intractable. We propose a new learning method for wind farm control that leverages the sparse wind farm structure to factorize the optimization problem. Using a Bayesian approach, based on multi-agent Thompson sampling, we explore the factored joint action space for configurations that match the demand, while considering the lifetime of turbines. We apply our method to a grid-like wind farm layout, and evaluate configurations using a state-ofthe-art wind flow simulator. Our results are competitive with a physicsbased heuristic approach in terms of demand error, while, contrary to the heuristic, our method prolongs the lifetime of high-risk turbines.

Whereas most work in reinforcement learning (RL) ignores the structure or relationships between actions, in this paper we show that exploiting structure in the action space can improve sample efficiency during exploration. To show this we focus on concurrent action spaces where the RL agent selects multiple actions per timestep. Concurrent action spaces are challenging to learn in especially if the number of actions is large as this can lead to a combinatorial explosion of the action space.

When multiple agents learn in a decentralized manner, the environment appears non-stationary from the perspective of an individual agent due to the exploration and learning of the other agents. Recently proposed deep multi-agent reinforcement learning methods have tried to mitigate this non-stationarity by attempting to determine which samples are from other agent exploration or suboptimality and take them less into account during learning. Based on the same philosophy, this paper introduces a decentralized quantile estimator, which aims to improve performance by distinguishing non-stationary samples based on the likelihood of returns. In particular, each agent considers the likelihood that other agent exploration and policy changes are occurring, essentially utilizing the agent’s own estimations to weigh the learning rate that should be applied towards the given samples. We introduce a formal method of calculating differences of our return distribution representations and methods for utilizing it to guide updates. We also explore the effect of risk-seeking strategies for adjusting learning over time and propose adaptive risk distortion functions which guides risk sensitivity. Our experiments, on traditional benchmarks and new domains, show our methods are more stable, sample efficient and more likely to converge to a joint optimal policy than previous methods. ACM Reference Format: Xueguang Lyu and Christopher Amato. 2020. Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning. In Proc. of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2020), Auckland, New Zealand, May 9–13, 2020, IFAAMAS, 9 pages.

When multiple agents learn in a decentralized manner, the environment appears non-stationary from the perspective of an individual agent due to the exploration and learning of the other agents. Recently proposed deep multi-agent reinforcement learning methods have tried to mitigate this non-stationarity by attempting to determine which samples are from other agent exploration or suboptimality and take them less into account during learning. Based on the same philosophy, this paper introduces a decentralized quantile estimator, which aims to improve performance by distinguishing non-stationary samples based on the likelihood of returns. In particular, each agent considers the likelihood that other agent exploration and policy changes are occurring, essentially utilizing the agent's own estimations to weigh the learning rate that should be applied towards the given samples. We introduce a formal method of calculating differences of our return distribution representations and methods for utilizing it to guide updates. We also explore the effect of risk-seeking strategies for adjusting learning over time and propose adaptive risk distortion functions which guides risk sensitivity. Our experiments, on traditional benchmarks and new domains, show our methods are more stable, sample efficient and more likely to converge to a joint optimal policy than previous methods.

When multiple agents act in the same environment, single-agent reinforcement learning (RL) techniques often fail, as they do not take into account other agents. An agent using single agent RL generally does not have sufficient information to obtain a good policy. However, multi-agent techniques that simply extend the state space to include information on the other agents suffer from a large overhead, leading to very slow learning. In this paper we describe a multi-level RL algorithm which acts independently whenever possible and learns in which states it should enrich its state information with information about other agents. Such states, which we call conflict states are detected using statistical information about expected payoffs in these states. We demonstrate through experiments that our approach learns a good trade-off between learning in the singleagent state space and learning in the multi-agent state space.

When it comes to planning for joint human-agent activities, one has to consider not only flexible plan execution and social constraints but also the dynamic nature of humans. This can be achieved by providing additional information about the characteristics of a human. As an example one need to take the physical and psychological condition of the elderly into consideration when developing collaborative applications like socially assistive robots. This work outlines Hplan, an extension to the agent-framework JIAC V, that takes this requirement into account. Hplan is strongly related to the conceptual model of dynamic planning components and integrates humans as avatars into a life cycle of planning, execution and learning.