New Zealand


We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. 
Theoretically, actor-critic algorithms have discounting for both the actor and critic,
i.e., there is a $\gamma^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function.
Practitioners, however, usually ignore the discounting ($\gamma^t$) for the actor while still using a discounted critic.
We investigate this mismatch in two scenarios.
In the first one, 
we consider optimizing an undiscounted objective $(\gamma = 1)$ where $\gamma^t$ disappears naturally $(1^t = 1)$. 
We then propose to interpret the discounting in the critic in terms of a bias-variance-representation trade-off and provide supporting empirical results.
In the second scenario,
we consider optimizing a discounted objective ($\gamma &lt; 1$) and propose to interpret the omission of the discounting in the actor update
from an auxiliary task perspective and provide supporting empirical results.

AAMAS 2022

A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

discount factor

actor critic

reinforcement learning

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

The AAMAS conference series was initiated in 2002 in Bologna, Italy as a joint event comprising the 6th International Conference on Autonomous Agents (AA), the 5th International Conference on Multiagent Systems (ICMAS), and the 9th International Workshop on Agent Theories, Architectures, and Languages (ATAL).

Subsequent AAMAS conferences have been held in Melbourne, Australia (July 2003), New York City, NY, USA (July 2004), Utrecht, The Netherlands (July 2005), Hakodate, Japan (May 2006), Honolulu, Hawaii, USA (May 2007), Estoril, Portugal (May 2008), Budapest, Hungary (May 2009), Toronto, Canada (May 2010), Taipei, Taiwan (May 2011), Valencia, Spain (June 2012), Minnesota, USA (May 2013), Paris, France (May 2014), Istanbul, Turkey (May 2015), Singapore (May 2016), São Paulo (2017) and Stockholm, Sweden (2018), Montreal (May 2019), Auckland (May 2020, Virtual), London (May 2021, Virtual).
<br>
<br>



A ticket is required to attend this event, please register using the link below:

https://aamas2022-conference.auckland.ac.nz/attending/registration/

Registration Is Required

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).


We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. 
Theoretically, actor-critic algorithms have discounting for both the actor and critic,
i.e., there is a $\gamma^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function.
Practitioners, however, usually ignore the discounting ($\gamma^t$) for the actor while still using a discounted critic.
We investigate this mismatch in two scenarios.
In the first one, 
we consider optimizing an undiscounted objective $(\gamma = 1)$ where $\gamma^t$ disappears naturally $(1^t = 1)$. 
We then propose to interpret the discounting in the critic in terms of a bias-variance-representation trade-off and provide supporting empirical results.
In the second scenario,
we consider optimizing a discounted objective ($\gamma < 1$) and propose to interpret the omission of the discounting in the actor update
from an auxiliary task perspective and provide supporting empirical results.

technical paper


Despite recent advances in RL, many problems, especially real-world tasks, remain prohibitively expensive to learn. To address this issue, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum to learn a problem that may otherwise be too difficult to learn from scratch. However, generating and optimizing a curriculum in a realistic scenario still requires extensive interactions with the environment. To address this challenge, we formulate the curriculum transfer problem, in which the schema of a curriculum optimized in a simpler, easy-to-solve environment (e.g., a grid world) is transferred to a complex, realistic scenario (e.g., a physics-based robotics simulation or the real world). We present "ACuTE" (Automatic Curriculum Transfer from Simple to Complex Environments), a novel framework to solve this problem, and evaluate our proposed method by comparing it to other baseline approaches (e.g., domain adaptation) designed to speed up learning. We observe that our approach produces improved jumpstart and time-to-threshold performance when adding task elements that further increase the difficulty of the realistic scenario. Finally, we demonstrate our approach is independent of the learning algorithm used for curriculum generation, and is Sim2Real transferrable to a real world scenario using a physical robot.

ACuTE: Automatic Curriculum Transfer from Simple to Complex Environments


Recent studies show that Deep Reinforcement Learning (DRL) models are vulnerable to
adversarial attacks. Most of these adversarial attacks have been conducted against DRL
by adding small perturbations to the observations. However, some attacks hold 
strong assumptions on the availability of the victim model, and some other attacks require a huge
amount of computation, which makes them less feasible for real world applications. In this work,
we expand on the existing literature by studying other aspects of performing such
attacks and provide a further exploration for vulnerabilities of DRL with realistic and efficient attacks. First, for the
plausible scenario where the attacker does not have access to the model parameters, we 
adapt and propose efficient black-box attack techniques to DRL settings. Second, to 
address the high computational demands of existing attacks, we introduce an efficient online 
sequential attacks that exploits temporal consistency across consecutive steps. Third, 
we explore the possibility of an attacker perturbing other quantities in the DRL 
setting, such as the environment dynamics. Finally, to account for imperfections in 
how an attacker would inject a perturbation in the physical world, we devise a method
for generating a robust physical perturbation to be printed, placed under various physical 
conditions, and observed by a real-world agent from different viewpoints. We conduct
extensive experiments both in simulation such as Atari games, robotics and
autonomous driving, and on real-world robotics, to compare the effectiveness of
the proposed attacks with several baseline approaches. To the best of our knowledge, we are the first to apply adversarial attacks on DRL systems to physical robots.

Characterizing Attacks on Deep Reinforcement Learning


Ridge Rider (RR) is an algorithm for finding diverse solutions to optimization problems by following eigenvectors of the Hessian (“ridges”). RR is designed for conservative gradient systems (i.e., settings involving a single loss function), where it branches at saddles — easy-to-find bifurcation points. We generalize this idea to non-conservative, multi-agent gradient systems by proposing a method – denoted Generalized Ridge Rider (GRR) – for finding arbitrary bifurcation points. We give theoretical motivation for our method by leveraging machinery from the field of dynamical systems. We construct novel toy problems where we can visualize new phenomena while giving insight into high-dimensional problems of interest. Finally, we empirically evaluate our method by finding diverse solutions in the iterated prisoners’ dilemma and relevant machine learning problems including generative adversarial networks.

Lyapunov Exponents for Diversity in Differentiable Games


Learning from Demonstrations (LfD) refers to using expert demonstrations combined with the reward information given by the environment to jointly guide the learning of policy in Reinforcement Learning. Previous LfD methods usually assume that provided demonstrations are perfect. While in real-world applications, demonstrations are often collected from multiple sources, which may contain imperfect ones. In this work, we aim to deal with the latter situation, i.e., Learning from Imperfect Demonstrations (LfID), where demonstrations only include trajectories with state-action pairs. To this end, two challenges need to be solved: evaluation for the demonstrations and calibration for the bonus model. Both challenges can be more severe in sparse reward environments, since the exploration problem will appear while learning. In this work, we focus on bridging the exploration and LfID problems in view of anomaly detection, and further proposing AGPO method to deal with these problems. Compared with state-of-the-art methods, empirical studies on some continuous control benchmarks show the superiority of AGPO in this scenario.

Anomaly Guided Policy Learning from Imperfect Demonstrations


As reinforcement learning (RL) systems are deployed in various safety-critical applications, it is imperative to understand how vulnerable they are to adversarial attacks. Of these, an environment-poisoning attack (EPA) is considered particularly insidious, since environment hyper-parameters are significant factors in determining an RL policy, yet prone to be accessed by third parties. The success of existing EPAs relies on comprehensive prior knowledge of the attacked RL system, including RL agent’s learning mechanism and/or its environment model. Unfortunately, such an assumption of prior knowledge creates an unrealistic attack, one that poses little threat to real-world RL systems.

In this paper, we propose a Double-Black-Box EPA framework, only assuming the attacker's ability to alter environment hyper-parameters. Considering that environment alteration comes at a cost, we seek minimal poisoning in an unknown environment, that aims to force a black-box RL agent to learn an attacker-designed policy. To this end, we incorporate an inference module in our framework to capture the internal information of the unknown RL system and, accordingly, learn an adaptive strategy based on an approximation of our attack objective. We empirically show the threat posed by our attack to both tabular-RL and deep-RL algorithms, in both discrete and continuous environments.

Spiking Pitch Black: Poisoning an Unknown Environment to Attack Unknown Reinforcement Learners


Although learning has found wide application in multi-agent systems, its effects on the temporal evolution of a system are far from understood. This paper focuses on the dynamics of Q-learning in large-scale multi-agent systems modeled as population games. We revisit the replicator equation model for Q-learning dynamics and observe that this model is inappropriate for our concerned setting. Motivated by this, we develop a new formal model, which bears a formal connection with the continuity equation in physics. 
We show that our model always accurately describes the Q-learning dynamics in population games across different initial settings of MASs and game configurations. We also show that our model can be applied to different exploration mechanisms, describe the mean dynamics, and be extended to Q-learning in 2-player and n-player games. Last but not least, we show that our model can be used to gain insights into algorithm parameters, which potentially can facilitate parameter tuning.

The Dynamics of Q-learning in Population Games: a Physics-inspired Continuity Equation Model


A key challenge of evolutionary game theory and multi-agent learning is to characterize the limit behavior of game dynamics. Whereas convergence is often a property of learning algorithms in games satisfying a particular reward structure (e.g. zero-sum), it is well known, that for general payoffs even basic learning models, such as the replicator dynamics, are not guaranteed to converge. Worse yet, chaotic behavior is possible even in rather simple games, such as variants of Rock-Paper-Scissors games (Sato et al. 2002). Although chaotic behavior in learning dynamics can be precluded by the celebrated Poincaré-Bendixson theorem, it is only applicable to low-dimensional settings. Are there other characteristics of a game, which can force regularity in the limit sets of learning?

We show that behaviors consistent with the Poincaré-Bendixson theorem (limit cycles, but no chaotic attractor) follows purely based on the topological structure of the interaction graph, even for high-dimensional settings with arbitrary number of players and arbitrary payoff matrices. We prove our result for a wide class of follow-the-regularized leader (FoReL) dynamics, which generalize replicator dynamics, for games where each player has two strategies at disposal,and for interaction graphs where payoffs of each agent are only affected by one other agent. Since chaos has been observed in a game with only two players and three strategies, this class of non-chaotic games is in a sense maximal. Moreover, we provide simple conditions, under which such behavior translates to efficiency guarantees, implying that FoReL learning achieves time average sum of payoffs at least as good as that of a Nash equilibrium; and connecting the topology of the dynamics to the social welfare analysis.

Poincaré-Bendixson Limit Sets in Multi-Agent Learning


Concave Utility Reinforcement Learning (CURL) extends RL from linear to concave utilities in the occupancy measure induced by the agent's policy. This encompasses not only RL but also imitation learning and exploration, among others. Yet, this more general paradigm invalidates the classical Bellman equations, and calls for new algorithms. Mean-field Games (MFGs) are a continuous approximation of many-agent RL. They consider the limit case of a continuous distribution of identical agents, anonymous with symmetric interests, and reduce the problem to the study of a single representative agent in interaction with the full population. Our core contribution consists in showing that CURL is a subclass of MFGs. We think this important to bridge together both communities. It also allows to shed light on aspects of both fields: we show the equivalence between concavity in CURL and monotonicity in the associated MFG, between optimality conditions in CURL and Nash equilibrium in MFG, or that Fictitious Play (FP) for this class of MFGs is simply Frank-Wolfe, bringing the first convergence rate for discrete-time FP for MFGs. We also experimentally demonstrate that, using algorithms recently introduced for solving MFGs, we can address the CURL problem more efficiently.

Concave Utility Reinforcement Learning: the Mean-field Game viewpoint


Recent advances in multiagent learning have seen the introduction of a family of algorithms that revolve around the population-based training method PSRO, showing convergence to Nash, correlated and coarse correlated equilibria. 
Notably, when the number of agents increases, learning best-responses becomes exponentially more difficult, and as such hampers PSRO training methods. 
The field of mean-field games provides an asymptotic solution to this problem when the considered games are anonymous-symmetric.
Unfortunately, the mean-field approximation introduces non-linearities which prevent a straightforward adaptation of PSRO.
Building upon optimization and adversarial regret minimization, this paper sidesteps this issue and introduces mean-field PSRO, an adaptation of PSRO which learns Nash, coarse correlated and correlated equilibria in mean-field games. 
The key is to replace the exact distribution computation step by newly-defined mean-field no-adversarial-regret learners, or by black-box optimization. 
We compare the asymptotic complexity of the approach to standard PSRO, greatly improve empirical bandit convergence speed by compressing temporal mixture weights, and ensure it is theoretically robust to payoff noise.
Finally, we illustrate the speed and accuracy of mean-field PSRO on several mean-field games, demonstrating convergence to strong and weak equilibria.

Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO


As one of the solutions to the decentralized partially observable Markov decision process (Dec-POMDP) problems, the value decomposition method has achieved significant results recently. However, most value decomposition methods require the fully observable state of the environment during training, but this is not feasible in some scenarios where only incomplete and noisy observations can be obtained. Therefore, we propose a novel value decomposition framework, named State Inference for value DEcomposition (SIDE), which eliminates the need to know the global state by simultaneously seeking solutions to the two problems of optimal control and state inference. SIDE can be extended to any value decomposition method to tackle partially observable problems. By comparing with the performance of different algorithms in StarCraft II micromanagement tasks, we verified that though without accessible states, SIDE can infer the current state that contributes to the reinforcement learning process based on past local observations and even achieve superior results to many baselines in some complex scenarios.

A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

Downloads

Next from AAMAS 2022

ACuTE: Automatic Curriculum Transfer from Simple to Complex Environments

Similar lecture

Off-Policy Correction For Multi-Agent Reinforcement Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2022

ACuTE: Automatic Curriculum Transfer from Simple to Complex Environments

Similar lecture

Off-Policy Correction For Multi-Agent Reinforcement Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads