In many multi-agent interactions in the real world, agents receive payoffs over multiple distinct criteria; i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. We investigate here the effects of opponent modelling on multi-objective multi-agent interactions with non-linear utilities. Specifically, we consider multi-objective normal form games (MONFGs) with non-linear utility functions under the scalarised expected returns optimisation criterion. We contribute a novel actor-critic formulation to allow reinforcement learning of mixed strategies in this setting, along with an extension that incorporates opponent policy reconstruction using conditional action frequencies. Our empirical results demonstrate that opponent modelling can drastically alter the learning dynamics in this setting.


Opponent Modelling for Reinforcement Learning in Multi-Objective Normal Form Games

Toll-Based Learning for Minimising Congestion under Heterogeneus Preferences

Many real-world decision problems are inherently multi-objective in nature and concern multiple actors, making multi-objective multi-agent systems a key domain to study. We argue that trade-offs between conflicting objective functions should be analysed on the basis of the utility that these trade-offs have for the users of a system. We develop a new taxonomy which classifies multi-objective multi-agent decision making settings, on the basis of the reward structures and utility functions. We analyse which solution concepts apply to the different settings in our taxonomy, which allows us to offer a structured view of the field and identify promising directions for future research.


Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey

Multi-objective multi-agent decision making: a utility based analysis and survey

In many multi-agent interactions in the real world, agents receive payoffs over multiple distinct criteria; i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with non-linear utilities. Specifically, we consider multi-objective normal form games with non-linear utility functions under the scalarised expected returns optimisation criterion. We contribute a novel actor-critic formulation to allow reinforcement learning of mixed strategies in this setting, along with an extension that incorporates opponent policy reconstruction using conditional action frequencies. Empirical results in five different MONFGs demonstrate that opponent modelling can drastically alter the learning dynamics in this setting. When equilibria are present opponent modelling can confer significant benefits on agents that implement it. However, when there are no Nash equilibria, opponent modelling can have adverse effects on utility, and has a neutral effect at best (after extensive hyper-parameter optimisation).


Opponent Modelling using Policy Reconstruction for Multi-Objective Normal Form Games

technical paper

AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). The aim of the joint conference is to provide a single, high-profile, internationally-respected archival forum for scientific research in the theory and practice of autonomous agents and multi-agent systems.

Browse keynotes, discussions, panels and over 300 presentations.


AAMAS 2020

AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). 

Roxana Radulescu

5

SHORT BIO

Presentations

Opponent Modelling for Reinforcement Learning in Multi-Objective Normal Form Games

Toll-Based Learning for Minimising Congestion under Heterogeneus Preferences

Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey

Multi-objective multi-agent decision making: a utility based analysis and survey

Opponent Modelling using Policy Reconstruction for Multi-Objective Normal Form Games

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES