New Zealand


Multi-agent reinforcement learning (MARL) is a promising framework for solving complex tasks with many agents. However, a key challenge in MARL is defining private utility functions that ensure coordination when training decentralized agents. This challenge is especially prevalent in unstructured tasks with sparse rewards and many agents. We show that successor features can help address this challenge by disentangling an individual agent&#39;s impact on the global value function from that of all other agents. We use this disentanglement to compactly represent private utilities that support stable training of decentralized agents in unstructured tasks. We implement our approach using a centralized training, decentralized execution architecture and test it in a variety of multi-agent environments. Our results show improved performance and training time relative to existing methods and suggest that disentanglement of successor features offers a promising approach to coordination in MARL.

AAMAS 2022

Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning

reward distribution

multi-agent coordination

reinforcement learning

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

The AAMAS conference series was initiated in 2002 in Bologna, Italy as a joint event comprising the 6th International Conference on Autonomous Agents (AA), the 5th International Conference on Multiagent Systems (ICMAS), and the 9th International Workshop on Agent Theories, Architectures, and Languages (ATAL).

Subsequent AAMAS conferences have been held in Melbourne, Australia (July 2003), New York City, NY, USA (July 2004), Utrecht, The Netherlands (July 2005), Hakodate, Japan (May 2006), Honolulu, Hawaii, USA (May 2007), Estoril, Portugal (May 2008), Budapest, Hungary (May 2009), Toronto, Canada (May 2010), Taipei, Taiwan (May 2011), Valencia, Spain (June 2012), Minnesota, USA (May 2013), Paris, France (May 2014), Istanbul, Turkey (May 2015), Singapore (May 2016), São Paulo (2017) and Stockholm, Sweden (2018), Montreal (May 2019), Auckland (May 2020, Virtual), London (May 2021, Virtual).
<br>
<br>



A ticket is required to attend this event, please register using the link below:

https://aamas2022-conference.auckland.ac.nz/attending/registration/

Registration Is Required

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).


Multi-agent reinforcement learning (MARL) is a promising framework for solving complex tasks with many agents. However, a key challenge in MARL is defining private utility functions that ensure coordination when training decentralized agents. This challenge is especially prevalent in unstructured tasks with sparse rewards and many agents. We show that successor features can help address this challenge by disentangling an individual agent's impact on the global value function from that of all other agents. We use this disentanglement to compactly represent private utilities that support stable training of decentralized agents in unstructured tasks. We implement our approach using a centralized training, decentralized execution architecture and test it in a variety of multi-agent environments. Our results show improved performance and training time relative to existing methods and suggest that disentanglement of successor features offers a promising approach to coordination in MARL.

technical paper


Gerrymandering is the process of creating electoral districts for partisan advantage, allowing a party to win more seats than what is reasonable for their vote. While research on gerrymandering has recently grown, many issues are still not fully understood such as what influences the degree to which a party can gerrymander and what techniques can be used to counter it. One commonly suggested (and, in some US states, mandated) requirement is that districts be “geographically compact”. However, there are many competing compactness definitions and the impact of compactness on the gerrymandering abilities of the parties is not well understood. Also not well understood is how the growing urban-rural divide between supporters of different parties impacts redistricting.

We develop a modular, scalable, and efficient algorithm that can design districts for various criteria. We confirm its effectiveness on several US states by pitting it against maps “hand-drawn” by political experts. Using real data from US political elections we use our algorithm to study the interaction between population distri- bution, partisanship, and geographic compactness. We find that compactness can lead to more fair plans (compared to implemented plans) and limit gerrymandering potential, but there is a consistent asymmetry where the party with rural supporters has an advan- tage. We also show there are plans which are fair from a partisan perspective, but they are far from optimally compact.

Little House (Seat) on the Prairie: Compactness, Gerrymandering, and Population Distribution


Multi-Objective Reinforcement Learning (MORL) setup naturally arises in many places where an agent optimizes multiple objectives. We consider the problem of MORL where multiple objectives are combined using a non-linear scalarization. We combine the vector objectives with a concave scalarization function and maximize this scalar objective. To work with the non-linear scalarization, in this paper, we propose a solution using steady-state occupancy measures and long-term average rewards. We show that when the scalarization function is element-wise increasing, the optimal policy for the scalarization is also Pareto optimal. To maximize the scalarized objective, we propose a model-based posterior sampling algorithm that obtains a regret bound of $\Tilde{O}(KDS\sqrt{A/T})$ for $K$ objectives. Additionally, we propose policy-gradient and actor-critic algorithms for MORL. For the policy gradient actor, we obtain the gradient using chain rule, and we learn different critics for each of the $K$ objectives. Finally, we implement our algorithms on multiple environments including deep-sea treasure, and network scheduling setups to demonstrate that the proposed algorithms can optimize non-linear scalarization of multiple objectives.

Multi-Objective Reinforcement Learning with Non-Linear Scalarization

We examined principles of graded argumentation semantics (independence, anonymity, void precedence, and maximality) to explore if (a) they realistically model human reasoning, (b) graphical representation of arguments facilitates compliance with the principles, (c) there is a positive correlation between compliance with different principles, and (d) this compliance is related to cognitive reflection, need for cognition and faith in intuition. Our results indicate that there are major differences in the compliance with the several argumentation principles studied in this paper. However, compliance with argumentation principles was consistently better and more strongly correlated in the group presented with graphs. Moreover, cognitive reflection correlated with compliance to some principles, but only in the graph group.

Graphical Representation Enhances Human Compliance with Principles for Graded Argumentation Semantics


In a cooperative multiagent system, a collection of agents executes a joint policy in order to achieve some common objective. The successful deployment of such systems hinges on the availability of reliable inter-agent communication. However, many sources of potential disruption to communication exist in practice, such as radio interference, hardware failure, and adversarial attacks. In this work, we develop joint policies for cooperative multiagent systems that are robust to potential losses in communication. More specifically, we develop joint policies for cooperative Markov games with reach-avoid objectives. First, we propose an algorithm for the decentralized execution of joint policies during periods of communication loss. Next, we use the total correlation of the state-action process induced by a joint policy as a measure of the intrinsic dependencies between the agents. We then use this measure to lower-bound the performance of a joint policy when communication is lost. Finally, we present an algorithm that maximizes a proxy to this lower bound in order to synthesize minimum-dependency joint policies that are robust to communication loss. Numerical experiments show that the proposed minimum-dependency policies require minimal coordination between the agents while incurring little to no loss in performance; the total correlation of the synthesized policy is more than three orders of magnitude smaller than that of a baseline policy which does not take potential communication losses into account. As a result, the performance of the minimum-dependency policies remains consistently high regardless of whether or not communication is available. By contrast, the performance of the baseline policy decreases by twenty percent when communication is lost.

Planning Not to Talk: Multiagent Systems that are Robust to Communication Loss

Despite its rise as a prominent solution to the data inefficiency of today's machine learning models, self-supervised learning has yet to be studied from a purely multi-agent perspective. In this work, we propose that aligning internal subjective representations, which naturally arises in a multi-agent setup where agents receive partial observations of the same underlying environmental state, can lead to more data-efficient representations. We propose that multi-agent environments, where agents do not have access to the observations of others but can communicate within a limited range, guarantees a common context that can be leveraged in individual representation learning. The reason is that subjective observations necessarily refer to the same subset of the underlying environmental states and that communication about these states can freely offer a supervised signal. To highlight the importance of communication, we refer to our setting as socially supervised representation learning. We present a minimal architecture comprised of a population of autoencoders, where we define loss functions, capturing different aspects of effective communication, and examine their effect on the learned representations. We show that our proposed architecture allows the emergence of aligned representations. The subjectivity introduced by presenting agents with distinct perspectives of the environment state contributes to learning abstract representations that outperform those learned by a single autoencoder and a population of autoencoders, presented with identical perspectives of the environment state. Altogether, our results demonstrate how communication from subjective perspectives can lead to the acquisition of more abstract representations in multi-agent systems, opening promising perspectives for future research at the intersection of representation learning and emergent communication.

Socially supervised representation learning: the role of subjectivity in learning efficient representations


In the abstract argumentation setting, gradual semantics have been proposed to assess the individual strength of arguments. A number of such semantics have been proposed recently, and their formal properties have been studied. While these semantics are sometimes motivated by their better adequacy to capture debates, their behaviour in such multiagent settings is largely unexplored. In this paper, we undertake a study of the multiagent dynamics of a standard gradual semantics. We propose a simple protocol, where agents exchange arguments in order to provide a collective evaluation of the value of a given argument (\emph{i.e} an issue), and may learn new arguments from the other agents. The debate proceeds following a better response dynamics. We study how the value of the issue and the agents opinion evolve, depending on various parameters of this setting, both experimentally and theoretically.

Multiagent Dynamics of Gradual Argumentation Semantics


We propose an agent-based network formation model under uncertainty with the objective of relaxing the common assumption of complete information, calling attention to the role beliefs may play in segregation. We demonstrate that our model is capable of generating a set of networks that encompasses those of a complete information model. Further, we show that by allowing agents to hold biased beliefs toward each other based on social group membership, individual utility-maximising decisions may lead to group segregation at the cost of social welfare. We accompany our theoretical results with a simulation-based investigation of the relationship between beliefs and segregation and show that biased beliefs are an important driver of segregation under incomplete information.

Segregation in social networks of heterogeneous agents acting under incomplete information


The way opinions spread in a society decides the fate of elections, the success of products, and the impact of political or social movements. The model by Hegselmann and Krause is a well-known theoretical model to study such opinion formation processes in social networks. In contrast to many other theoretical models, it does not converge towards a situation where all agents agree on the same opinion.
Instead, it assumes that people find an opinion reasonable if and only if it is close to their own. The system converges towards a stable situation where agents sharing the same opinion form a cluster, and agents in different clusters do not influence each other. 

In the social variant of the Hegselmann-Krause model, agents are connected by a social network and their opinions evolve in an iterative process. Opinions are modeled as elements in the $d$-dimensional Euclidean space. When an agent is activated it adopts the average of the opinions of its influencing neighbors which are those neighbors having a similar opinion. Note that by this update process the set of influencing neighbors of an agent may change over time. Activation schemes focused on in the literature are synchronous agent activation where all agents are activated in parallel and asynchronous random agent activation where randomly chosen agents are activated sequentially.

Interestingly, social Hegselmann-Krause systems with asynchronous opinion updates have only been studied assuming the complete graph as social network. In this paper we show that the social Hegselmann-Krause opinion dynamics with asynchronous random agent activation is guaranteed to converge for any social network. We provide an upper bound of $\mathcal{O}(n|E|^2 (\eps/\delta)^2)$ on the expected number of opinion updates until convergence, where $|E|$ is the number of edges of the social network. For the already investigated setting on a complete social network we show an even better bound of $\mathcal{O}(n^3(n^2 + (\eps/\delta)^2))$ that yields a major improvement over the previously best known upper bound of $\mathcal{O}(n^9 (\eps/\delta)^2)$. We complement our upper bounds by providing an example showing that our potential function based analysis cannot be improved easily. Moreover, empirical agent-based simulations on specific social network topologies yield asymptotically matching lower bounds.

Asynchronous Opinion Dynamics in Social Networks


Influence maximization has now been a widely studied topic, but only in recent years have studies considered overexposure. Overexposure is usually measured as the negative cost associated with reaching unintended recipients during an information cascade. A polynomial-time algorithm is known for cascades with overexposure when we can select as many seed nodes as we want. This paper focuses on overexposure for the budgeted case of seeding, which has received little to no attention. We show that the problem is NP-hard even for a restricted case. For various special cases, we devise provable approximation algorithms, dynamic programming solutions, linear programming solutions, and heuristics. For the general case, we provide a linear programming solution and several fast and effective heuristics, mostly of the greedy flavor. We perform an extensive experimental study using synthetic and real-world networks. We investigate how network properties and model parameters impact our algorithms. It brings out interesting findings like why a low-quality product needs a smarter algorithm, and why certain algorithms do well on some networks but not others.

Cascades and Overexposure in Social Networks: The Budgeted Case


Collective risk dilemmas (CRDs) are a class of $n$-player games that represent societal challenges where groups need to coordinate to avoid the risk of hazardous consequences. Multi-agent systems incurring such dilemmas face difficulties achieving cooperation and often converge to sub-optimal solutions where everyone defects. In these contexts, empirical observations suggest that there is a strong discrepancy in risk, as often some parties are (or feel) more threatened than others. Here we investigate the role of risk diversity in groups of agents learning to play CRDs. We show that risk diversity places new challenges to cooperation that are not observed in homogeneous groups. While increasing average risk contributes, in general, for agents to cooperate with higher probability, increasing risk diversity significantly reduces a population's ability to achieve a collective target. Risk diversity leads to asymmetrical changes in agents policies --- i.e. the increase in contributions from individuals at high risk is unable to compensate for the decrease in contributions from individuals at low risk --- which reduces the total contributions in a population and overall social welfare. At the same time, risk diversity offers novel opportunities in the design of financial incentives, which, as we show, can foster higher cooperation, target achievement and global welfare than the ones obtained in the absence of diversity. Our results highlight the need to align risk perceptions among agents when possible and implement diversity-based incentive policies in order to improve collectives' abilities to coordinate and avoid future catastrophic events.

Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning

Downloads

Next from AAMAS 2022

Little House (Seat) on the Prairie: Compactness, Gerrymandering, and Population Distribution

Similar lecture

Near On-Policy Experience Sampling in Multi-Objective Reinforcement Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2022

Little House (Seat) on the Prairie: Compactness, Gerrymandering, and Population Distribution

Similar lecture

Near On-Policy Experience Sampling in Multi-Objective Reinforcement Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads