New Zealand

Dennis Barrios-Aranibar and Luiz M. G. Gon&#231;alves (January 1st 2009). Influence Value Q-Learning: A Reinforcement Learning Algorithm for Multi Agent Systems, Theory and Novel Applications of Machine Learning, Meng Joo Er and Yi Zhou, IntechOpen, DOI: 10.5772/6675.

Mengchun Xie (March 18th 2019). Improvement of Cooperative Action for Multi-Agent System by Rewards Distribution, Assistive and Rehabilitation Engineering, Yves Rybarczyk, IntechOpen, DOI: 10.5772/intechopen.85109.

Bart&#322;omiej &#346;nie&#380;y&#324;ski (January 1st 2009). Supervised Rule Learning and Reinforcement Learning in A Multi-Agent System for the Fish Banks Game, Theory and Novel Applications of Machine Learning, Meng Joo Er and Yi Zhou, IntechOpen, DOI: 10.5772/6672.

Counterfactual Regret Minimization (CFR) has found success in settings like poker which have both terminal states and perfect recall. We seek to understand how to relax these requirements. As a first step, we introduce a simple algorithm, local no-regret learning (LONR), which uses a Q-learning-like update rule to allow learning without terminal states or perfect recall. We discuss its provable convergence for the basic case of MDPs and present empirical results showing that it achieves last iterate convergence in NoSDE games, a class of general sum Markov games specifically designed to be challenging to learn where no prior algorithm is known to achieve convergence to a stationary equilibrium even on average. 


AAMAS 2020

Combining No-Regret and Q-Learning

no-regret

counterfactual regret minimization

optimism

multi-agent

reinforcement learning

The purpose of the AAAI conference is to promote research in artificial intelligence (AI) and scientific exchange among AI researchers, practitioners, scientists, and engineers in affiliated disciplines. 

AAAI 2021

technical paper

AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). The aim of the joint conference is to provide a single, high-profile, internationally-respected archival forum for scientific research in the theory and practice of autonomous agents and multi-agent systems.

Browse keynotes, discussions, panels and over 300 presentations.


AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). 

This presentation is about our AAMAS 2020 paper ""Plannable Approximations to MDP Homomorphisms: Equivariance under Actions"". This work exploits action equivariance for representation learning in reinforcement learning. Equivariance under actions states that
transitions in the input space are mirrored by equivalent transitions in latent space, while the map and transition functions should also commute. We introduce a contrastive loss function that enforces action equivariance on the learned representations. We prove that when our loss is zero, we have a homomorphism of a deterministic Markov Decision Process (MDP). Learning equivariant maps leads to structured latent spaces, allowing us to build a model on which we plan through value iteration. We show experimentally that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP. Moreover, the approach easily adapts to changes in the goal states. Empirically, we show that in such MDPs, we obtain better representations in fewer epochs compared to representation learning approaches using reconstructions, while generalizing better to new goals than model-free approaches.


Plannable Approximations to MDP Homomorphisms: Equivariance under Actions

Consider a closed environment with static obstacles and mobile agents moving around. There are hider agents that hide from the seeker agents. The seeker has a limited visibility range, and if a hider comes into the visibility region of a seeker, it is considered caught. The practical applications range from gaming to security. In this work, we focus on deterministic capture of hiders, even if they are guided by an Oracle which knows the future positions of seekers. We develop strategies for seekers, having limited visibility ranges, to catch all hiders and establish minimum bounds on the number of seekers required to catch the hiders, on a per strategy basis. We use spatio-temporal graph models and reasoning to formulate and address the problem.


Capturing Oracle Guided Hiders

Effective coordination is a critical requirement for human teaming, and is increasingly needed in teams of humans and robots. Building on decades of work in the behavioral literature, we have implemented a computational framework for coordination based on Shared Mental Models (SMMs) in which robots use a distributed knowledge base to coordinate activity. We also built a novel system connecting the robotic architecture, DIARC, to the 3D simulation environment, Unity, to serve as an evaluation platform for the framework implementation, and also for more general explorations of teaming with autonomous robots. Using this platform, we ran a user study to evaluate the framework by comparing performance of teams in which the robots used SMMs with those that did not. We found that teams in which the robots used SMMs significantly outperformed those without SMMs. This represents the first empirical demonstration that SMMs can be successfully used by fully autonomous robots interacting in natural language to improve team performance, bringing robots a step closer to genuine teammate


Toward Genuine Robot Teammates: Improving Human-Robot Team Performance Using Robot Shared Mental Models

An important aspect of multi-agent systems concerns the formation of coalitions that are stable or optimal in some well-defined way. The notion of popularity has recently received a lot of attention in this context. A partition is popular if there is no other partition in which more agents are better off than worse off. 
In 2019, a long-standing open problem concerning popularity was solved by proving that computing popular partitions in roommate games is NP-hard, even when preferences are strict. We show that this result breaks down when allowing for randomization: mixed popular partitions can be found efficiently via linear programming and a separation oracle. Mixed popular partitions are particularly attractive because they are guaranteed to exist in any coalition formation game.
Our result implies that one can efficiently verify whether a given partition in a roommate game is popular and that strongly popular partitions can be found in polynomial time (resolving an open problem). By contrast, we prove that both problems become computationally intractable when moving from coalitions of size 2 to coalitions of size 3, even when preferences are strict and globally ranked.


Finding and Recognizing Popular Coalition Structures

Designers of virtual agents have a combinatorically large space of choices for different media that comprise the look and behavior of their characters. We explore the systematic manipulation of animation quality, speech quality, and rendering style, and its impact on the perceptions of virtual agents in terms of naturalness, engagement, trust, credibility, and persuasion within a health counseling domain. The agent’s counseling behavior was based on live video footage of a human counselor. We conducted a between-subjects study that had 12 conditions. Character animation was varied between a static image, procedural animation using a gestuary, and manually rotoscoped animation. Voice quality was varied between recorded audio of the human counselor and synthetic speech. Character rendering style was varied between 3D-shaded realistic and toon-shaded. Prior studies indicate that people prefer and attribute more sociality to other people and agents when modalities are consistent in their level of quality. Thus, we hypothesize that people will be most affected by agents whose animation, voice, and rendering style are consistent, rather than the effects of channel quality being purely additive. Results indicate that natural animations and recorded voice are more appropriate for general acceptance, trust, and credibility of the agent, and how appropriate she seems for the task. However, our results indicate that for a brief health counseling task, animation might actually be distracting from the persuasive message, with the highest levels of persuasion found when the amount of agent animation is minimized.


Navigating the Combinatorics of Virtual Agent Design Space to Maximize Persuasion

We initiate the study of cloning in multiwinner elections, focusing
on single-transferable vote (STV), single-nontransferable vote
(SNTV), bloc, k-Borda, t-approval-CC, and Borda-CC. Transferring
the model of cloning due to Elkind et al. [15] from single-winner to
multiwinner elections, we consider decision problems describing
possible and necessary cloning in the zero-cost, the unit-cost, and
the general-cost model and study their computational complexity.
We show that, depending on the multiwinner voting rule and on
the cost model chosen, some of these cloning problems are in P,
some are NP-hard, and some of the latter (for which, in fact, already
winner determination is NP-hard) are fixed-parameter tractable.

The Complexity of Cloning Candidates in Multiwinner Elections

The hypergraph matching game is a cooperative game defined on a hypergraph such that the vertices are the players, and the characteristic function is the value of a maximum hypergraph matching on a hypergraph induced by a coalition. 
In this talk, we study a computationally tractable condition of the hypergraph matching game, called the convexity.
First, we prove that the problem of checking whether a given hypergraph matching game is convex or not is solvable in polynomial time.
Second, we prove that the Shapley value of a given convex hypergraph matching game is exactly computable in polynomial time.
Finally, we consider the fractional hypergraph matching game and prove that if the fractional hypergraph matching game is convex, then its characteristic function coincides with the characteristic function of the corresponding (integral) hypergraph matching game.

Convexity of Hypergraph Matching Game

We live in a world full of complex social systems. Achieving opti- mal control in a complex social system is challenging due to the difficulty in modeling and optimization. To capture the complex social system dynamics accurately and succinctly, we model the decision-making problem as a partially observable discrete event decision process. To withstand the curse of dimensionality in high- dimensional belief state spaces and to optimize the problem in an amenable searching space, we investigate the connections between the value function of a partially observable decision process and that in the corresponding fully-observable scenario, and reduce the optimal control of a partially observable discrete event decision process to a policy optimization with a specially formed fully ob- servable decision process and a belief state estimation. When tested in real-world transportation scenarios, in comparison with other state-of-the-art approaches, our proposed algorithm leads to the least average time on-road, the largest number of vehicles at work during work hours and the fewest training epochs to converge to the highest total rewards per episode.


Optimal Control in Partially Observable Complex Social Systems

We introduce a model for agent-environment systems where the agents are implemented via feed-forward ReLU neural networks and the environment is non-deterministic. We study the verification problem of such systems against CTL properties. We introduce a bounded fragment of CTL and show its usefulness in identifying shallow bugs in the system. We present a novel parallel algorithm for MILP-based verification of agent-environment systems, present an implementation, and report the experimental results obtained against a variant of the VerticalCAS use-case.


Formal Verification of Neural Agents in Non-deterministic Environments

Coupon has been a major marketing tool that promotes sales for new and repeated buyers and proven effective in numerous realistic scenarios. In this paper, we mainly focus on the problem of designing coupons to maximize the revenue in second-price auction. 

Firstly, we derive the dominant strategies of bidders if they are provided with coupons in second-price auction and prove that the revenue optimization problem with coupons for all the bidders is NP-complete. Secondly, we cast the problem of designing coupons to maximize revenue into a learning framework. With well-designed loss functions, we perform theoretical analysis of its properties and propose corresponding algorithms to solve the problem. Finally, with both synthetic data and industrial data, extensive experiments are conducted to demonstrate their effectiveness.


Downloads

Next from AAMAS 2020

Plannable Approximations to MDP Homomorphisms: Equivariance under Actions

Similar lecture

Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2020

Plannable Approximations to MDP Homomorphisms: Equivariance under Actions

Similar lecture

Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads