United Kingdom

In modular reinforcement learning (MRL), a complex decision making problem is decomposed into multiple simpler subproblems each solved by a separate module. Often, these subproblems have conflicting goals, and incomparable reward scales. A composable decision making architecture requires that even the modules authored separately with possibly misaligned reward scales can be combined coherently. An arbitrator should consider different module&#39;s action preferences to learn effective global action selection. We present a novel framework called GRACIAS that assigns fine-grained importance to the different modules based on their relevance in a given state, and enables composable decision making based on modern deep RL methods such as deep deterministic policy gradient (DDPG) and deep Q-learning. We provide insights into the convergence properties of GRACIAS and also show that previous MRL algorithms reduce to special cases of our framework. We experimentally demonstrate on several standard MRL domains that our approach works significantly better than the previous MRL methods, and is highly robust to incomparable reward scales. Our framework extends MRL to complex Atari games such as Qbert, and has a better learning curve than the conventional RL algorithms.

AAMAS 2021

Action Selection for Composable Modular Deep Reinforcement Learning

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-23 is the Thirty-Seventh AAAI Conference on Artificial Intelligence. The theme of this conference is to create collaborative bridges within and beyond AI. Like previous AAAI conferences, AAAI-23 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and two new activities: a Bridge Program and a Lab Program. Many of these activities are tailored to the theme of bridges and all are selected according to the highest standards, with additional programs for students and young researchers. 
AAAI is providing you with a conference planner, which you can use to help organize your itinerary of activities. This includes talks to attend in person, talks to attend remotely, breaks with colleagues and your site seeing activities. To access this conference planner, please go to [https://aaai-2023.takemobi.io](https://aaai-2023.takemobi.io).

In order to access this site, you need to register. If you haven't already, please register [here](https://aaai.org/Conferences/AAAI-23/registration/).


AAAI 2023

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines.

In modular reinforcement learning (MRL), a complex decision making problem is decomposed into multiple simpler subproblems each solved by a separate module. Often, these subproblems have conflicting goals, and incomparable reward scales. A composable decision making architecture requires that even the modules authored separately with possibly misaligned reward scales can be combined coherently. An arbitrator should consider different module's action preferences to learn effective global action selection. We present a novel framework called GRACIAS that assigns fine-grained importance to the different modules based on their relevance in a given state, and enables composable decision making based on modern deep RL methods such as deep deterministic policy gradient (DDPG) and deep Q-learning. We provide insights into the convergence properties of GRACIAS and also show that previous MRL algorithms reduce to special cases of our framework. We experimentally demonstrate on several standard MRL domains that our approach works significantly better than the previous MRL methods, and is highly robust to incomparable reward scales. Our framework extends MRL to complex Atari games such as Qbert, and has a better learning curve than the conventional RL algorithms.

technical paper

AAMAS is the largest and most influential conference in the area of agents and multiagent systems, bringing together researchers and practitioners in all areas of agent technology and providing and internationally renowned high-profile forum for publishing and finding out about the latest developments in the field. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).  

![](https://assets.underline.io/uploads/markdown_image/1/image/ce4604bd54468a006164d16e080204fb.jpg)

Please click on the button below to register for this event.

If you are registered and see this page please check the account email you are logged in with. It must be the same as for the registration.

If you still have trouble accessing the contact please contact support at aamas2021@underline.io 

In this paper, centralized and decentralized laws for the coverage problem under connectivity constraints are proposed. We formulated the problem as a continuous optimization task with the inequality constraint that algebraic connectivity must not be less than a small positive number, and solved the problem based on the active set method. This formulation can cause agents to be trapped in undesired local minima; to address this, we added the squared distance between the centroid of the coverage area and that of the agent system to the objective function of the coverage control. To derive the decentralized law, we employed an average consensus estimator that determines the algebraic connectivity and system centroid. When the algebraic connectivity is greater than the threshold, the proposed control laws allow agents to advance along the direction of steepest descent of the objective function. Once the connectivity is equal to the threshold, agents maintain the connectivity while decreasing the objective function by moving along the projection vector of the steepest descent in the direction in which the connectivity is constant. Simulation results confirmed the effectiveness of our proposed laws.

Coverage Control under Connectivity Constraints

Designing autonomous agents that act collaboratively to achieve a common set of goals has been a major goal for Artificial Intelligence research for many years. Collaborative Privacy-Preserving Planning (cppp) is a multi-agent planning task in which agents need to achieve a common set of goals without revealing certain private information [2]. In particular, in cppp an individual agent may have a set of private facts and actions that it does not share with the other agents. cppp has important motivating examples, such as planning for organizations that outsource some of their tasks, and has recently received considerable attention from the academic community.

Partial Disclosure of Private Dependencies in Privacy Preserving Planning

Pool block withholding attack, which reduces the effective mining power in the system and leads to potential systemic instability in the blockchain, can be modeled as a non-cooperative game called "the miner's dilemma". However, existing literature on the game-theoretic properties of this attack only gives a preliminary analysis. In this paper, we establish the existence and uniqueness of pure Nash equilibrium for the two-player miner's dilemma. Then we give a tight upper bound 2 for PPoA, which measures how much mining power is wasted in the game. Moreover, we show the uniqueness and the tight bound holds in a more general setting with betrayal assumption. Inspired by the experiments on the games among three mining pools, we conjecture that similar results should hold for the N-player miner's dilemma game (N ≥ 2).

The Tight Bound for Pure Price of Anarchy in an Extended Miner's Dilemma Game

Business models rooted in shared economy, electrification, and automation are transforming urban mobility. Accounting for how these transformations interact is crucial if synergies are to be exploited. In this paper, we focus on how a cost-effective charging infrastructure for e-mobility can support the emergence of shared, autonomous mobility. This study addresses the problem of siting and sizing of charging stations for a fleet of shared autonomous electric vehicles (SAEVs). We develop a hybrid simulation-optimization model to find locations and numbers of chargers needed to serve charging demands. Our agent-based model provides an enhanced representation of SAEV operations allowing for smart charging and vehicle cruising when parking/charging is not available. Also, we model charging station placement as full covering optimization and solve the location-allocation problem simultaneously. Finally, we employ real-world trip data from ShareNow in Berlin to evaluate our approach for realistic demand patterns under different charging strategies and fleet sizes. The results show that charging station locations depend mostly on the spatial distribution of installation costs and charging demands. Moreover, charging strategies and fleet size affect the charging patterns and the required number of chargers as well as fleet performance.

Siting and Sizing of Charging Infrastructure for Shared Autonomous Electric Fleets

We design a prediction market to recover a complete and fully general probability distribution over a random variable. Traders buy and sell interval securities that pay $1 if the outcome falls into an interval and $0 otherwise. Our market takes the form of a central automated market maker  and allows traders to express interval endpoints of arbitrary precision. We present two designs in both of which market operations take time logarithmic in the number of intervals (that traders distinguish), providing the first computationally efficient market for a continuous variable. Our first design replicates the popular logarithmic market scoring rule  (LMSR), but operates exponentially faster than a standard LMSR by exploiting its modularity properties to construct a balanced binary tree and decompose computations along the tree nodes. The second design consists of two or more parallel LMSR market makers that mediate submarkets of increasingly fine-grained outcome partitions.  This design remains computationally efficient for all operations, including arbitrage removal across submarkets. It adds two additional benefits for the market designer: (1)~the ability to express utility for information at various resolutions by assigning different liquidity values, and  (2)~the ability to guarantee a true constant bounded loss by appropriately decreasing the liquidity in each submarket.

Log-time Prediction Markets for Interval Securities

Equilibrium computation in markets usually considers settings where player valuation functions are known. We consider the setting where player valuations are unknown; using a PAC learning-theoretic framework, we analyze some classes of common valuation functions, and provide algorithms which output direct PAC equilibrium allocations, not estimates based on attempting to learn valuation functions. Since there exist trivial PAC market outcomes with an unbounded worst-case efficiency loss, we lower-bound the efficiency of our algorithms. While the efficiency loss under general distributions is rather high, we show that in some cases (e.g., unit-demand valuations), it is possible to find a PAC market equilibrium with significantly better utility.

The Price is (Probably) Right: Learning Market Equilibria from Samples

We introduce a new numerical framework to learn optimal bidding strategies in repeated auctions when the seller uses past bids to optimize her mechanism. Crucially, we do not assume that the bidders know which optimization mechanism is used by the seller. We recover essentially all state-of-the-art analytical results for the single-item framework derived previously in the setup where the bidder knows the optimization mechanism used by the seller and extend our approach to multi-item settings, in which no optimal shading strategies were previously known. Our approach yields substantial increases in bidder utility in all settings and has a strong potential for practical usage since it provides a simple way to optimize bidding strategies on modern marketplaces where buyers face unknown data-driven mechanisms.

Adversarial Learning in Revenue-Maximizing Auctions

Ensemble learning is a very prevalent method employed in machine learning. The relative success of ensemble methods is attributed to their ability to tackle a wide range of instances and complex problems that require different low-level approaches. However, ensemble methods are relatively less popular in reinforcement learning owing to the high sample complexity and computational expense involved in obtaining a diverse ensemble. We present a novel training and model selection framework for model-free reinforcement algorithms that use ensembles of policies obtained from a single training run. These policies are diverse in nature and are learned through directed perturbation of the model parameters at regular intervals. We show that learning and selecting an adequately diverse set of policies is required for a good ensemble while extreme diversity can prove detrimental to overall performance. Selection of an adequately diverse set of policies is done through our novel policy selection framework. We evaluate our approach on challenging discrete and continuous control tasks and also discuss various ensembling strategies. Our framework is substantially sample efficient, computationally inexpensive and is seen to outperform state-of-the-art (SOTA) scores in Atari 2600 and Mujoco.

SEERL: Sample Efficient Ensemble Reinforcement Learning

Blockchain-based Distributed Ledgers (DLs) promise to transform the existing financial system by making it truly democratic. In the past decade, blockchain technology has seen many novel applications ranging from the banking industry to real estate. However, in order to be adopted universally, blockchain systems must be scalable to support a high volume of transactions. As we increase the throughput of the DL system, the underlying peer-to-peer network might face multiple levels of challenges to keep up with the requirements. Due to varying network capacities, the slower nodes would be at a relative disadvantage compared to the faster ones, which could negatively impact their revenue. In order to quantify their relative advantage or disadvantage, we introduce two measures of network fairness, $p_f$, the probability of frontrunning and α_f$, the publishing fairness. We show that as we scale the blockchain, both these measures deteriorate, implying that the slower nodes face a disadvantage at higher throughputs. It results in the faster nodes getting more than their fair share of the reward while the slower nodes (slow in terms of network quality) get less. Thus, fairness and scalability in blockchain systems do not go hand in hand.

We might walk together, but I run faster: Network Fairness and Scalability in Blockchains

Complex networks and reputation systems are fundamental mechanisms to sustain cooperation in populations of self-regarding agents. These mechanisms are typically studied in isolation. In online social platforms, however, behavioral dynamics are likely to result from their combination. Here we investigate the relationship between social networks and reputation-based cooperation (in a Prisoner’s Dilemma setting) in large populations. We develop a new evolutionary game-theoretical model and study dynamics in networks with varying degrees of community structure. We show that networks exhibiting modular structures hamper global cooperation: reputation-based group identities emerge in different communities and strategies that uniquely cooperate with in-group members fixate, sustaining polarization and group bias. Global cooperation is recovered provided that inter-community edges are added.

Action Selection for Composable Modular Deep Reinforcement Learning

Downloads

Next from AAMAS 2021

Coverage Control under Connectivity Constraints

Similar lecture

SpatialFormer: Semantic and Target Aware Attentions for Few-Shot Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Action Selection for Composable Modular Deep Reinforcement Learning

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2021

Coverage Control under Connectivity Constraints

Similar lecture

SpatialFormer: Semantic and Target Aware Attentions for Few-Shot Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads