United States

Transformer-based autoregressive (AR) methods have achieved appealing performance for varied sequence-to-sequence generation tasks, e.g., neural machine translation, summarization, and code generation, but suffer from low inference efficiency. To speed up the inference stage, many non-autoregressive (NAR) strategies have been proposed in the past few years. Among them, the conditional masked language model (CMLM) is one of the most versatile frameworks, as it can support many different sequence generation scenarios and achieve very competitive performance on these tasks. In this paper, we further introduce a simple yet effective adaptive masking over masking strategy to enhance the refinement capability of the decoder and make the encoder optimization easier. Experiments on \textbf{3} different tasks (neural machine translation, summarization, and code generation) with \textbf{15} datasets in total confirm that our proposed simple method achieves significant performance improvement over the strong CMLM model. Surprisingly, our proposed model yields state-of-the-art performance on neural machine translation (\textbf{34.62} BLEU on WMT16 EN$\to$RO, \textbf{34.82} BLEU on WMT16 RO$\to$EN, and \textbf{34.84} BLEU on IWSLT De$\to$En) and even better performance than the \textbf{AR} Transformer on \textbf{7} benchmark datasets with at least \textbf{2.2$\times$} speedup. Our anonymous code is available at GitHub\footnote{\url{ https://github.com/amom-nar/AMOM}}.

AAAI 2023

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

sequence generation

non-autoregressive

natural language processing

transformer

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-23 is the Thirty-Seventh AAAI Conference on Artificial Intelligence. The theme of this conference is to create collaborative bridges within and beyond AI. Like previous AAAI conferences, AAAI-23 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and two new activities: a Bridge Program and a Lab Program. Many of these activities are tailored to the theme of bridges and all are selected according to the highest standards, with additional programs for students and young researchers. 
AAAI is providing you with a conference planner, which you can use to help organize your itinerary of activities. This includes talks to attend in person, talks to attend remotely, breaks with colleagues and your site seeing activities. To access this conference planner, please go to [https://aaai-2023.takemobi.io](https://aaai-2023.takemobi.io).

In order to access this site, you need to register. If you haven't already, please register [here](https://aaai.org/Conferences/AAAI-23/registration/).


The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines.

poster

Existing knowledge graph (KG) embedding models have primarily focused on static KGs. However, real-world KGs do not remain static, but rather evolve and grow in tandem with the development of KG applications. Consequently, new facts and previously unseen entities and relations continually emerge, necessitating an embedding model that can quickly learn and transfer new knowledge through growth. Motivated by this, we delve into an expanding field of KG embedding in this paper, i.e., lifelong KG embedding. We consider knowledge transfer and retention of the learning on growing snapshots of a KG without having to learn embeddings from scratch. The proposed model includes a masked KG autoencoder for embedding learning and update, with an embedding transfer strategy to inject the learned knowledge into the new entity and relation embeddings, and an embedding regularization method to avoid catastrophic forgetting. To investigate the impacts of different aspects of KG growth, we construct four datasets to evaluate the performance of lifelong KG embedding. Experimental results show that the proposed model outperforms the state-of-the-art inductive and lifelong embedding baselines.

Lifelong Embedding Learning and Transfer for Growing Knowledge Graphs

The prevalence of region-based urban data has enabled us to explore the correlations of regions for better urban planning and smart-city solutions, in which region embedding plays an important role. However, this is a challenging problem considering the following two factors: different city data for region embedding and the effectiveness for various downstream applications. 
In this paper, we propose a novel framework, named \textbf{HREP} (\textbf{H}eterogeneous \textbf{R}egion \textbf{E}mbedding with \textbf{P}rompt learning), considering both intra-region and inter-region correlations, which includes two major modules, \textbf{H}eterogeneous graph learning for \textbf{R}egion \textbf{E}mbedding (\textbf{HRE}) and prompt learning for different downstream tasks. In the HRE module, we construct a region heterogeneous graph according to three categories of data, capturing inter-region contexts, e.g., human mobility and region geographic neighbors, and intra-region contexts, e.g., POI information. Specially, we design a \textit{relation-aware graph embedding} to learn region embedding and relation embeddings of edge types based on the heterogeneous graph. Then we introduce self-attention to capture the global correlation of regions. Moreover, we develop a \textit{attention-based fusion module} to integrate sharing information among different types of correlations. 
To enable region embedding to be effectively utilized in downstream tasks, we migrate the prompt learning, i.e., continuous prompt
prefix-tuning, which can guide the learning of downstream tasks, leading to better prediction results. Extensive experimental results on real-world datasets show that our model outperforms all state-of-the-art methods dramatically.

Heterogeneous Region Embedding with Prompt Learning

In many applications we want to influence the decisions of independent agents by designing incentives for their actions. We revisit a fundamental problem in this area, called Game Implementation: Given a game in standard form and a set of desired strategies, can we design a set of payment promises such that, if the players take the payment promises into account, then all undominated strategies are desired? Furthermore, we aim to minimize the cost, that is, the worst-case amount of payments.

We study the tractability of computing such payment promises and determine more closely what obstructions we may have to overcome in doing so. We show that Game Implementation is NP-hard even for two players, solving in particular a long open question (Eidenbenz et al. 2011). This suggests more restrictions are necessary to obtain tractability. We thus study the regime in which players have only a small constant number of strategies and obtain the following. First, this case remains NP-hard even if each player's utility depends only on three others. Second, we repair a flawed efficient algorithm for the case of both small number of strategies and small number of players. Among further results, we characterize sets of desired strategies that can be implemented at zero cost as a kind of stable core of the game.

Game Implementation: What are the Obstructions?

Digital advertising constitutes one of the main revenue sources for online platforms. In recent years, some advertisers tend to adopt auto-bidding tools to facilitate advertising performance optimization, making the classical \emph{utility maximizer} model in auction theory not fit well. Some recent studies proposed a new model, called \emph{value maximizer}, for auto-bidding advertisers with return-on-investment (ROI) constraints. However, the model of either utility maximizer or value maximizer could only characterize partial advertisers in real-world advertising platforms. In a mixed environment where utility maximizers and value maximizers coexist, the truthful ad auction design would be challenging since bidders could manipulate both their values and affiliated classes, leading to a multi-parameter mechanism design problem. In this work, we address this issue by proposing a payment rule which combines the corresponding ones in classical VCG and GSP mechanisms in a novel way. Based on this payment rule, we propose a truthful auction mechanism with an approximation ratio of $2$ on social welfare, which is close to the lower bound of at least $\frac{5}{4}$ that we also prove. The designed auction mechanism is a generalization of VCG for utility maximizers and GSP for value maximizers.

Utility Maximizer or Value Maximizer: Mechanism Design for Mixed Bidders in Online Advertising

Time or money? That is a question! In this paper, we consider this dilemma in the pricing regime, in which we try to find the optimal pricing scheme for identical items with heterogenous time-sensitive buyers. We characterize the revenue-optimal solution and propose an efficient algorithm to find it in a Bayesian setting. Our results also demonstrate the tight ratio between the value of wasted time and the seller's revenue, as well as that of two common-used pricing schemes, the $k$-step function and the fixed pricing. To explore the nature of the optimal scheme in the general setting, we present the closed forms over the product distribution and show by examples that positive correlation between the valuation of the item and the cost per unit time could help increase revenue. To the best of our knowledge, it is the first step towards understanding the impact of the time factor as a part of the buyer cost in pricing problems, in the computational view.

Optimal Pricing Schemes for Identical Items with Time-Sensitive Buyers

AAAI Award for Artificial Intelligence for the Benefit of Humanity

I will share experiences from working on organ exchanges for the last 18 years, ranging from market designs to new optimization algorithms to large-scale fielding of the techniques and even to computational policy optimization. Originally in kidney exchange, patients with kidney disease obtained compatible donors by swapping their own willing but incompatible donors. I will discuss many modern generalizations of this basic idea. For one, I will discuss never-ending altruist donor chains that have become the main modality of kidney exchanges worldwide and have led to over 10,000 life-saving transplants. Since 2010, our algorithms have been running the national kidney exchange for United Network for Organ Sharing, which has grown to include 80% of the transplant centers in the US. Our algorithms autonomously make the transplant plan each week for that exchange, and have been used by two private exchanges before that. I will summarize the state of the art in algorithms for the batch problem, approaches for the dynamic problem where pairs and altruists arrive and depart, techniques that find the highest-expected-quality solution under the real challenge of unforeseen pre-transplant incompatibilities, algorithms for pre-match compatibility testing, and approaches for striking fairness-efficiency tradeoffs. I will describe the FUTUREMATCH framework that combines these elements and uses data and supercomputing to optimize the policy from high-level human value judgments. The approaches therein may be able to serve as ways of designing policies for many kinds of complex real-world AI systems. I will also discuss the idea of liver lobe exchanges and cross-organ exchanges, and how they have started to emerge for real.

Modern Organ Exchanges: Market Designs, Algorithms, and Opportunities

Robert S. Engelmore Memorial Award Lecture

I share insights on the interaction of humans and AI to jointly solve end-to-end complex problems. The talk is based on my experience with research and engineering of real autonomous mobile service robots, and AI in the finance domain. I will illustrate the integration of perception (data), cognition (reasoning and learning), and action (execution and feedback). And I will focus on several components, including AI to effectively discover and standardize data, for behavior understanding, to simulate and generate synthetic data, to learn from data, principles, and experience, and to provide trustful explanations.

Symbiotic Human-AI Interaction: Experience-Based Insights from AI in Robotics and AI in Finance

Computer vision is one of the great AI success stories. Yet we are still far from having machine systems that can reliably and robustly see everything a human being sees in an image or in the real world. Despite rapid advances in self-supervised visual and multimodal representation learning, we are also far from having systems that can learn to see as richly as a human does, from so little data, or that can learn new visual concepts or adapt their representations as quickly as a human does. And even today’s remarkable generative image synthesis systems imagine the world in a very different and fundamentally less flexible way than human beings do. How can we close these gaps? I will describe several core insights from the study of human vision and visual cognitive development that run counter to the dominant trends in today's computer vision and machine learning world, but that can motivate and guide an alternative approach to building practical machine vision systems.

Technically, this approach rests on advances in differentiable and probabilistic programming: hybrids of neural, symbolic and probabilistic modeling and inference that can be more robust, more flexible and more data-efficient than purely neural approaches to learning to see. New probabilistic programming platforms offer to make these approaches scalable as well. Conceptually, this approach draws on classic proposals for understanding vision as ‘inverse graphics’, ‘analysis by synthesis’ or ‘inference to the best explanation’, and the notion that at least some high-level architecture for scene representation is built into the brain by evolution rather than learned from experience, reflecting invariant properties of the physical world. Learning then enables, enriches and extends these built-in representations; it does not create them from scratch. I will show a few examples of recent machine vision successes based on these ideas, from our group and others. But the hardest problems are still very open. I will highlight some ‘Grand Challenge’ tasks for building machines that learn to see like people: problems that far outstrip the abilities of any current system, and that I hope can inspire the next steps towards progress for computer vision researchers regardless of which approach they favor.

Learning to See the Human Way

The development of robots that can learn to interact with the world and manipulate the objects in it has emerged as one of the greatest and so far largely unsolved challenges in robotics research. In this talk, I will argue that the development of such advanced machines requires a transition from classical manual design with purely model-based control to a novel synthesis paradigm. We need to allow the machine to autonomously develop its own blueprint and algorithmically generate its topological, kinematic, and dynamic self. Building on this, it shall develop controls for its own body as it moves, learns to manipulate objects in a controlled way, and sensitively interacts with the world.

Drawing from our work in torque-controlled lightweight robots towards human-safe tactile robots that can manipulate, fly, or drive, I outline the technological quantum leaps that recently have taken place. In particular, this progress was made possible by a human-centered design, soft and force-sensitive control, contact reflexes, and model-based machine learning. In the real world, by enabling human-robot coexistence, collaboration, and interaction for the first time, this robotic technology has proven transformative to traditional manufacturing already around the globe. Increasingly, it is now impacting professional services, domestic applications, medicine and healthcare.

After that, I will use our current work to chart the path toward the next generation of tactile machines. We have taken first steps towards autonomously designing and building machines that have the ability to learn their self and thus adapt to changes in body topology and ultimately their entire dynamics. Finally, I will present recent results on designing modular control and learning architectures that achieve complex behaviors for challenging manipulation problems while being provably stable.

Robots with a sense of touch: self-replicating the machine and learning the self

Consider an intelligent virtual assistant such as Siri, or perhaps a more capable future version of it. Should we think of all of Siri as one big agent? Or is there a separate agent on every phone, each with its own objectives and/or beliefs? And what should those objectives and beliefs be? Such questions reveal that the traditional, somewhat anthropomorphic model of an agent – with clear boundaries, centralized belief formation and decision making, and a clear given objective – falls short for thinking about today’s AI systems. We need better methods for specifying the objectives that these agents should pursue in the real world, especially when their actions have ethical implications. I will discuss some methods that we have been developing for this purpose, drawing on techniques from preference elicitation and computational social choice. But we need to specify more than objectives. When agents are distributed, systematically forget what they knew before (say, for privacy reasons), can be simulated by others, and potentially face copies of themselves, it is no longer obvious what the correct way is even to do probabilistic reasoning, let alone to make optimal decisions. I will explain why this is so and discuss our work on doing these things well. (No previous background required.)

Premium content

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

Downloads

Next from AAAI 2023

Lifelong Embedding Learning and Transfer for Growing Knowledge Graphs

Similar lecture

RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES