Thailand

Previous studies have argued that pre-trained language models encode commonsense relational knowledge (e.g. that apples are edible). However, simultaneous work has revealed that such models are often insensitive to context, even ignoring overt contextual cues such as negations. In this paper, we investigate whether masked language models (the BERT family), can move beyond naive associative biases (e.g., apple -&gt; edible) when the context warrants (e.g. ranking inedible higher when presented with the information that the apple is rotten). We introduce the WinoVenti procedure, which adversarially exploits associative biases in masked language models to create model-specific Winograd-style entailment schemas. Using our constructed WinoVenti challenges set of over 2000 schemas, we discover that language models in the BERT family experience a steep drop in performance on prompts that require them to pick answers which require reasoning about context (e.g., from 89.8% to 18.4% for BERT Large). We present evidence to show that language models exhibit different associative biases, suggesting a need for future work in developing and analyzing frameworks similar to WinoVenti that are tuned to model-specific weaknesses.

IJCNLP-AACL 2021

Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions

masked language models

commonsense inference

challenge set

commonsense

bert

adversarial

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

Previous studies have argued that pre-trained language models encode commonsense relational knowledge (e.g. that apples are edible). However, simultaneous work has revealed that such models are often insensitive to context, even ignoring overt contextual cues such as negations. In this paper, we investigate whether masked language models (the BERT family), can move beyond naive associative biases (e.g., apple -> edible) when the context warrants (e.g. ranking inedible higher when presented with the information that the apple is rotten). We introduce the WinoVenti procedure, which adversarially exploits associative biases in masked language models to create model-specific Winograd-style entailment schemas. Using our constructed WinoVenti challenges set of over 2000 schemas, we discover that language models in the BERT family experience a steep drop in performance on prompts that require them to pick answers which require reasoning about context (e.g., from 89.8% to 18.4% for BERT Large). We present evidence to show that language models exhibit different associative biases, suggesting a need for future work in developing and analyzing frameworks similar to WinoVenti that are tuned to model-specific weaknesses.

technical paper

Mixed initiative in open-domain dialogue requires a system to pro-actively introduce new topics. The one-turn topic transition task explores how a system connects two topics in a cooperative and coherent manner. The goal of the task is to generate a “bridging” utterance connecting the new topic to the topic of the previous conversation turn. We are especially interested in commonsense explanations of how a new topic relates to what has been mentioned before. We first collect a new dataset of human one-turn topic transitions, which we callOTTers. We then explore different strategies used by humans when asked to complete such a task, and notice that the use of a bridging utterance to connect the two topics is the approach used the most. We finally show how existing state-of-the-art text generation models can be adapted to this task and examine the performance of these baselines on different splits of the OTTers data.

OTTers: One-turn Topic Transitions for Open-Domain Dialogue

Recently, there has been significant progress in studying neural networks to translate text descriptions into SQL queries. Despite achieving good performance on some public benchmarks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the models vulnerable to attacks that break the schema linking mechanism. In this work, we investigate the robustness of text-to-SQL models to synonym substitution. In particular, we introduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question paraphrases. We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case attacks. Finally, we present two categories of approaches to improve the model robustness. The first category of approaches utilizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective.

Towards Robustness of Text-to-SQL Models against Synonym Substitution

In order to better understand the reason behind model behaviors (i.e., making predictions), most recent works have exploited generative models to provide complementary explanations. However, existing approaches in NLP mainly focus on ``WHY A'' rather than contrastive ``WHY A NOT B'', which is shown to be able to better distinguish confusing candidates and improve data efficiency in other research fields.In this paper, we focus on generating contrastive explanations with counterfactual examples in NLI and propose a novel \textbf{K}nowledge-\textbf{A}ware \textbf{C}ontrastive \textbf{E}xplanation generation framework (\textbf{KACE}).Specifically, we first identify rationales (i.e., key phrases) from input sentences, and use them as key perturbations for generating counterfactual examples. After obtaining qualified counterfactual examples, we take them along with original examples and external knowledge as input, and employ a knowledge-aware generative pre-trained language model to generate contrastive explanations. Experimental results show that contrastive explanations are beneficial to fit the scenarios by clarifying the difference between the predicted answer and other possible wrong ones. Moreover, we train an NLI model enhanced with contrastive explanations and achieves an accuracy of 91.9\% on SNLI, gaining improvements of 5.7\% against ETPA (``Explain-Then-Predict-Attention'') and 0.6\% against NILE (``WHY A'').

KACE: Generating Knowledge Aware Contrastive Explanations for Natural Language Inference

This work aims to tackle the challenging heterogeneous graph encoding problem in the text-to-SQL task. Previous methods are typically node-centric and merely utilize different weight matrices to parameterize edge types, which 1) ignore the rich semantics embedded in the topological structure of edges, and 2) fail to distinguish local and non-local relations for each node. To this end, we propose a Line Graph Enhanced Text-to-SQL (LGESQL) model to mine the underlying relational features without constructing meta-paths. By virtue of the line graph, messages propagate more efficiently through not only connections between nodes, but also the topology of directed edges. Furthermore, both local and non-local relations are integrated distinctively during the graph iteration. We also design an auxiliary task called graph pruning to improve the discriminative capability of the encoder. Our framework achieves state-of-the-art results (62.8% with Glove, 72.0% with Electra) on the cross-domain text-to-SQL benchmark Spider at the time of writing.

LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations

Sentence embedding methods using natural language inference (NLI) datasets have been successfully applied to various tasks.
However, these methods are only available for limited languages due to relying heavily on the large NLI datasets.
In this paper, we propose DefSent, a sentence embedding method that uses definition sentences from a word dictionary, which performs comparably on unsupervised semantics textual similarity (STS) tasks and slightly better on SentEval tasks than conventional methods.
Since dictionaries are available for many languages, DefSent is more broadly applicable than methods using NLI datasets without constructing additional datasets.
We demonstrate that DefSent performs comparably on unsupervised semantics textual similarity (STS) tasks and slightly better on SentEval tasks to the methods using large NLI datasets.
Our code is publicly available at https://github.com/hpprc/defsent.

DefSent: Sentence Embeddings using Definition Sentences

We introduce categorical modularity, a novel low-resource intrinsic metric to evaluate word embedding quality. Categorical modularity is a graph modularity metric based on the k-nearest neighbor graph constructed with embedding vectors of words from a fixed set of semantic categories, in which the goal is to measure the proportion of words that have nearest neighbors within the same categories. We use a core set of 500 words belonging to 59 neurobiologically motivated semantic categories in 29 languages and analyze three word embedding models per language (FastText, MUSE, and subs2vec). We find moderate to strong positive correlations between categorical modularity and performance on the monolingual tasks of sentiment analysis and word similarity calculation and on the cross-lingual task of bilingual lexicon induction both to and from English. Overall, we suggest that categorical modularity provides non-trivial predictive information about downstream task performance, with breakdowns of correlations by model suggesting some meta-predictive properties about semantic information loss as well.

Evaluating Word Embeddings with Categorical Modularity

Language modeling (LM) for automatic speech recognition (ASR) does not usually incorporate utterance level contextual information. For some domains like voice assistants, however, additional context, such as time at which an utterance was spoken, provides a rich input signal. We introduce an attention mechanism for training neural speech recognition language models on both text and non-linguistic contextual data. When applied to a large dataset of utterances collected by a popular voice assistant platform, our method reduces perplexity by 7.0% relative over a standard LM that does not incorporate contextual information. When evaluated on utterances extracted from the long tail of the dataset, our method improves perplexity by 9.0% relative over a standard LM and by over 2.8% when compared to a state-of-the-art model for contextual LM.

Attention-based Contextual Language Model Adaptation for Speech Recognition

Screenplays refer to characters using different names, pronouns, and nominal expressions.
We need to resolve these mentions to the correct referent character for better story understanding and holistic research in computational narratology.
Coreference resolution of character mentions in screenplays becomes challenging because of the large document lengths, unique structural features like scene headers, interleaving of action and speech passages, and reliance on the accompanying video. 
In this work, we first adapt widely-used annotation guidelines to address domain-specific issues in screenplays.
We develop an automatic screenplay parser to extract the structural information and design coreference rules based upon the structure. 
Our model exploits these structural features and outperforms a benchmark coreference model on the screenplay coreference resolution task.

Annotation and Evaluation of Coreference Resolution in Screenplays

object discovery and segmentation, notably MONet and Slot Attention. While these algorithms learn an object-centric representation just by reconstructing the input image, LORL enables them to further learn to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input. These object-centric concepts derived from language facilitate the learning of object-centric representations. LORL can be integrated with various unsupervised object discovery algorithms that are language-agnostic. Experiments show that the integration of LORL consistently improves the performance of unsupervised object discovery methods on two datasets via the help of language. We also show that concepts learned by LORL, in conjunction with object discovery methods, aid downstream tasks such as referring expression comprehension.

Language-Mediated, Object-Centric Representation Learning

Enthusiasm plays an important role in engaging communication. It enables speakers to be distinguished and remembered, creating an emotional bond that inspires and motivates their addressees to act, listen, and coordinate (Bettencourt et al., 1983). Although people can easily identify enthusiasm, this is a rather difficult task for machines due to the lack of resources and models that can help them understand or generate enthusiastic behavior. 
We introduce Entheos, the first multimodal dataset for studying enthusiasm composed of video, audio, and text. We present several baseline models and an ablation study using different features, showing the importance of pitch, loudness, and discourse relation parsing in distinguishing enthusiastic communication.

Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions

Downloads

Next from IJCNLP-AACL 2021

OTTers: One-turn Topic Transitions for Open-Domain Dialogue

Similar lecture

TIMEDIAL: Temporal Commonsense Reasoning in Dialog

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from IJCNLP-AACL 2021

OTTers: One-turn Topic Transitions for Open-Domain Dialogue

Similar lecture

TIMEDIAL: Temporal Commonsense Reasoning in Dialog

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads