Thailand

Analogies play a central role in human commonsense reasoning. The ability to recognize analogies such as “eye is to seeing what ear is to hearing”, sometimes referred to as analogical proportions, shape how we structure knowledge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the language model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using benchmarks obtained from educational settings, as well as more commonly used datasets. We find that off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations, and results are highly sensitive to model architecture and hyperparameters. Overall the best results were obtained with GPT-2 and RoBERTa, while configurations using BERT were not able to outperform word embedding models. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations.

IJCNLP-AACL 2021

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

relational knowledge

analogy

lexical semantics

knowledge representation

language model

# Welcome everyone to ACL 2022!

The 60th Annual Meeting of the Association for Computational Linguistics is taking place May 22-27, 2022 as a hybrid event, in Dublin and online. We are happy to welcome all of you to this anniversary edition with an almost 50-50 in-person and virtual participation. 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, three amazing keynote events and two new initiatives of invited talks: Spotlight Talks for Young Rising Stars (STIRS) and The Next Big Idea Talks. Posters (including Findings of ACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop will have an oral session and a poster session as part of Poster Session 1. The program also features eight Tutorials and 28 Workshops. 

 
We wish you a wonderful conference! 
[**The ACL 2022 Organizing Committee**](https://www.2022.aclweb.org/organisers)
 
[**Conference Handbook**](https://drive.google.com/file/d/1_BUCMfhMVrjG9E2e71aHdHeE28KSje0l/view?usp=sharing) 
[**Mini Handbook**](https://drive.google.com/file/d/1qlBKl0wzmlVF1oCeMQl3BahLd9nLP5Ce/view?usp=sharing) 
[**Posters and Demo guides**](https://drive.google.com/file/d/1UucMAoCNncIOaH1rMMDa0owuG9qgvJTG/view?usp=sharing)

ACL 2022

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing (NLP). 

technical paper

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

How does the input segmentation of pretrained language models (PLMs) 
affect their interpretations of complex words?
We present the first study investigating this question,
taking BERT as the example PLM and focusing on its semantic representations
of English derivatives.
We show that PLMs can be interpreted as serial dual-route models, i.e.,
the meanings of complex words are either stored or else need to 
be computed from the subwords,
which implies that maximally meaningful input tokens should
allow for the best generalization on new words. This hypothesis
is confirmed by a series of semantic probing 
tasks on which DelBERT (Derivation leveraging BERT), a model with 
derivational input segmentation, substantially outperforms
BERT with WordPiece segmentation. Our results suggest
that the generalization capabilities of PLMs
could be further improved if a morphologically-informed vocabulary 
of input tokens were used.

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

Prior studies in multilingual language modeling (e.g., Cotterell et al., 2018; Mielke et al., 2019) disagree on whether or not inflectional morphology makes languages harder to model. We attempt to resolve the disagreement and extend those studies. We compile a larger corpus of 145 Bible translations in 92 languages and a larger number of typological features. We fill in missing typological data for several languages and consider corpus-based measures of morphological complexity in addition to expert-produced typological features. We find that several morphological measures are significantly associated with higher surprisal when LSTM models are trained with BPE-segmented data. We also investigate linguistically motivated subword segmentation strategies like Morfessor and Finite-State Transducers (FSTs) and find that these segmentation strategies yield better performance and reduce the impact of a language’s morphology on language modeling.

Morphology Matters: A Multilingual Language Modeling Analysis

Arabic diacritization is a fundamental task for Arabic language processing. Previous studies have demonstrated that automatically generated knowledge can be helpful to this task. However, these studies regard the auto-generated knowledge instances as gold references, which limits their effectiveness since such knowledge is not always accurate and inferior instances can lead to incorrect predictions. In this paper, we propose to use regularized decoding and adversarial training to appropriately learn from such noisy knowledge for diacritization. Experimental results on two benchmark datasets show that, even with quite flawed auto-generated knowledge, our model can still learn adequate diacritics and outperform all previous studies, on both datasets.

Improving Arabic Diacritization with Regularized Decoding and Adversarial Training

Subword segmentation algorithms have been a \textit{de facto} choice when building neural machine translation systems. However, most of them need to learn a segmentation model based on some heuristics, which may produce sub-optimal segmentation. This can be problematic in some scenarios when the target language has rich morphological changes or there is not enough data for learning compact composition rules.
Translating at fully character level has the potential to alleviate the issue, but empirical performances of character-based models has not been fully explored. 
In this paper, we present an in-depth comparison between character-based and subword-based NMT systems under three settings: translating to typologically diverse languages, training with low resource, and adapting to unseen domains. Experiment results show strong competitiveness of character-based models. Further analyses show that compared to subword-based models, character-based models are better at handling morphological phenomena, generating rare and unknown words, and more suitable for transferring to unseen domains.

When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation

Query focused summarization (QFS) models aim to generate summaries from source documents that can answer the given query. Most previous work on QFS only considers the query relevance criterion when producing the summary. However, studying the effect of answer relevance in the summary generating process is also important. In this paper, we propose QFS-BART, a model that incorporates the explicit answer relevance of the source documents given the query via a question answering model, to generate coherent and answer-related summaries. Furthermore, our model can take advantage of large pre-trained models which improve the summarization performance significantly. Empirical results on the Debatepedia dataset show that the proposed model achieves the new state-of-the-art performance.

Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance

We propose a Bi-Directional Manifold Alignment (BDMA) that learns a non-linear mapping between two manifolds by explicitly training it to be bijective. We demonstrate BDMA by training a model for a pair of languages rather than individual, directed source and target combinations, reducing the number of models by 50%. We show that models trained with BDMA in the "forward'' (source to target) direction can successfully map words in the "reverse'' (target to source) direction, yielding equivalent (or better) performance to standard unidirectional translation models where the source and target language is flipped. We also show how BDMA reduces the overall size of the model.

Learning a Reversible Embedding Mapping using Bi-Directional Manifold Alignment

In this paper, we investigate reasoning over natural language based on modern neural networks.
Prior efforts such as PRover train the question answering module and proof generation module through a multi-task classification framework, which does not consider explicit inter-dependency among nodes and edges in the proof graph.
In this paper, we propose PRobr,
a novel approach for answer prediction and proof generation.
PRobr defines a joint probabilistic distribution over all possible proofs and answers via an induced graphical model. 
We then optimize the model using variational approximation on top of neural textual representation. 
Experiments on several datasets verify the effectiveness of PRobr under multiple diverse settings (fully supervised, few-shot and zero-shot evaluation),
e.g., achieving 10\%-30\% improvement on QA accuracy in few/zero-shot evaluation.

Probabilistic Graph Reasoning for Natural Proof Generation

In this paper, we consider a realistic scenario on stance detection with more application potential, i.e., zero-shot and few-shot stance detection, which identifies stances for a wide range of topics with no or very few training examples. Conventional data-driven approaches may be not applicable to the above zero-shot and few-shot scenarios. For human beings, the commonsense knowledge is a crucial element of understanding and reasoning. In the absence of annotated data and cryptic expression of users' stance, we believe that introducing commonsense relational knowledge as support for reasoning can further improve the generalization and reasoning ability of the model in the zero-shot and few-shot scenarios. Furthermore, we introduce a commonsense knowledge enhanced module to exploit both the structural-level and semantic-level information of the relational knowledge. Extensive experiments demonstrate that our model outperforms the state-of-the-art methods for zero-shot and few-shot stance detection task.

Enhancing Zero-shot and Few-shot Stance Detection with Commonsense Knowledge Graph

Conversational Machine Reading (CMR) aims at answering questions in complicated interactive scenarios. Machine needs to answer questions through interactions with users based on given rule document, user scenario and dialogue history, and even initiatively asks questions for clarification if necessary. Namely, the answer to the task needs a machine in the response of either "Yes, No, Irrelevant" or to raise a follow-up question for further clarification. To effectively capture multiple objects in such a challenging task, graph modeling is supposed to be adopted, though it is surprising that this does not happen until this work proposes a dialogue graph modeling framework by incorporating two complementary graph models, i.e., explicit discourse graph and implicit discourse graph, which respectively capture explicit and implicit interactions hidden in the rule documents. The proposed model is evaluated on the ShARC benchmark and achieves new state-of-the-art by first exceeding the milestone accuracy score of 80%.

Dialogue Graph Modeling for Conversational Machine Reading

Indonesian language is heavily riddled with colloquialism whether in written or spoken forms. In this paper, we identify a class of Indonesian colloquial words that have undergone morphological transformations from their standard forms, categorize their word formations, and propose a benchmark dataset of Indonesian Colloquial Lexicons (IndoCollex) consisting of informal words on Twitter expertly annotated with their standard forms and their word formation types/tags. We evaluate several models for character-level transduction to perform morphological word normalization on this testbed to understand their failure cases and provide baselines for future work. As IndoCollex catalogues word formation phenomena that are also present in the non-standard text of other languages, it can also provide an attractive testbed for methods tailored for cross-lingual word normalization and non-standard word formation.

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

Downloads

Next from IJCNLP-AACL 2021

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

Similar lecture

Distilling Relation Embeddings from Pre-trained Language Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from IJCNLP-AACL 2021

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

Similar lecture

Distilling Relation Embeddings from Pre-trained Language Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads