Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recently proposed pretrained language models (PLMs) and analyze the impact of different task-adaptive pretraining strategies for PLMs in graph-to-text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that the PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further. In particular, we report new state-of-the-art BLEU scores of 49.72 on LDC2017T10, 59.70 on WebNLG, and 25.66 on AGENDA datasets - a relative improvement of 31.8%, 4.5%, and 42.4%, respectively. In an extensive analysis, we identify possible reasons for the PLMs' success on graph-to-text tasks. We find evidence that their knowledge about true facts helps them perform well even when the input graph representation is reduced to a simple bag of node and edge labels.

Investigating Pretrained Language Models for Graph-to-Text Generation

Massively multilingual processing is becoming an important NLP scenario, and pretrained language and machine translation models that are truly multilingual are proposed. While word alignment is an important task, most word-alignment algorithms rely on bitexts only and do not leverage the multilingual models nor multiparallel corpora. I will present how we can exploit multiparallelity of a corpus to find new word alignment links. I will present two graph algorithms for edge prediction: one inspired by recommender systems and one based on network link prediction. I will show that experimental results can achieve absolute improvements of F1 of up to 21% over the baseline bilingual word aligner in different datasets.

Graph Algorithms for Multiparallel Word Alignment

How does the input segmentation of pretrained language models (PLMs) 
affect their interpretations of complex words?
We present the first study investigating this question,
taking BERT as the example PLM and focusing on its semantic representations
of English derivatives.
We show that PLMs can be interpreted as serial dual-route models, i.e.,
the meanings of complex words are either stored or else need to 
be computed from the subwords,
which implies that maximally meaningful input tokens should
allow for the best generalization on new words. This hypothesis
is confirmed by a series of semantic probing 
tasks on which DelBERT (Derivation leveraging BERT), a model with 
derivational input segmentation, substantially outperforms
BERT with WordPiece segmentation. Our results suggest
that the generalization capabilities of PLMs
could be further improved if a morphologically-informed vocabulary 
of input tokens were used.

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

Static word embeddings that represent words by a single vector 
cannot capture the variability of word meaning in different linguistic
and extralinguistic contexts. Building on prior work on contextualized 
and dynamic word embeddings, we introduce dynamic contextualized 
word embeddings that represent words as a function of both linguistic
and extralinguistic context. Based on a pretrained language model (PLM), 
dynamic contextualized word embeddings model time and social space 
jointly, which makes them attractive for a range of NLP tasks involving
semantic variability. We highlight potential application scenarios by means 
of qualitative and quantitative analyses on four English datasets.

Dynamic Contextualized Word Embeddings

Few-shot crosslingual transfer
has been shown to outperform its
zero-shot counterpart with pretrained encoders like multilingual BERT.
Despite its growing popularity, little to no attention has been paid
to standardizing and analyzing the design of few-shot experiments.
In this work, we highlight a fundamental risk posed by
this shortcoming, illustrating
that the model exhibits a high degree
of sensitivity to the selection of few shots.
We conduct a large-scale experimental study on 40 sets of sampled few
shots for six diverse NLP tasks across up to 40 languages.
We provide an analysis of success and failure cases of few-shot
transfer, which highlights the role of lexical features.
Additionally, we show that a straightforward full model finetuning
approach is quite effective for few-shot transfer, outperforming
several state-of-the-art few-shot approaches.
As a step towards standardizing few-shot crosslingual
experimental designs, we make our sampled few shots publicly
available.

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained language model and map the predicted words to labels. Manually defining this mapping between words and labels requires both domain expertise and an understanding of the language model’s abilities. To mitigate this issue, we devise an approach that automatically finds such a mapping given small amounts of training data. For a number of tasks, the mapping found by our approach performs almost as well as hand-crafted label-to-word mappings.

Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

Pretrained language models (PLMs) learn stereotypes held by humans and reflected in text from their training corpora, including gender bias. When PLMs are used for downstream tasks such as picking candidates for a job, people’s lives can be negatively affected by these learned stereotypes. Prior work usually identifies a linear gender subspace and removes gender information by eliminating the subspace. Following this line of work, we propose to use DensRay, an analytical method for obtaining interpretable dense subspaces. We show that DensRay performs on-par with prior approaches, but provide arguments that it is more robust and provide indications that it preserves language model performance better. By applying DensRay to attention heads and layers of BERT we show that gender information is spread across all attention heads and most of the layers. Also we show that DensRay can obtain gender bias scores on both token and sentence levels. Finally, we demonstrate that we can remove bias multilingually, e.g., from Chinese, using only English training data.

Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations



[![](https://s3.amazonaws.com/pf-upload-01/u-59356/0/2021-09-18/me23j72/button_workshop-website.png)](https://sites.google.com/view/3rdnlp4convai/) 

W3: 3rd Workshop on NLP for Conversational AI (NLP4ConvAI)

workshop paper

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/" seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

Virtual Poster Session III: Machine Translation and Multilinguality

poster

**Session Chair: ** Xuanjing Huang 
**Session Volunteer: ** Shayekh Bin Islam

10D: Phonology, Morphology and Word Segmentation #1

technical paper

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

IJCNLP-AACL 2021

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

**Session Chair: ** Frank Ferraro 
**Session Volunteer: ** Ricardo Muñoz Sánchez

15E: Semantics: Lexical Semantics #2

Poster 3K: Machine Translation and Multilinguality

POSTER20: Text Classification, Text Mining and Knowledge Extraction

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING 2020

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

Hinrich Schutze

8

10

Presentations

Investigating Pretrained Language Models for Graph-to-Text Generation

Graph Algorithms for Multiparallel Word Alignment

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

Dynamic Contextualized Word Embeddings

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES