Thailand

Lexical Complexity Prediction (LCP) involves assigning a difficulty score to a particular word or expression, in a text intended for a target audience. In this paper, we introduce a deep learning system based on a pre-trained transformer encoder, for word and Multi-Word Expression (MWE) complexity prediction. The proposed model is trained to minimize both the Root Mean Square Error (RMSE) and the auxiliary loss associated to the negative Pearson Correlation. The two losses are combined using the uncertainty loss weighting. We investigate two pre-trained transformer networks, namely BERT and RoBERTa. Moreover, we evaluate both single-task and joint training of word and MWE complexity prediction sub-tasks.

IJCNLP-AACL 2021

A Deep Learning Model-based Pre-trained Transformer Encoder for Lexical Complexity

lexical complexity prediction

contextualized word embedding

text simplification

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

workshop paper

Neural machine translation models are often biased toward the limited translation references seen during training. To amend this form of overfitting, in this paper we propose fine-tuning the models with a novel training objective based on the recently-proposed BERTScore evaluation metric. BERTScore is a scoring function based on contextual embeddings that overcomes the typical limitations of n-gram-based metrics (e.g. synonyms, paraphrases), allowing translations that are different from the references, yet close in the contextual embedding space, to be treated as substantially correct. To be able to use BERTScore as a training objective, we propose three approaches for generating soft predictions, allowing the network to remain completely differentiable end-to-end. Experiments carried out over four, diverse language pairs show improvements of up to 0.58 pp (3.28%) in BLEU score and up to 0.76 pp (0.98%) in BERTScore (F_BERT) when fine-tuning a strong baseline.

BERTTune: Fine-Tuning Neural Machine Translation with BERTScore

Neural chat translation aims to translate bilingual conversational text, which has a broad application in international exchanges and cooperation. Despite the impressive performance of sentence-level and context-aware Neural Machine Translation (NMT), there still remain challenges to translate bilingual conversational text due to its inherent characteristics such as role preference, dialogue coherence, and translation consistency. In this paper, we aim to promote the translation quality of conversational text by modeling the above properties. Specifically, we design three latent variational modules to learn the distributions of bilingual conversational characteristics. Through sampling from these learned distributions, the latent variables, tailored for role preference, dialogue coherence, and translation consistency, are incorporated into the NMT model for better translation. We evaluate our approach on the benchmark dataset BConTrasT (English<->German) and a self-collected bilingual dialogue corpus, named BMELD (English<->Chinese). Extensive experiments show that our approach notably boosts the performance over strong baselines by a large margin and significantly surpasses some state-of-the-art context-aware NMT models in terms of BLEU and TER. Additionally, we make the BMELD dataset publicly available for the research community.

Modeling Bilingual Conversational Characteristics for Neural Chat Translation

Multilingual neural machine translation with a single model has drawn much attention due to its capability to deal with multiple languages. However, the current multilingual translation paradigm often makes the model tend to preserve the general knowledge, but ignore the language-specific knowledge. Some previous works try to solve this problem by adding various kinds of language-specific modules to the model, but they suffer from the parameter explosion problem and require specialized manual design. To solve these problems, we propose to divide the model neurons into general and language-specific parts based on their importance across languages. The general part is responsible for preserving the general knowledge and participating in the translation of all the languages, while the language-specific part is responsible for preserving the language-specific knowledge and participating in the translation of some specific languages. Experimental results on several language pairs, covering IWSLT and Europarl corpus datasets, demonstrate the effectiveness and universality of the proposed method.

Importance-based Neuron Allocation for Multilingual Neural Machine Translation

Few-shot crosslingual transfer
has been shown to outperform its
zero-shot counterpart with pretrained encoders like multilingual BERT.
Despite its growing popularity, little to no attention has been paid
to standardizing and analyzing the design of few-shot experiments.
In this work, we highlight a fundamental risk posed by
this shortcoming, illustrating
that the model exhibits a high degree
of sensitivity to the selection of few shots.
We conduct a large-scale experimental study on 40 sets of sampled few
shots for six diverse NLP tasks across up to 40 languages.
We provide an analysis of success and failure cases of few-shot
transfer, which highlights the role of lexical features.
Additionally, we show that a straightforward full model finetuning
approach is quite effective for few-shot transfer, outperforming
several state-of-the-art few-shot approaches.
As a step towards standardizing few-shot crosslingual
experimental designs, we make our sampled few shots publicly
available.

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

In SemEval-2021 Task-5 Toxic Spans Detection, the focus is on detecting toxic spans within English passages.

Our contributions in our paper:
- Performance Analysis of BERT-based models on Token Classification and Span-Prediction approaches.
- Hybrid Approaches: Multi-Span, Span+Token, LSTM-CRF, and Union/Intersection.
- A thorough ablative analysis and analysis/interpretation of our observed results. 

Our best submission is a combination of SpanBERT Span Predictor and RoBERTa Token Classifier predictions - achieves an F₁ score of 0.6753 on the test set. 
Our best post-eval F₁ score is 0.6895 on intersection of predicted offsets from top-3 RoBERTa Token Classification checkpoints.

Toxic Spans Detection Leveraging BERT-based Token Classification and Span Prediction Techniques

Research in Natural Language Processing is making rapid advances, resulting in the publication of a large number of research papers. Finding relevant research papers and their contribution to the domain is a challenging problem. In this paper, we address this challenge via the SemEval 2021 Task 11: NLPContri- butionGraph, by developing a system for a research paper contributions-focused knowledge graph over Natural Language Processing literature. The task is divided into three sub-tasks: extracting contribution sentences that show important contributions in the research article, extracting phrases from the contribution sentences, and predicting the information units in the research article together with triplet formation from the phrases. The proposed system is agnostic to the subject domain and can be applied for building a knowledge graph for any area. We found that transformer-based language models can significantly improve exist- ing techniques and utilized the SciBERT-based model. Our first sub-task uses Bidirectional LSTM (BiLSTM) stacked on top of SciBERT model layers, while the second sub-task uses Conditional Random Field (CRF) on top of SciBERT with BiLSTM. The third sub-task uses a combined SciBERT based neural approach with heuristics for information unit pre- diction and triplet formation from the phrases. Our system achieved F1 score of 0.38 in end-to-end pipeline testing, F1 score of 0.63 in phrase extraction testing, and 0.76 in triplet ex- traction testing.

KnowGraph@IITK at SemEval-2021 Task 11: Building KnowledgeGraph for NLP Research

In this paper, we describe our system submitted to SemEval 2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense. The task aims at predicting whether the given text is humorous, the average humor rating given by the annotators, and whether the humor rating is controversial. In addition, the task also involves predicting how offensive the text is. Our approach adopts the DeBERTa architecture with disentangled attention mechanism, where the attention scores between words are calculated based on their content vectors and relative position vectors. We also took advantage of the pre-trained language models and fine-tuned the DeBERTa model on all the four subtasks. We experimented with several BERT-like structures and found that the large DeBERTa model generally performs better. During the evaluation phase, our system achieved an F-score of 0.9480 on subtask 1a, an RMSE of 0.5510 on subtask 1b, an F-score of 0.4764 on subtask 1c, and an RMSE of 0.4230 on subtask 2a (rank 3 on the leaderboard).

HumorHunter at SemEval-2021 Task 7: Humor and Offense Recognition with Disentangled Attention

The present work aims at assigning a complexity score between 0 and 1 to a target word or phrase in a given sentence. For each Single Word Target, a Random Forest Regressor is trained on a feature set consisting of lexical, semantic, and syntactic information about the target. For each Multiword Target, a set of individual word features is taken along with single word complexities in the feature space. The system yielded the Pearson correlation of 0.7402 and 0.8244 on the test set for the Single and Multiword Targets, respectively.

LangResearchLabNC at SemEval-2021 Task 1: Linguistic Feature Based Modelling for Lexical Complexity

Recently, there has been an interest in factual verification and prediction over structured data like tables and graphs. To circumvent any false news incident, it is necessary to not only model and predict over structured data efficiently but also to explain those predictions. In this paper, as part of the SemEval-2021 Task 9, we tackle the problem of fact verification and evidence finding over tabular data. There are two subtasks. Given a table and a statement/fact, subtask A determines whether the statement is inferred from the tabular data, and subtask B determines which cells in the table provide evidence for the former subtask. We make a comparison of the baselines and state-of-the-art approaches over the given SemTabFact dataset. We also propose a novel approach CellBERT to solve evidence finding as a form of the Natural Language Inference task. We obtain a 3- way F1 score of 0.69 on subtask A and an F1 score of 0.65 on subtask B.

BreakingBERT@IITK at SemEval-2021 Task 9 : Statement Verification and Evidence Finding with Tables

This paper describes our system for Task 4 of SemEval-2021: Reading Comprehension of Abstract Meaning (ReCAM). We participated in all subtasks where the main goal was to predict an abstract word missing from a statement.
We fine-tuned the pre-trained masked language models namely BERT and ALBERT and used an Ensemble of these as our submitted system on Subtask 1 (ReCAM-Imperceptibility) and Subtask 2 (ReCAM-Nonspecificity). For Subtask 3 (ReCAM-Intersection), we submitted the ALBERT model as it gives the best results. We tried multiple approaches and found that Masked Language Modeling(MLM) based approach works the best.

Content not yet available

A Deep Learning Model-based Pre-trained Transformer Encoder for Lexical Complexity

Next from IJCNLP-AACL 2021

BERTTune: Fine-Tuning Neural Machine Translation with BERTScore

Similar lecture

SemEval-2021 Task 1: Lexical Complexity Prediction

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES