Dominican Republic

This paper provides an overview of NVIDIA NeMo&#39;s neural machine translation systems for the constrained data track of the WMT21 News and Biomedical Shared Translation Tasks. Our news task submissions for English-German (En-De) and English-Russian (En-Ru) are built on top of a baseline transformer-based sequence-to-sequence model \cite{vaswani2017}. Specifically, we use a combination of 1) checkpoint averaging 2) model scaling 3) data augmentation with backtranslation and knowledge distillation from right-to-left factorized models 4) finetuning on test sets from previous years 5) model ensembling 6) shallow fusion decoding with transformer language models and 7) noisy channel re-ranking. Additionally, our biomedical task submission for English $\leftrightarrow$ Russian uses a biomedically biased vocabulary and is trained from scratch on news task data, medically relevant text curated from the news task dataset, and biomedical data provided by the shared task. Our news system achieves a sacreBLEU score of 39.5 on the WMT&#39;20 En-De test set outperforming the best submission from last year&#39;s task of 38.8. Our biomedical task Ru-En and En-Ru systems reach BLEU scores of 43.8 and 40.3 respectively on the WMT&#39;20 Biomedical Task Test set, outperforming the previous year&#39;s best submissions.

EMNLP 2021

NVIDIA NeMo&#39;s Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

This paper provides an overview of NVIDIA NeMo's neural machine translation systems for the constrained data track of the WMT21 News and Biomedical Shared Translation Tasks. Our news task submissions for English-German (En-De) and English-Russian (En-Ru) are built on top of a baseline transformer-based sequence-to-sequence model \cite{vaswani2017}. Specifically, we use a combination of 1) checkpoint averaging 2) model scaling 3) data augmentation with backtranslation and knowledge distillation from right-to-left factorized models 4) finetuning on test sets from previous years 5) model ensembling 6) shallow fusion decoding with transformer language models and 7) noisy channel re-ranking. Additionally, our biomedical task submission for English $\leftrightarrow$ Russian uses a biomedically biased vocabulary and is trained from scratch on news task data, medically relevant text curated from the news task dataset, and biomedical data provided by the shared task. Our news system achieves a sacreBLEU score of 39.5 on the WMT'20 En-De test set outperforming the best submission from last year's task of 38.8. Our biomedical task Ru-En and En-Ru systems reach BLEU scores of 43.8 and 40.3 respectively on the WMT'20 Biomedical Task Test set, outperforming the previous year's best submissions.

NVIDIA NeMo's Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21

workshop paper

Existing approaches for audio-language task- specific prediction focus on building compli- cated late-fusion mechanisms. However, these models face challenges of overfitting with lim- ited labels and poor generalization. In this pa- per, we present a Cross-modal Transformer for Audio-and-Language, i.e., CTAL, which aims to learn the intra- and inter- modali- ties connections between audio and language through two proxy tasks from a large num- ber of audio-and-language pairs: masked language modeling and masked cross-modal acoustic modeling. After fine-tuning our CTAL model on multiple downstream audio- and-language tasks, we observe significant improvements on different tasks, including emotion classification, sentiment analysis, and speaker verification. Furthermore, we design a fusion mechanism in the fine-tuning phase, which allows CTAL to achieve better perfor- mance. Lastly, we conduct detailed ablation studies to demonstrate that both our novel cross-modality fusion component and audio- language pre-training methods contribute to the promising results.

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Opinion Role Labeling (ORL), aiming to identify the key roles of opinion, has received increasing interest. Unlike most of the previous works focusing on the English language, in this paper, we present the first work of Chinese ORL. We construct a Chinese dataset by manually translating and projecting annotations from a standard English MPQA dataset. Then, we investigate the effectiveness of cross-lingual transfer methods, including model transfer and corpus translation. We exploit multilingual BERT with Contextual Parameter Generator and Adapter methods to examine the potentials of unsupervised cross-lingual learning and our experiments and analyses for both bilingual and multilingual transfers establish a foundation for the future research of this task.

Chinese Opinion Role Labeling with Corpus Translation: A Pivot Study

Aspect-based sentiment analysis (ABSA) predicts the sentiment polarity towards a particular aspect term in an input sentence, which is significantly important in real-world applications. To perform ABSA, one faces a long-standing challenge that ABSA requires the model to have a good understanding of the contextual information, especially the particular patterns that suggests the sentiment polarity. However, these patterns vary in different sentences, especially when the sentences come from different sources (domains). Although leveraging extra labeled data from different sources (domains) is always a promising solution to address the challenge, in practical applications, these labeled data are always stored at different locations and might be inaccessible to each other due to privacy or legal concerns (e.g., the data are owned by different companies). To address the issue and benefit from extra labeled data, we propose a novel ABSA model with federated learning (FL) adopted to overcome the data isolation limitations and topic memory (TM) proposed to enhance FL by accommodating the cases of data from diverse sources (domains). Particularly, TM contributes to overcoming difficulties in identifying domains due to data inaccessibility by providing useful categorical information for localized predictions. Experimental results on a simulated environment for FL with three nodes demonstrate the effectiveness of our approach, where TM-FL outperforms different baselines including some well-designed FL frameworks.

Improving Federated Learning for Aspect-based Sentiment Analysis via Topic Memories

As an important task in opinion mining, comparative opinion mining aims to identify comparative sentences from product reviews, extract the comparative elements, and obtain the corresponding comparative opinion tuples. However, most previous studies simply regarded comparative tuple extraction as comparative element extraction, but ignored the fact that many comparative sentences may contain multiple comparisons. The comparative opinion tuples defined in these studies also failed to explicitly provide comparative preferences. To address these limitations, in this work we first introduce a new Comparative Opinion Quintuple Extraction (COQE) task, to identify comparative sentences from product reviews and extract all comparative opinion quintuples (Subject, Object, Comparative Aspect, Comparative Opinion, Comparative Preference). Secondly, based on the existing comparative opinion mining corpora, we make supplementary annotations and construct three datasets for the COQE task. Finally, we benchmark the COQE task by proposing a new BERT-based multi-stage approach as well as three baseline systems extended from previous methods. Experimental results show that the new approach significantly outperforms three baseline systems on three datasets for the COQE task.

Comparative Opinion Quintuple Extraction from Product Reviews

This paper describes Lingua Custodia's submission to the WMT21 shared task on machine translation using terminologies. 
We consider three directions, namely English to French, Russian, and Chinese.
We rely on a Transformer-based architecture as a building block, and we explore a method which introduces two main changes to the standard procedure to handle terminologies. The first one consists in augmenting the training data in such a way as to encourage the model to learn a copy behavior when it encounters terminology constraint terms. The second change is constraint token masking, whose purpose is to ease copy behavior learning and to improve model generalization. 
Empirical results show that our method satisfies most terminology constraints while maintaining high translation quality.

Lingua Custodia's participation at the WMT 2021 Machine Translation using Terminologies shared task

We describe the EdinSaar submission to the shared task of Multilingual Low-Resource Translation for North Germanic Languages at the Sixth Conference on Machine Translation (WMT2021). We submit multilingual translation models for translations to/from Icelandic (is), Norwegian-Bokmal (nb), and Swedish (sv). We employ various experimental approaches, including multilingual pre-training, back-translation, fine-tuning, and ensembling. In most translation directions, our models outperform other submitted systems.

EdinSaar@WMT21: North-Germanic Low-Resource Multilingual NMT

We present the joint contribution of IST and Unbabel to the WMT 2021 Shared Task on Quality Estimation. Our team participated on two tasks: Direct Assessment and Post-Editing Effort, encompassing a total of 35 submissions. For all submissions, our efforts focused on training multilingual models on top of OpenKiwi predictor-estimator architecture, using pre-trained multilingual encoders combined with adapters. We further experiment with and uncertainty-related objectives and features as well as training on out-of-domain direct assessment data.

IST-Unbabel 2021 Submission for the Quality Estimation Shared Task

This paper describes Charles University sub-mission for Terminology translation shared task at WMT21. The objective of this task is to design a system which translates certain terms based on a provided terminology database, while preserving high overall translation quality. We competed in English-French language pair. Our approach is based on providing the desired translations alongside the input sentence and training the model to use these provided terms. We lemmatize the terms both during the training and inference, to allow the model to learn how to produce correct surface forms of the words, when they differ from the forms provided in the terminology database.

CUNI systems for WMT21: Multilingual Low-Resource Translation for Indo-European Languages Shared Task

This work introduces a simple regressive ensemble for evaluating machine translation quality based on a set of novel and established metrics. 
We evaluate the ensemble using a correlation to expert-based MQM scores of the WMT 2021 Metrics workshop.
In both monolingual and zero-shot cross-lingual settings, we show a significant performance improvement over single metrics.
In the cross-lingual settings, we also demonstrate that an ensemble approach is well-applicable to unseen languages.
Furthermore, we identify a strong reference-free baseline that consistently outperforms the commonly-used BLEU and METEOR measures and significantly improves our ensemble's performance.

Regressive Ensemble for Machine Translation Quality Evaluation

This paper presents the University of Edinburgh's constrained submissions of English-German and English-Hausa systems to the WMT 2021 shared task on news translation. We build En-De systems in three stages: corpus filtering, back-translation, and fine-tuning. For En-Ha we use an iterative back-translation approach on top of pre-trained En-De models and investigate vocabulary embedding mapping.

Content not yet available

NVIDIA NeMo's Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21

Next from EMNLP 2021

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Similar lecture

HW-TSC's Participation in the WMT 2021 Large-Scale Multilingual Translation Task

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES