Singapore

Instruction following language models have shown a remarkable ability to perform a wide range of tasks with little to no additional training data. Do these abilities come from a revolution in pre-training and instruction-following, or are there other more mundane explanations for how these models work? In this talk, I will discuss our efforts to answer these questions by replicating instruction-following models that generalize across tasks, studying the consistency of these models across different task formats, and building tests for benchmark contamination in pretraining.

EMNLP 2023

Understanding generalization for instruction following and black-box language models | VIDEO

invited talk

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

Recently, using deep neural networks for machine translation (MT) tasks has received great attention. To learn more abstract representations of the input and store them as continuous vectors, these networks need a large amount of data. However, very few research studies have been conducted, and only a few data points are available for Amharic. The progress of the Amharic to English MT task is affected by a lack of a relatively large and available benchmark dataset. This paper presents the first relatively large-scale Amharic-English parallel corpora (1.1M) for the MT task. We ran experiments using a pre-trained language model (M2M100\_48M) and Transformer from scratch. The pre-trained models outperformed the baseline Transformer model.

AmEn: Amharic-English Large Parallel Corpus for Machine Translation

Question Answering (QA) systems return concise answers or answer lists based on natural language text, which uses a given context document. Large amounts of resources go into curating QA datasets, to push the advancement of robust models. Although there has been a surge in QA datasets for languages such as English this is not the case for Amharic. Indeed, there is no published or publicly available Amharic QA dataset. Hence, to foster further research in Amharic QA, we present the first Amharic QA dataset (Am-QuAD). We developed this dataset by crowdsourcing 2,628 question-answer pairs from over 378 Wikipedia articles written in Amharic. Additionally, we run an XLMR-based baseline model to spark open-domain QA research interest. The best-performing baseline achieves an F-score of 69.58 and 71.74 in reader-retriever QA and reading comprehension settings.

Am-QuAD: Amharic Question Answering Dataset

Adversarial Robustness of Transformer-based Arabic Offensive Language Detectors

This study proposes a data-centric approach
for Neural Machine Translation (NMT) in low
resource languages, focusing on Moroccan di
alect called Darija. Synthetic data genera
tion from a Moroccan Arabic-English dictio
nary is employed, along with DarijaBERT and
ChatGPT for sequence generation and filter
ing. Transfer learning using mT5 models is
utilized, comparing a baseline model with a
synthetic data augmentation model. Results
show a significant improvement in translation
quality, demonstrating the efficacy of synthetic
data augmentation in addressing low-resource
NMT challenges.

Synthetic Data Augmentation for Low-Resource NMT A Case Study on Moroccan dialect

The Fon language, spoken by an average 2 million of people, is a truly low-resourced African language, with a limited online presence, and existing datasets (just to name but a few). Multitask learning is a learning paradigm that aims to improve the generalization capacity of a model by sharing knowledge across different but related tasks: this could be prevalent in very data-scarce scenarios. In this paper, we present the first explorative approach to multitask learning, for model capabilities enhancement in Natural Language Processing for the Fon language. Specifically, we explore the tasks of Named Entity Recognition (NER) and Part of Speech Tagging (POS) for Fon. We leverage two language model heads as encoders to build shared representations for the inputs, and we use linear layer blocks for classification relative to each task. Our results on the NER and POS tasks for Fon, show competitive (or better) performances compared to several multilingual pretrained language models finetuned on single tasks. Additionally, we perform a few ablation studies to leverage the efficiency of two different loss combination strategies and find out that the equal loss weighting approach works best in our case. Our code is open-sourced at https://github.com/bonaventuredossou/multitask_fon.

FonMTL: Towards Multitask Learning for the Fon Language

Currently there are a lot of State of the art (SOTA) models for text generation. These models are capable enough to generate text that looks as if they are written by a human. Despite the assistance it can provide for humans, these models can be leveraged to malicious tasks such as plagiarism. Especially when it comes to academia, there is an ethical concern based on the authenticity of the content in articles. Thus it is vital to distinguish machine generated text from human written content. Several novel detection tools claim higher detectability accuracies of more than 95%. However in our research we show that, when academic content is composed in a manner how a human would utilize generation models for the task, then those generated texts are difficult to be detected. We conduct experiments in detection by leveraging tools such as GPT2 output detector, DetectGPT and ZeroGPT and classification models such as SciBERT, RoBERTa and DeBERTa.

Is it an Easy Task to Accurately Detect Automatically Generated Academic Content?

This paper examines the main challenges in Amharic hate speech classification tasks. The potential sources of errors in Amharic hate speech classification are explored in detail. Out of a total of more than 18 million tweets in our Twitter corpus, we annotate over 15k tweets with two independent Amharic native-speaker annotators. A Cohen's kappa score of 48% for inter-annotator agreement is achieved. An adjudicator is employed to decide on disputed annotations and determine the final gold labels. Logistic regression, Linear SVM, and the two fine-tuned contextual embedding transformer models: AmFLAIR and AmRoBERTa are employed. Among all the models, AmFLAIR achieves the best performance result with an F1-score of 72%.

Error Analysis in Amharic Hate Speech Detection Task

Evaluation after the LLM boom: frustrations, fallacies, and the future | VIDEO

One of the frequent points in the mainstream narrative about large language models is that they have “emergent properties” (sometimes even dangerous enough to be considered existential risk to mankind). However, there is much disagreement about even the very definition of such properties. If they are understood as a kind of generalization beyond training data - as something that a model does without being explicitly trained for it - I argue that we have not in fact established the existence of any such properties, and at the moment we do not even have the methodology for doing so.

Premium content

Understanding generalization for instruction following and black-box language models | VIDEO

Downloads

Next from EMNLP 2023

AmEn: Amharic-English Large Parallel Corpus for Machine Translation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES