Korea, Republic of

This paper describes the models developed by the AILAB-Udine team for the SMM4H’22 Shared Task. We explored the limits of Transformer based models on text classification, entity extraction and entity normalization, tackling
Tasks 1, 2, 5, 6 and 10. The main takeaways we got from participating in different
tasks are: the overwhelming positive effects of combining different architectures when using ensemble learning, and the great potential of generative models for term normalization.

COLING 2022

Limits of Transformers and BERT Ensembles

ensemble models

text generation

transformers

[![](https://assets.underline.io/uploads/markdown_image/1/image/81a3c0317d24f663b49b996875024d45.png)](https://aclanthology.org/volumes/2022.coling-1/)



### THE CONFERENCE WE KNOW AND WE WANT.
**COLING**, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes but also in industrial research departments including tech-startups. COLING provides opportunities for all these communities to showcase their exciting discovery.

In fall of 2022, COLING will be held in Gyeongju in a hybrid format. All participants can either present at the venue site or join virtually. As more people get vaccinated, we are happy to provide safer environments for our colleagues. We believe that COLING 2022 will be one of the conferences, free from the pandemic. The hybrid format gives presenters and sponsors a valuable opportunity to promote their companies in both an online and in-person venue. For the first time in a long time, customers can interact with their sponsor's products first-hand. The online venue, too, gives sponsors the chance to network with those unable to attend the in-person session.

The hybrid format ultimately lends itself to greater exposure for our sponsors: COLING2022 will let you reach more potential partners and customers than ever before!<br><br><br>

To gain access to this event page you are required to register.<br> Find more information and pay the registration fee on event the organizer’s website **[https://coling2022.org/reg](https://coling2022.org/reg)**

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

workshop paper

This paper presents our submission for the SMM4H 2022-Shared Task on the classification of self-reported intimate partner violence on Twitter (in English). The goal of this task was to accurately determine if the contents of a given tweet demonstrated someone reporting their own experience with intimate partner violence. The submitted system is an ensemble of five RoBERTa models each weighted by their respective F1-scores on the validation data-set. This system performed 13% better than the baseline and was the best performing system overall for this shared task.

Classifying Self-Reported Intimate Partner Violence in Tweets with Multiple BERT-based Models

With the increase in the use of social media, people have become more outspoken and are using platforms like Reddit, Facebook, and Twitter to express their views and share the medical challenges they are facing. This data is a valuable source of medical insight and is often used for healthcare research. This paper describes our participation in Task 1a, 2a, 2b, 3, 5, 6, 7, and 9 organized by SMM4H 2022. We have proposed two transformer-based approaches to handle the classification tasks. The first approach is fine-tuning single language models. The second approach is ensembling the results of BERT, RoBERTa, and ERNIE 2.0.

Classification of HealthRelated Tweets Using Ensemble, Zero-Shot andFine-Tuned Language Model

This paper presents our approaches to SMM4H’22 task 5 - Classification of tweets of self-reported COVID-19 symptoms in Spanish, and task 10 - Detection of disease mentions in tweets – SocialDisNER (in Spanish). We have presented hybrid systems that combine Deep Learning techniques with linguistic rules and medical ontologies, which have allowed us to achieve outstanding results in both tasks.

Tasks 5 \& 10 - Hybrid models everywhere

This paper reports on the performance of Edinburgh_UCL_Health's models in the Social Media Mining for Health (SMM4H) 2022 shared tasks. Our team participated in the tasks related to the Identification of Adverse Drug Events (ADEs), the 
classification of change in medication (change-med) and the classification of self-report of vaccination (self-vaccine). Our best performing models are based on DeepADEMiner (with respective F1= 0.64, 0.62 and 0.39 for ADE identification), on a GloVe model trained on Twitter (with F1=0.11 for the change-med) and finally on a stack embedding including a layer of Glove embedding and two layers of Flair 
embedding (with F1= 0.77 for self-report).

From Glove to Flair for handling imbalanced healthcare corpora related to Adverse Drug Events, Change in medication and self-reporting vaccination

This paper describes models developed for the Social Media Mining for Health 2022 shared tasks. Our team participated in the first sub-task that classifies tweets with Adverse Drug Effect mentions. Our best-performing model comprises of a template augmented task adaptive pre-training and further fine-tuning on target task data. Augmentation with random prompt templates increases the amount of task-specific data to generalize the pretrained language model to the target task domain. We explore 2 pre-training strategies: masked language modeling and simple contrastive pre-training and the impact of adding template augmentations with these pre-training strategies. Our system achieves an F1 score of 0.433 on the test set without using supplementary resources and medical dictionaries.

Template Augmented Adaptive Pre-training for Tweet Classification

This poster presentation covers our approaches for the Social Media Mining for Health (SMM4H) Shared Tasks 2a and 2b. Apart from the baseline architectures, we experiment with Parts of Speech (PoS), dependency parsing, and Tf-Idf features. Additionally, we perform contrastive pretraining on our best models using a supervised contrastive loss function. In both the tasks, we outperformed the mean and median scores and ranked first on the validation set. For stance classification, we achieved an F1-score of 0.636 using the CovidTwitterBERT model, while for premise classification, we achieved an F1-score of 0.664 using BART-base model on test dataset.

Leveraging Pre-trained Language Models for Stance And Premise Classification

We present our response to Task 5 of the Social Media Mining for Health Applications (SMM4H) 2022 competition. Our goal was to classify whether a tweet in Spanish regarding COVID-19 symptoms pertained to the author of the tweet, others, or presented news regarding COVID-19. Using ensembles of BERT-based models, we were able to achieve results that were higher than the median result of the competition. 


Identifying Self-Reported Spanish COVID-19 Symptom Tweets Through Multiple-Model Ensembling

This paper presents my submission for Tasks 1 and 2 for the Social Media Mining for Health (SMM4H) 2022 Shared Tasks competition. I first describe the background behind each of these tasks, followed by the descriptions of the various subtasks of Tasks 1 and 2, then present the methodology. Through model ensembling, this methodology was able to achieve higher results than the mean and median of the competition for classification tasks.

Too Cool: Using Multiple BERT Models Together for Various COVID-19 Tweet Identification Tasks

Social media platforms such as Twitter and Reddit are full of various statements made by users, and there is abundant social media data that can be used for mining health-related information. To fulfill the 10 text classification tasks of Task 1a, 2a, 2b, 3a, 4, 5, 6, 7, 8, and 9, in the Social Media Mining for Health (SMM4H) 2022, we propose a generic framework based on pre-trained BERT-based models. For fine-tuning, various techniques are applied, including regularized dropout, focal loss, exponential moving average, 5-fold cross-validation, ensemble prediction, and pseudo-labeling. In the evaluation, the proposed framework achieves the 1st place in Task 3a.

A Generic Pre-trained BERT-based Framework for Social Media Health Text Classification

We propose adversarial methods for increasing the robustness of disease mention detection on social media. Our method applies adversarial data augmentation on the input and the embedding spaces to the English BioBERT model. We evaluate our method in the SocialDisNER challenge at SMM4H'22 on an annotated dataset of disease mentions in Spanish tweets. We find that both methods outperform a heuristic vocabulary-based baseline by a large margin. Additionally, utilizing the English BioBERT model shows a strong performance and outperforms the data augmentation methods even when applied to the Spanish dataset, which has a large amount of data, while augmentation methods show a significant advantage in a low-data setting.

Premium content

Limits of Transformers and BERT Ensembles

Next from COLING 2022

Classifying Self-Reported Intimate Partner Violence in Tweets with Multiple BERT-based Models

Similar lecture

Leveraging Pre-trained Language Models for Stance And Premise Classification

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES