United Arab Emirates

This paper presents an Amharic hate speech annotation using the Yandex Toloka crowdsourcing platform. The dataset is collected from 5 consecutive years 2018-2022, following some controversial events in Ethiopia that put the country in violence. We consider one-month data starting from the event date in each year. Accordingly, we annotate 5,400 tweets, nearly 1k tweets for each year. We explore the main challenges of crowdsourcing annotation for Amharic hate speech data collection using Toloka. We attain a Fleiss kappa score of 0.34 using three independent annotators that annotate the tweets where the gold label is determined using majority voting. Using the datasets, we build classification models using deep learning (LSTM and BiLSTM) and achieved a 0.49 F1-score for both models.  We will publicly release the dataset, source code, and models with a permissive license.

EMNLP 2022

Challenges of Amharic Hate Speech Data Annotation Using Yandex Toloka Crowdsourcing Platform

yandex toloka

hate speech

crowdsourcing

annotation

workshop paper

## Welcome to EMNLP 2022!
I am delighted to welcome you to EMNLP 2022! I believe this conference has been complicated beyond any precedent. Over the past year, it’s been thrilling to see the organization team approach each new puzzle with creativity and enthusiasm. We hope that those participating in Abu Dhabi as well as those joining remotely will leave the conference feeling newly inspired by the program and newly connected to our ever-growing community. Following EMNLP 2021 and major NLP conferences since, EMNLP 2022 is “hybrid,” serving both virtual and in-person participants.

Our key innovations for EMNLP 2022 include:

* EMNLP 2022 is “hybrid” in a second sense, as well: we allowed both direct and rolling review paper submissions, building on the pilot experiment of EMNLP 2021, which considered a small number of ARR submissions. 
* Familiar from NAACL but new to EMNLP, we’ve added an industry track.
* During the conference, “portals” will link virtual poster sessions to in-person conference participants during poster sessions each day.
* The first ACL-family conference in the United Arab Emirates.

 *Message from Noah A. Smith, University of Washington and Allen Institute for AI, Seattle, Washington, USA* 
***EMNLP 2022 General Chair***
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/9eec7d4a287ee18c278b08229290aa83.png)](https://drive.google.com/file/d/1OlPv6QBeo62VVTughj2jkiLeyHd1WnUt/view)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/a3db7a768409f05192210d98601edb25.png)](https://emnlp2022.rocket.chat/)

To access this site you need to register. Please register [here](https://2022.emnlp.org/registration/).

Register here

Welcome!
EMNLP 2022 will take place in Abu Dhabi from December 7th to December 11th, 2022. And it will be held in hybrid mode, both online and offline.

In this work, we build a Question classification (QC) dataset from a social media platform, namely the Telegram public channel called @AskAnythingEthiopia. The platform allows asking questions that belong to various domains, like Politics, Music, Technology, Religion and so on. Questions are posted in Amharic, English, or Amharic in Latin script. Since the questions are posed in a mixed-code, we apply different strategies to pre-process the dataset. As part of the pre-processing tools, we build a Latin-to-Ethiopic-Script transliteration tool. We collect 8k Amharic and 24K Amharic but written in Latin script questions and develop deep learning-based question answering classifiers that attain an F-score of 57.79 in 20 different question categories. The datasets and pre-processing scripts are open-sourced to facilitate further research on the Amharic community-based question answering.

Question Answering Classification for Amharic Social Media Community Based Questions

In this ongoing work, we investigate a potential bias to associate terms related to ``female'' genitalia to reproduction, and terms related to ``male'' genitalia to sexual pleasure. While some of our suspicions are confirmed (stronger association of "female" genitalia to reproduction than to sexual pleasure), others are not (for example, our current data does not suggest a stronger association of reproduction to ``female'' genitalia than to ``male'' genitalia). We plan to continue this line of work by including more languages to get a better overview of the situation in languages other than English. Furthermore, we intend to critically assess cosine similarity as a proxy for association, as well as looking into (statistical) significance of the differences in cosine similarity that we found, and the effects of the skewed frequency distributions of the terms related to ``female'' and ``male'' genitals, and reproduction and sexual pleasure.

On Genitalia, Reproduction and Pleasure: Biases in the Representation of Sexes

Research writing can be technical and diffi- cult to understand. Manual assignments of re- search areas and related faculties can be time- consuming and error-prone. Therefore, the first goal of this project is to build an unsu- pervised model to classify an unseen project proposal. We explored unsupervised learning methods such as K-means and topic models as well as a combination of Latent Dirichlet Annotation (LDA), Bidirectional Encoder Rep- resentations from Transformers (BERT) and K-means to cluster project proposals into dif- ferent categories. The second goal is to recom- mend papers for particular project proposals based on other similar publications. We can as- sume that the authors of the closest papers can be suitable supervisors for the research project. We investigated different features that can be used as numerical vector representation of doc- uments and apply cosine similarity method to find matching pairs of paper and proposal. We found out that TF-IDF gives the most accurate results.

An Unsupervised Learning Approach for Categorising Research Proposals and Recommending Papers

The aim of this paper is present and regroup 70 research works that have been done for detecting Adverse Drug Events from social media. For each work, we focus on its description, its aim, its approach, the used models, the used datasets, its novelty and its limitation. We also add in the appendix part a table summarising all the studied works.

Detecting Adverse Drug Events from social media: A brief literature review

The gender bias present in the data on which language models are pre-trained gets reflected in the systems that use these models. The model's intrinsic gender bias shows an outdated and unequal view of women in our culture and encourages discrimination. Therefore, in order to establish more equitable systems and increase fairness, it is crucial to identify and mitigate the bias existing in these models. While there is a significant amount of work in this area in English, there is a dearth of research being done in other gendered and low resources languages, particularly the Indian languages. English is a non-gendered language, where it has genderless nouns. The methodologies for bias detection in English cannot be directly deployed in other gendered languages, where the syntax and semantics vary. In our paper, we measure gender bias associated with occupations in Hindi language models. Our major contributions in this paper are the construction of a novel corpus to evaluate occupational gender bias in Hindi and quantify this existing bias in these systems using a well-defined metric.

Evaluating Gender Bias in Pre-trained Indic Language Models

Despite recent advances in vision-and-language tasks, most progress is still focused on resource-rich languages such as English. Furthermore, widespread vision-and-language datasets directly adopt images representative of American or European cultures resulting in bias. Hence we introduce ParsVQA-Caps, the first benchmark in Persian for Visual Question Answering and Image Captioning tasks. We utilise two ways to collect datasets for each task, human-based and template-based for VQA and human-based and web-based for image captioning.

ParsVQA-Caps: A Benchmark for Visual Question Answering and Image Captioning in Persian

Social media platforms allow users to freely share their sentiments and opinions about is- sues and events or anything they feel like; however, they also make it easier to spread hate and abusive content. This paper introduces the HERDPhobia dataset, a manually annotated Twitter hate speech dataset against Fulani herders in Nigeria. The paper describes the data collection method, the annotation process, and the baseline experiment. We present a benchmark experiment using pre-trained languages models to classify the tweets as either hate speech or not hate speech, and the results were promising. We will release the dataset for further research in this direction.

HERDPhobia: A Dataset for Hate Speech Detection against Fulani Herdsmen in Nigeria

Topic modeling is a widely used analytical tool for evaluating large collections of textual data. One of the most popular topic modeling techniques is Latent Dirichlet Allocation (LDA). Although flexible and adaptive, it is not optimal for less conventional texts such as short texts from various domains. In this paper, we explore how the state-of-the-art BERTopic algorithm performs on short multi-domain text. We find that BERTopic HDBSCAN generalizes better than LDA in terms of topic coherence (0.091) and diversity (0.880). We further analyze the performance of the HDBSCAN clustering algorithm that BERTopic utilizes and find that suffers from predicting 74\% of the documents as outliers. We argue that this is a crucial, and yet overseen problem since it excludes the majority of the documents from the prediction. We propose to replace HDBSCAN with k-Means and show that similar performance can be achieved while controlling the number of outlier documents.

Experiments on Generalizability of BERTopic on Short Text

Visual Question Answering is a multi-modal task under the consideration of both the Vision and Language communities. Present VQA models are limited to classification answers and cannot provide answers for reasoning questions. In this work, we introduce an encoder-decoder model using vision-and-language pretrained embedding, which delivers multi-word generated sentences as answers. We utilise LXMERT and VisualBERT embedding space with three different generative decoder heads, including RNNs, Attention RNNs and Transformers. Extensive experiments show competitive performance on the FSVQA dataset through qualitative and quantitative evaluation and a Human Error Analysis.

Generate Answer to Visual Questions with Pre-trained Vision-and-Language Embeddings

This paper presents a new multi-modal dataset for identifying hateful content on social media, consisting of 5,680 text-image pairs collected from Twitter, labeled across two labels. Experimental analysis of the presented dataset has shown that understanding both modalities is essential for detecting these techniques. It is confirmed in our experiments with several state-of-the-art multi-modal models. In future work, we plan to extend the dataset in size. We further plan to develop new multi-modal models tailored explicitly to hate-speech detection, aiming for a deeper understanding of the text and image relation. It would also be interesting to perform experiments in a direction that explores what social entities the given hate speech tweet targets.

Challenges of Amharic Hate Speech Data Annotation Using Yandex Toloka Crowdsourcing Platform

Downloads

Next from EMNLP 2022

Question Answering Classification for Amharic Social Media Community Based Questions

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES