Canada

ACL 2023

Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree

subjectivity

disagreement

active learning

workshop paper

### Welcome to ACL 2023, the 61st Annual Meeting of the Association for Computational Linguistics! 
 The conference will be held in Toronto, Canada, July 9-14, 2023. 
Following the succession of the recent conferences in our field, ACL 2023 will adopt a hybrid format.
While the impact of Covid has considerably diminished in terms of traveling, obtaining visas to Canada
entails a very long process. Moreover, the global economic conditions pose challenges for many individuals to travel to conferences. Recognizing these circumstances, we know many participants may not be
able to attend the conference in person. Therefore, we are committed to providing a great virtual platform
so everyone has the opportunity to interact with other participants and enjoy the conference. Based on the
current registered participants, approxiately 30% have chosen to attend the conference virtually. Whether
you join us in person or virtually, we sincerely hope everyone has a remarkable conference experience. 
This General Chair’s message is where I express my gratitude to the many individuals who have made
enormous contributions to the conference over the past year.

Read [**ACL 2023 General Chair's message**](https://docs.google.com/document/d/1WobYM7norbG4dI48s75HfJoD89qgX5a_F-6U8AteLSA/edit?usp=sharing/) in full.

##### **[Conference Handbook](https://2023.aclweb.org/downloads/acl2023-handbook.pdf)**

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing.

Research on abusive content detection on social media has primarily focused on explicit forms of hate speech (HS), that are often identifiable by recognizing hateful words and expressions. Messages containing linguistically subtle and implicit forms of hate speech still constitute an open challenge for automatic hate speech detection. In this paper, we propose a new framework for generating adversarial implicit HS short-text messages using Auto-regressive Language Models. Moreover, we propose a strategy to group the generated implicit messages by their complexity levels (EASY, MEDIUM, and HARD categories) characterizing how challenging these messages are for supervised classifiers. Finally, relying on (Dinan et al. 2019; Vidgen et al. 2021), we propose a "build it, break it, fix it", training scheme using HARD messages showing how iteratively retraining on HARD messages substantially leverages SOTA models' performances on implicit HS benchmarks.

Playing the Part of the Sharp Bully: Generating Adversarial Examples for Implicit Hate Speech Detection

Annotator disagreement is common whenever human judgment is needed for supervised learn- ing. It is conventional to assume that one label per item represents ground truth. However, this obscures minority opinions, if present. We re- gard “ground truth” as the distribution of all labels that a population of annotators could pro- duce, if asked (and of which we only have a small sample). We next introduce DisCo (Dis- tribution from Context), a simple neural model that learns to predict this distribution. The model takes annotator-item pairs, rather than items alone, as input, and performs inference by aggregating over all annotators. Despite its simplicity, our experiments show that, on six benchmark datasets, our model is competitive with, and frequently outperforms, other, more complex models that either do not model spe- cific annotators or were not designed for label distribution learning.

Disagreement Matters: Preserving Label Diversity by Jointly Modeling
Item and Annotator Label Distributions with DisCo

We introduce SexTok, a multi-modal dataset composed of 1000 TikTok videos labeled as sexually suggestive (from the annotator’s point of view), sex-educational
content, or neither. Our dataset contains video URLs, perceived gender expression,
and audio transcription. To validate its importance, we explore two transformer-based models for classifying the videos. Our preliminary results suggest that the task of distinguishing between these types of videos is learnable but challenging.
We invite further work on this task.

It’s not Sexually Suggestive; It’s Educative | Separating Sex Education from
Suggestive Content on TikTok videos

Debiasing methods that seek to mitigate the tendency of Language Models (LMs) to occasionally output toxic or inappropriate text have recently gained traction. In this paper, we propose a standardized protocol which distinguishes methods that yield not only desirable results, but are also consistent with their mechanisms and specifications. For example, we ask, given a debiasing method that is developed to reduce toxicity in LMs, if the definition of toxicity used by the debiasing method is reversed, would the debiasing results also be reversed? We used such considerations to devise three criteria for our new protocol: Specification Polarity, Specification Importance, and Domain Transferability. As a case study, we apply ourprotocol to a popular debiasing method, Self-Debiasing, and compare it to one we propose, called Instructive Debiasing, and demonstrate that consistency is as important an aspect to debiasing viability as is simply a desirable result. We show that our protocol provides essential insights into the generalizability and interpretability of debiasing methods that may otherwise go overlooked.

Debiasing should be Good and Bad

The State of Profanity Obfuscation in Natural Language Processing Scientific Publications

COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements

Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models

Large Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks, based on their internal knowledge stored in parameters during pre-training. However, such internalized knowledge might be insufficient and incorrect, which could lead LLMs to generate factually wrong answers. Furthermore, fine-tuning LLMs to update their knowledge is expensive. To this end, we propose to augment the knowledge directly in the input of LLMs. Specifically, we first retrieve the relevant facts to the input question from the knowledge graph based on semantic similarities between the question and its associated facts. After that, we prepend the retrieved facts to the input question in the form of the prompt, which is then forwarded to LLMs to generate the answer. Our framework, Knowledge-Augmented language model PromptING (KAPING), requires no model training, thus completely zero-shot. We validate the performance of our KAPING framework on the knowledge graph question answering task, that aims to answer the user's question based on facts over a knowledge graph, on which ours outperforms relevant zero-shot baselines by up to 48% in average, across multiple LLMs of various sizes.

Knowledge-Augmented Language Model Prompting with Knowledge Graphs for Zero-Shot Question Answering

Despite their impressive scale, knowledge
bases (KBs), such as Wikidata, still contain
significant gaps. Language models (LMs) have
been proposed as a source for filling these gaps.
However, prior works have focused on promi-
nent entities with rich coverage by LMs, ne-
glecting the crucial case of long-tail entities.
In this paper, we present a novel method for
LM-based-KB completion that is specifically
geared for facts about long-tail entities. The
method leverages two different LMs in two
stages: for candidate retrieval and for candidate
verification and disambiguation. To evaluate
our method and various baselines, we introduce
a novel dataset, called MALT, rooted in Wiki-
data. Our method outperforms all baselines in
F1, with major gains especially in recall.

Knowledge Base Completion for Long-Tail Entities

Entity standardization maps noisy mentions from free-form text to standard entities in a knowledge base. The unique challenge of this task relative to other entity-related tasks is the lack of surrounding context and numerous variations in the surface form of the mentions, especially when it comes to generalization across domains where labeled data is scarce. Previous research mostly focuses on developing models either heavily relying on context, or dedicated solely to a specific domain. In contrast, we pro- pose CoSiNES, a generic and adaptable framework with Contrastive Siamese Network for Entity Standardization that effectively adapts a pretrained language model to capture the syntax and semantics of the entities in a new domain.
We construct a new dataset in the technology domain, which contains 640 technical stack entities and 6,412 mentions collected from industrial content management systems. We demonstrate that CoSiNES yields higher accuracy and faster runtime than baselines derived from leading methods in this domain. CoSiNES also achieves competitive performance in four standard datasets from the chemistry, medicine, and biomedical domains, demonstrating its cross-domain applicability.


Premium content

Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree

Downloads

Next from ACL 2023

Playing the Part of the Sharp Bully: Generating Adversarial Examples for Implicit Hate Speech Detection

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES