Thailand

Online conversations, just like offline ones, are susceptible to influence by bad actors. These users have the capacity to derail neutral or even prosocial discussions through adverse behavior. Moderators and users alike would benefit from more resilient online conversations, i.e., those that can survive the influx of antisocial behavior to which many conversations fall victim. In this paper, we examine the notion of conversational resilience: what makes a conversation more or less capable of withstanding an antisocial interruption? Understanding the factors that contribute to resilience in online conversations can help identify what behaviors we should encourage and better inform moderation practices. Working with 10.5M comments from five mainstream subreddits (over 2.9M subscribers each), we compiled more than 5.8M comment threads (i.e., conversations). Using 20K relevant conversations, we investigate the effects of different norm violations on the outcome of a conversation while accounting for community characteristics and the context in which the violation occurs. We find that norm violations that occur at the start of a conversation do not necessarily stop the flow of subsequent comments. Additionally, highly-toxic violations are more likely to occur in comment threads containing other toxic comments, both before and after the violation itself. Finally, more severe violations may not necessarily lead to higher rates of subsequent violations, while the comment author’s history of toxic behavior might.

IJCNLP-AACL 2021

Conversation-Level Resilience to Bad Actors in Reddit Communities

online conversations

human-centered computing

prosocial behavior

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

workshop paper

Leaderboards are widely used in NLP and push the field forward. While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models). Rather than replace leaderboards, we advocate a re-imagining so that they better highlight if and where progress is made. Building on educational testing, we create a Bayesian leaderboard model where latent subject skill and latent item difficulty predict correct responses. Using this model, we analyze the ranking reliability of leaderboards. Afterwards, we show the model can guide what to annotate, identify annotation errors, detect overfitting, and identify informative examples. We conclude with recommendations for future benchmark tasks.

Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?

Manual fact-checking does not scale well to serve the needs of the internet. This issue is further compounded in non-English contexts. In this paper, we discuss claim matching as a possible solution to scale fact-checking. We define claim matching as the task of identifying pairs of textual messages containing claims that can be served with one fact-check. We construct a novel dataset of WhatsApp tipline and public group messages alongside fact-checked claims that are first annotated for containing "claim-like statements" and then matched with potentially similar items and annotated for claim matching. Our dataset contains content in high-resource (English, Hindi) and lower-resource (Bengali, Malayalam, Tamil) languages. We train our own embedding model using knowledge distillation and a high-quality "teacher" model in order to address the imbalance in embedding quality between the low- and high-resource languages in our dataset. We provide evaluations on the performance of our solution and compare with baselines and existing state-of-the-art multilingual embedding models, namely LASER and LaBSE. We demonstrate that our performance exceeds LASER and LaBSE in all settings. We release our annotated datasets, codebooks, and trained embedding model to allow for further research.

Claim Matching Beyond English to Scale Global Fact-Checking

In this work, we introduce \datasetName: the largest publicly available multilingual dataset for factual verification of naturally existing real-world claims. The dataset contains short statements in 25 languages and is labeled for veracity by expert fact-checkers. The dataset includes a multilingual evaluation benchmark that measures both out-of-domain generalization, and zero-shot capabilities of the multilingual models. Using state-of-the-art multilingual transformer-based models, we develop several automated fact-checking models that, along with textual claims, make use of additional metadata and evidence from news stories retrieved using a search engine. Empirically, our best model attains an F-score of around 40%, suggesting that our dataset is a challenging benchmark for the evaluation of multilingual fact-checking models.

X-Fact: A New Benchmark Dataset for Multilingual Fact Checking

While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively use it for neural machine translation (NMT) still remains a tricky issue. The main reason is that the cross-attention module between the encoder and decoder cannot be pre-trained, and the combined encoder-decoder model cannot work well in the fine-tuning stage because the inputs of the decoder cross-attention come from unknown encoder outputs. In this paper, we propose a better pre-training method for NMT by defining a semantic interface (SemFace) between the pre-trained encoder and the pre-trained decoder. Specifically, we propose two types of semantic interfaces, including CL-SemFace which regards cross-lingual embeddings as an interface, and VQ-SemFace which employs vector quantized embeddings to constrain the encoder outputs and decoder inputs in the same language-independent space. We conduct massive experiments on six supervised translation pairs and three unsupervised pairs. Experimental results demonstrate that our proposed SemFace can effectively connect the pre-trained encoder and decoder, and achieves significant improvement by 3.7 and 1.5 BLEU points on the two tasks respectively compared with previous pre-training-based NMT models.

SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation

We introduce Explainable Scientific Research Assistant (ESRA), a literature discovery platform that augments search results with relevant details and explanations, aiding users in understanding more about their queries and the returned papers beyond existing literature search systems. Enabled by a knowledge graph we extracted from abstracts of 23k papers on the arXiv's cs.CL category, ESRA provides three main features: explanation (for why a paper is returned to the user), list of facts (that are relevant to the query), and graph visualization (drawing connections between the query and each paper with surrounding related entities). The experimental results with humans involved show that ESRA can accelerate the users' search process with paper explanations and helps them better explore the landscape of the topics of interest by exploiting the underlying knowledge graph. We provide the ESRA web application at http://esra.cp.eng.chula.ac.th/.

{ESRA}: Explainable Scientific Research Assistant

This paper presents REM, a novel tool for the semi-automated real-time moderation of large scale online forums. The growing demand for online participation and the increasing number of user comments  raise challenges in filtering out harmful and undesirable content from public debates in online forums. Since a manual moderation does not scale well and pure automated approaches often lack the required level of accuracy, we suggest a semi-automated moderation approach. Our approach maximizes the efficiency of manual efforts by targeting only those comments for which human intervention is needed, e.g. due to high classification uncertainty.  Our tool offers a rich visual interactive environment enabling the exploration of online debates.  We conduct a preliminary evaluation experiment to demonstrate the suitability of our approach and publicly release the source code of REM.

REM: Efficient Semi-Automated Real-Time Moderation of Online Forums

An essential operation in web corpus construction consists in retaining the desired content while discarding the rest. Another challenge finding one's way through websites. This article introduces a text discovery and extraction tool published under open-source license. Its installation and use is straightforward, notably from Python and on the command-line. The software allows for main text, comments and metadata extraction, while also providing building blocks for web crawling tasks. A comparative evaluation on real-world data also shows its interest as well as the performance of other available solutions.

The contributions of this paper are threefold: it references the software, features a benchmark, and provides a meaningful baseline for similar tasks. The tool performs significantly better than other open-source solutions in this evaluation and in external benchmarks.

Trafilatura: {A} Web Scraping Library and Command-Line Tool for Text Discovery and Extraction

This paper examines relationships between online self-disclosure and social support during the COVID-19 crisis. A total of 2,399 posts and 30,073 comments are crawled from the r/COVID19_support subreddit. We manually extract fine-grained personal information categories and types of requested social support from a post. We further develop a novel AI-driven classifier to automatically identify types of social support offered in a comment. Our findings show that users tend to share their health and occupation related information most frequently. Variability in the types of information disclosed amongst informational and emotional support-seeking content is explored. Our analyses indicate that: 1) users are more likely to share their age, education, and location information when seeking both informational and emotional support, as opposed to pursuing either one; 2) while personal information sharing increases the probability of users receiving informational support when requested, it has no correlation with emotional support.

Effects of Online Self-Disclosure on Receiving Social Support During the COVID-19 Pandemic

Stereotypes are inferences drawn about people based on their demographic attributes, which may result in harms to users when a system is deployed.
In generative language-inference tasks, given a premise, a model produces plausible hypotheses that follow either logically (natural language inference) or common-sensically (commonsense inference).
Such tasks are therefore a fruitful setting in which to explore the degree to which NLP systems encode stereotypes.
In our work, we study how stereotypes manifest when the potential targets of stereotypes are situated in real-life, neutral contexts.
We collect human judgments on the presence of stereotypes in generated inferences and compare how perceptions of stereotypes vary due to annotator positionality.

Analyzing Stereotypes in Generative Text Inference Tasks

Understanding the gaps between job requirements and university curricula is crucial for improving student success and institutional effectiveness in higher education. In this context, natural language processing (NLP) can be leveraged to generate granular insights into where the gaps are and how they change. This paper proposes a three-dimensional research framework that combines NLP techniques with economic and educational research to quantify the alignment between course syllabi and job postings. We elaborate on key technical details of the framework and further discuss its potential positive impacts on practice, including unveiling the inequalities in and long-term consequences of education-occupation alignment to inform policymakers, and fostering information systems to support students, institutions and employers in the school-to-work pipeline.

Conversation-Level Resilience to Bad Actors in Reddit Communities

Next from IJCNLP-AACL 2021

Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?

Similar lecture

Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES