United States

We argue that we need to evaluate model interpretability methods {\em in the wild}, in situations where professionals make critical decisions, and models can potentially assist them. We present an {\bf in-the-wild evaluation} of token attribution based on Deep Taylor Decomposition \cite{chefer2021transformer}, {\bf with professional journalists} performing reliability assessments. We find that using this method in conjunction with RoBERTa-Large, fine-tuned on the Gossip Corpus, led to {\bf faster and better human decision-making}, as well as a more critical attitude toward news sources among the journalists. We present a comparison of human and model rationales, as well as a qualitative analysis of the journalists&#39; experiences with machine-in-the-loop decision making.

ICWSM 2022

Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild

in-the-wild evaluation

deep taylor decomposition

interpretability

[![](https://assets.underline.io/uploads/markdown_image/1/image/81a3c0317d24f663b49b996875024d45.png)](https://aclanthology.org/volumes/2022.coling-1/)



### THE CONFERENCE WE KNOW AND WE WANT.
**COLING**, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes but also in industrial research departments including tech-startups. COLING provides opportunities for all these communities to showcase their exciting discovery.

In fall of 2022, COLING will be held in Gyeongju in a hybrid format. All participants can either present at the venue site or join virtually. As more people get vaccinated, we are happy to provide safer environments for our colleagues. We believe that COLING 2022 will be one of the conferences, free from the pandemic. The hybrid format gives presenters and sponsors a valuable opportunity to promote their companies in both an online and in-person venue. For the first time in a long time, customers can interact with their sponsor's products first-hand. The online venue, too, gives sponsors the chance to network with those unable to attend the in-person session.

The hybrid format ultimately lends itself to greater exposure for our sponsors: COLING2022 will let you reach more potential partners and customers than ever before!

To gain access to this event page you are required to register. Find more information and pay the registration fee on event the organizer’s website **[https://coling2022.org/reg](https://coling2022.org/reg)**

COLING 2022

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

We argue that we need to evaluate model interpretability methods {\em in the wild}, in situations where professionals make critical decisions, and models can potentially assist them. We present an {\bf in-the-wild evaluation} of token attribution based on Deep Taylor Decomposition \cite{chefer2021transformer}, {\bf with professional journalists} performing reliability assessments. We find that using this method in conjunction with RoBERTa-Large, fine-tuned on the Gossip Corpus, led to {\bf faster and better human decision-making}, as well as a more critical attitude toward news sources among the journalists. We present a comparison of human and model rationales, as well as a qualitative analysis of the journalists' experiences with machine-in-the-loop decision making.

poster

**ICWSM-2022 will be held from June 6th – 9th in a hybrid format, both in-person in Atlanta, Georgia and online.**

**The Workshop and Tutorial Sessions are accessible to those that registered separately for these sessions. If you registered to attend but are not able to access the these sessions from the [Schedule tab](https://underline.io/events/291/schedule?day=2022-06-06T04%3A00%3A00.000Z), please email icwsm@underline.io
**

The International AAAI Conference on Web and Social Media (ICWSM) is a forum for researchers from multiple disciplines to come together to share knowledge, discuss ideas, exchange information, and learn about cutting-edge research in diverse fields with the common theme of online social media. This overall theme includes research in new perspectives in social theories, as well as computational algorithms for analyzing social media. ICWSM is a singularly fitting venue for research that blends social science and computational approaches to answer important and challenging questions about human social behavior through social media while advancing computational tools for vast and unstructured data.


ICWSM, now in its sixteenth year, has become one of the premier venues for computational social science, and previous years of ICWSM have featured papers, posters, and demos that draw upon network science, machine learning, computational linguistics, sociology, communication, and political science. The uniqueness of the venue and the quality of submissions have contributed to a rapidly growing conference, and a competitive acceptance rate of approximately 20% for full-length research papers published in the proceedings by the Association for the Advancement of Artificial Intelligence (AAAI).


**ICWSM Code of Conduct**

All persons, organizations and entities that attend AAAI conferences and events are subject to the standards of conduct set forth on the [AAAI Code of Conduct for Events and Conferences](https://aaai.org/Conferences/code-of-conduct.php). AAAI expects all community members to formally endorse this code of conduct, and to actively prevent and discourage any undesired behaviors. Everyone should feel empowered to politely engage when they or others are disrespected, and to raise awareness and understanding of this code of conduct. AAAI event participants asked to stop their unacceptable behavior are expected to comply immediately. Sponsors are also subject to this code of conduct in their participation in AAAI events.


Additionally, participants are encouraged to be courteous when sharing screen captures and photographs of conference events. Seek permission when possible and respect requests to take down images if those featured ask. Concerns around code of conduct or inclusion may be sent to icwsm22@aaai.org. If you have any concerns or items to report, please reach out to the General Chairs: Diyi Yang and Yelena Mejova.

A ticket is required to attend the event, please use the below URL to register for the event if you have not already done so:

https://aaaiconf.cventevents.com/event/be8c26b0-a76f-4753-babb-de2bd9d2a619/summary

Virtual-Only:
$200 AAAI Member
$125 AAAI Student Member
$225 Nonmember
$150 Nonmember Student

Virtual Silver: Includes discounted conference registration, plus a one-year online new or renewed membership in AAAI.
$299 Silver Regular
$174 Silver Student

Virtual - Workshop and Tutorials:
$45 Regular
$25 Student


Registration Is Required

The International AAAI Conference on Web and Social Media (ICWSM) is a forum for researchers from multiple disciplines to come together to share knowledge, discuss ideas, exchange information, and learn about cutting-edge research in diverse fields with the common theme of online social media. This overall theme includes research in new perspectives in social theories, as well as computational algorithms for analyzing social media. ICWSM is a singularly fitting venue for research that blends social science and computational approaches to answer important and challenging questions about human social behavior through social media while advancing computational tools for vast and unstructured data.

Despite the increasing interest in cyberbullying detection, existing efforts have largely been limited to experiments on a single platform and their generalisability across different social media platforms has received less attention. We propose XP-CB, a novel cross-platform framework based on Transformers and adversarial learning. XP-CB can enhance a Transformer leveraging unlabelled data from the source and target platforms to come up with a common representation while preventing platform-specific training. To validate our proposed framework, we experiment on cyberbullying datasets from three different platforms through six cross-platform configurations, showing its effectiveness with both BERT and RoBERTa as the underlying Transformer models.

Cyberbullying detection across social media platforms via platform-aware adversarial encoding

Because of their stigmatized social status, sexual and gender minority (SGM; e.g., gay, transgender) people experience minority stress (i.e., identity-based stress arising from adverse social conditions). Given that minority stress is the leading framework for understanding health inequity among SGM people, researchers and clinicians need accurate methods to detect minority stress. Since social media fulfills important developmental, affiliative, and coping functions for SGM people, social media may be an ecologically valid channel for detecting minority stress. In this paper, we propose a bidirectional long short-term memory (BI-LSTM) network for classifying minority stress dis-closed on Reddit. Our experiments on a dataset of 12,645 Reddit posts resulted in an average accuracy of 65%.

Classifying Minority Stress Disclosure on Social Media with Bidirectional Long Short-Term Memory

While there have been many efforts to monitor or predict Covid using digital traces such as social media, one of the most distinctive and diagnostically important symptoms of Covid -- anosmia, or loss of smell -- remains elusive due to the infrequency of discussions of smell online. It was recently hypothesized that an inadvertent indicator of this key symptom may be misplaced complaints in Amazon reviews that scented products such as candles have no smell. We develop a novel Bayesian vector autoregression model to test this hypothesis, finding that "no smell" reviews do indeed reflect changes in US Covid cases even when controlling for the seasonality of those reviews. A series of robustness checks suggests that this effect is also seen in perfume reviews, and did not hold for the flu prior to Covid. These results suggests that hypothesis-driven models of inadvertent traces may be useful for tracking epidemics using digital records.

This Candle Has No Smell: Detecting the effect of Covid anosmia on Amazon reviews using Bayesian Vector Autoregression

During global health crises, the use of data becomes critical to control the spread of infections, to inform the general public and to foster safe behaviors. The ability of people to read and understand data (i.e., data literacy) has the potential to affect human behaviors. In this paper, we study the ability of non-expert human subjects making accurate interpretations of complex pandemic data visualisations designed for general public consumption. We present them with popular plots and graphs that have been shown by traditional and social media, and ask them to answer questions to assess their data literacy at three levels: extracting information, finding relationships among data, and expanding or predicting information. Our results show the presence of variance in interpretations, and reveal insights into how messages communicated through data may be perceived differently by different people. We also highlight the importance of designing communication strategies that ensure the spread of the right message through data.

Exploring Data Literacy Levels in the Crowd -- The Case of COVID-19

While the contagious nature of online toxicity sparked increasing interest in early detection and prevention of it, most of the previous studies focus on the Western world. In this work, we study toxicity triggers in an online community for discussing issues regarding Singapore and conduct a comparative study of toxicity triggers between Singapore and New York City.

Understanding Toxicity Triggers on Reddit in the Context of Singapore

Bias in news reporting can lead to tribalism and division on important issues. Scalable and reliable measurement of such biases is an important first step in addressing them. In this work, based on the intuition that media bias is captured by the tone and word choices in articles, we propose a framework for modeling the relative bias of media outlets through masked token prediction via large-scale pretrained masked language models fine-tuned on articles form news outlets. Through experiments on five diverse and politically polarized topics we show that our framework can capture media bias towards these topics with high reliability. Additionally, our experiments show that our framework is general, in that language models fine-tuned on one topic can be applied to other topics with little drop in performance.

Measuring Media Bias via Masked Language Modeling

The word embedding association test (WEAT) is an important method for measuring linguistic biases against social groups such as ethnic minorities in large text corpora. It does so by comparing the semantic relatedness of words prototypical of the groups (e.g., names unique to those groups) and attribute words (e.g., ‘pleasant’ and ‘unpleasant’ words). We show that anti-black WEAT estimates from geo-tagged social media data at the level of metropolitan statistical areas strongly correlate with several measures of racial animus—even when controlling for sociodemographic covariates. However, we also show that every one of these correlations is explained by a third variable: the frequency of Black names in the underlying corpora relative to White names. This occurs because word embeddings tend to group positive (negative) words and frequent (rare) words together in the estimated semantic space. As the frequency of Black names on social media is strongly correlated with Black Americans’ prevalence in the population, this results in spurious anti-Black WEAT estimates wherever few Black Americans live. This suggests that research using the WEAT to measure bias should consider term frequency, and also demonstrates the potential consequences of using black-box models like word embeddings to study human cognition and behavior.

Negative Associations in Word Embeddings Predict anti-Black Bias Across Regions--but only via Name Frequency

On Twitter, so-called verified accounts represent celebrities and organizations of public interest, selected by Twitter based on criteria for both activity and notability. Our work seeks to understand the involvement and influence of these accounts in patterns of self-disclosure, namely, voluntary sharing of personal information. In a study of 3 million COVID-19 related tweets, we present a comparison of self-disclosure in verified vs ordinary users. We discuss evidence of peer effects on self-disclosing behaviors and analyze topics of conversation associated with these practices.

The contribution of verified accounts to self-disclosure in COVID-related Twitter conversations

Hate speech has proliferated on social media platforms in recent years. While this has been the focus of many studies, most works have exclusively focused on a single language, generally English. Low-resourced languages have been neglected due to the dearth of labeled resources. These languages, however represent an important portion of the data due to the multilingual nature of social media. This work presents a novel zero-shot, cross-lingual transfer learning pipeline based on pseudo-label fine-tuning of Transformer Language Models for automatic hate speech detection. We employ our pipeline on benchmark datasets covering English (source) and 6 different non-English (target) languages written in 3 different scripts. Our pipeline achieves an average improvement of 7.6% (in terms of macro-F1) over previous zero-shot, cross-lingual models. This demonstrates the feasibility of high accuracy automatic hate speech detection for low-resource languages. We release our code and models at https://github.com/<redacted>.

Improving Zero-shot Cross-lingual Hate Speech Detection with pseudo-label fine-tuning of Transformer Language Models

A conversation corpus is essential to build interactive AI applications. 
However, participants' demographic information is relatively understudied mainly due to a lack of individual data. 
In this work, we analyzed a  Korean nationwide daily conversation corpus constructed by the National Institute of Korean Language (NIKL) to characterize the participation of different demographic groups in the corpus.

Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild

Downloads

Next from ICWSM 2022

Cyberbullying detection across social media platforms via platform-aware adversarial encoding

Similar lecture

Annotating targets of toxic language at the span level

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES