Thailand

This paper presents the Chinchunmei team&#39;s contributions to the WASSA2024 Shared-Task 1: Empathy Detection and Emotion Classification. We participated in Tracks 1, 2, and 3 to predict empathetic scores based on dialogue, article, and essay content. We choose Llama3-8b-instruct as our base model. We developed three supervised fine-tuning schemes: standard prediction, role-play, and contrastive prediction, along with an innovative scoring calibration method called Contrastive Reasoning Calibration during inference. Pearson Correlation was used as the evaluation metric across all tracks. For Track 1, we achieved 0.43 on the devset and 0.17 on the testset. For Track 2 emotion, empathy, and polarity labels, we obtained 0.64, 0.66, and 0.79 on the devset and 0.61, 0.68, and 0.58 on the testset. For Track 3 empathy and distress labels, we got 0.64 and 0.56 on the devset and 0.33 and 0.35 on the testset.

ACL 2024

Chinchunmei at WASSA 2024 Empathy and Personality Shared Task:Boosting LLM’s Prediction with Role-play Augmentation and ContrastiveReasoning Calibration

contrastive reasoning

large language model

empathy detection

This paper presents the Chinchunmei team's contributions to the WASSA2024 Shared-Task 1: Empathy Detection and Emotion Classification. We participated in Tracks 1, 2, and 3 to predict empathetic scores based on dialogue, article, and essay content. We choose Llama3-8b-instruct as our base model. We developed three supervised fine-tuning schemes: standard prediction, role-play, and contrastive prediction, along with an innovative scoring calibration method called Contrastive Reasoning Calibration during inference. Pearson Correlation was used as the evaluation metric across all tracks. For Track 1, we achieved 0.43 on the devset and 0.17 on the testset. For Track 2 emotion, empathy, and polarity labels, we obtained 0.64, 0.66, and 0.79 on the devset and 0.61, 0.68, and 0.58 on the testset. For Track 3 empathy and distress labels, we got 0.64 and 0.56 on the devset and 0.33 and 0.35 on the testset.

workshop paper

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

In the realm of conversational empathy and emotion prediction, emotions are frequently categorized into multiple levels. This presentation seeks to enhance the performance of emotion prediction models by incorporating the Pearson correlation coefficient as a regularization term within the loss function. This regularization approach ensures closer alignment between predicted and actual emotion levels, mitigating extreme predictions and resulting in smoother and more consistent outputs. Such outputs are essential for capturing the subtle transitions between continuous emotion levels. Through experimental comparisons between models with and without Pearson regularization, our findings demonstrate that integrating the Pearson correlation coefficient significantly boosts model performance, yielding higher correlation scores and more accurate predictions. Our system officially ranked 9th at the Track 2: CONV-turn.

Zhenmei at WASSA-2024 Empathy and Personality Shared Track:
Incorporating Pearson Correlation Coefficient as a Regularization Term for
Enhanced Empathy and Emotion Prediction in Conversational Turns

For the WASSA 2024 Empathy and Personality Prediction Shared Task, we propose a novel turn-level empathy detection method that decomposes empathy into six psychological indicators: Emotional Language, PerspectiveTaking, Sympathy and Compassion, Extroversion, Openness, and Agreeableness. A pipeline of text enrichment using a Large Language Model (LLM) followed by DeBERTA finetuning demonstrates a significant improvement in the Pearson Correlation Coefficient and F1 scores for empathy detection, highlighting the effectiveness of our approach. Our system officially ranked 7th at the CONV-turn track.

Empaths at WASSA 2024 Empathy and Personality Shared Task:
Turn-Level Empathy Prediction Using Psychological Indicators

This paper proposes a novel ensemble ap- proach that combines Graph Neural Networks (GNNs) and LightGBM to enhance personal- ity prediction based on the personality Big 5 model. By integrating BERT embed- dings from user essays with knowledge graph- derived embeddings, our method accurately captures rich semantic and relational informa- tion. Additionally, a special loss function that combines Mean Squared Error (MSE), Pear- son correlation loss, and contrastive loss to im- prove model performance is introduced. The proposed ensemble model, made of Graph Convolutional Networks (GCNs), Graph At- tention Networks (GATs), and LightGBM, demonstrates superior performance over other models, with significant improvements in pre- diction accuracy for the Big Five personality traits achieved. Our system officially ranked 2nd at the Track 4: PER track

NU at WASSA 2024 Empathy and Personality Shared Task: Enhancing
Personality Predictions with Knowledge Graphs; A Graphical Neural
Network and LightGBM Ensemble Approach

Emotion detection and empathy analysis are important and inevitable topics with great application potentials. To provide more insights, WASSA 2024 Shared Task focuses on Empathy Detection and Emotion Classiﬁcation and Personality Detection. We propose a solution towards Track 2: Empathy and Emotion Prediction in Conversations Turns (CONV-turn), predicting the Emotion, Emotion Polarity and Empathy according to turn-level information during conversations. To achieve this goal: We adopt BERT and its variation of DeBERTa as base models, and ﬁne-tuned them on task-oriented data with adversarial training by Fast Gradient Method (FGM). We also designed the CombinedLoss, which consisted of a structured contrastive loss and a Pearson loss. After submitting to the competition: The Segmented Mix-up was proposed for data augmentation, and boosting was adopted as ensemble strategy. Regression experiments are further conducted.

hyy33 at WASSA 2024 Empathy and Personality Shared Task: Using the
CombinedLoss and FGM for Enhancing BERT-based Models in Emotion
and Empathy Prediction from Conversation Turns

Large Language Models (LLMs) can generate text by transferring style attributes like formality resulting in formal or informal text. However, instructing LLMs to generate text that when spoken, is more intelligible in an acoustically difficult environment, is an under-explored topic. We conduct the first study to evaluate LLMs on a novel task of generating acoustically intelligible paraphrases for better human speech perception in noise. Our experiments in English demonstrated that with standard prompting, LLMs struggle to control the non-textual attribute, i.e., acoustic intelligibility, while efficiently capturing the desired textual attributes like semantic equivalence. To remedy this issue, we propose a simple prompting approach, prompt-and-select, which generates paraphrases by decoupling the desired textual and non-textual attributes in the text generation pipeline. Our approach resulted in a 40% relative improvement in human speech perception, by paraphrasing utterances that are highly distorted in a listening condition with babble noise at signal-to-noise ratio (SNR) -5 dB. This study reveals the limitation of LLMs in capturing non-textual attributes, and our proposed method showcases the potential of using LLMs for better human speech perception in noise.

Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?

Traditional reference-based metrics, such as BLEU and ROUGE, are less effective for assessing outputs from Large Language Models (LLMs) that produce highly creative or superior-quality text, or in situations where reference outputs are unavailable. While human evaluation remains an option, it is costly and difficult to scale. Recent work using LLMs as evaluators (LLM-as-a-judge) is promising, but trust and reliability remain a significant concern. Integrating human input is crucial to ensure criteria used to evaluate are aligned with the human's intent, and evaluations are robust and consistent. This paper presents a user study of a design exploration called EvaluLLM, that enables users to leverage LLMs as customizable judges, promoting human involvement to balance trust and cost-saving potential with caution. Through interviews with eight domain experts, we identified the need for assistance in developing effective evaluation criteria aligning the LLM-as-a-judge with practitioners' preferences and expectations. We offer findings and design recommendations to optimize human-assisted LLM-as-judge systems.

Human-Centered Design Recommendations for LLM-as-a-judge

The field of natural language generation has witnessed significant advancements in recent years, including the development of controllable text generation techniques. However, controlling the attributes of the generated text remains a challenge, especially when aiming to avoid undesirable behavior such as toxicity. In this work, we introduce Detoxification Generator (DETOXIGEN), an inference-time algorithm that steers the generation away from unwanted styles. DETOXIGEN is an ensemble of a pre-trained language model (generator) and a detoxifier. The detoxifier is trained intentionally on the toxic data representative of the undesirable attribute, encouraging it to generate text in that style exclusively. During the actual generation, we use the trained detoxifier to produce undesirable tokens for the generator to contrast against at each decoding step. This approach directly informs the generator to avoid generating tokens that the detoxifier considers highly likely. We evaluate DETOXIGEN on the commonly used REALTOXICITYPROMPTS benchmark (Gehman et al., 2020) with various language models as generators. We find that it significantly outperforms previous approaches in detoxification metrics while not compromising on the generation quality. Moreover, the detoxifier is obtained by soft prompt-tuning using the same backbone language model as the generator. Hence, DETOXIGEN requires only a tiny amount of extra weights from the virtual tokens of the detoxifier to be loaded into GPU memory while decoding, making it a promising lightweight, practical, and parameter-efficient detoxification strategy.

Parameter-Efficient Detoxification with Contrastive Decoding

Motivational Interviewing is a counselling style that requires skillful usage of reflective listening and engaging in conversations about sensitive and personal subjects. In this paper, we investigate to what extent we can use generative large language models in motivational interviewing chatbots to generate precise and variable reflections on user responses. We conduct a two-step human evaluation where we first independently assess the generated reflections based on four criteria essential to health counseling; appropriateness, specificity, naturalness, and engagement. In the second step, we compare the overall quality of generated and human-authored reflections via a ranking evaluation. We use GPT-4, BLOOM, and FLAN-T5 models to generate motivational interviewing reflections, based on real conversational data collected via chatbots designed to provide support for smoking cessation and sexual health. We discover that GPT-4 can produce reflections of a quality comparable to human-authored reflections. Finally, we conclude that large language models have the potential to enhance and expand reflections in predetermined health counseling chatbots, but a comprehensive manual review is advised.

To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation

As Large Language Models (LLMs) and multi-agent systems are increasingly employed in intricate social environments, the research on how human interact with LLMs/agents become promising. This study aims to explore the dynamics of Human-AI interaction through a case study of the Diplomacy game, a strategic board game renowned for its emphasis on negotiation and alliance formation. The research investigates how agents and human influence each other on decision-making, trust, and strategy within interactive settings, specifically focusing on whether agents can mimic human-like strategic behaviors including deception, and how this affects human player perceptions and actions. To this end, we propose to utilize LLMs to play the game with integrated natural language processing capabilities for negotiating and strategizing. The experimental design\footnote{As this extended abstract is a position paper, we haven't yet provide experimental results but some tentative design.} will involve mixed groups of human and AI players, allowing for the analysis of game dynamics across different team configurations.

Exploring Human-AI Interaction: A Case Study on the Diplomacy Game

Active formulation of subgoals in problem-solving is an effective learning strategy for programming learners, allowing the transfer of knowledge across similar problems. Although proper guidance and feedback for learners are crucial to correct mistakes and misconceptions during subgoal formulation, providing them at scale is challenging and costly. With recent advances in generative AI, we investigate the practicality of using generative AI as TAs in programming education by examining their effectiveness in a subgoal learning environment. We explore whether programming learners can distinguish AI TAs from humans and whether the subgoal learning workflow with AI TAs yields learning gains and assess their capability to assist in coding the subgoals into executable programs. Our study shows that learners can distinguish AI TAs from human TAs based on response length and accuracy. Learners show learning gains over sessions with AI TAs in formulating subgoals and can produce code solutions faster with comparable satisfaction scores with AI TAs as human TAs.

Premium content

Chinchunmei at WASSA 2024 Empathy and Personality Shared Task:Boosting LLM’s Prediction with Role-play Augmentation and ContrastiveReasoning Calibration

Downloads

Next from ACL 2024

Zhenmei at WASSA-2024 Empathy and Personality Shared Track: Incorporating Pearson Correlation Coefficient as a Regularization Term for Enhanced Empathy and Emotion Prediction in Conversational Turns

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES