Thailand

Large language models (LLMs) often face knowledge conflicts when addressing questions that can yield different answers based on the cultural or regional context. In this work, we create the **QARV** benchmark to evaluate LLM&#39;s proficiency in managing these conflicts between two distinct countries, the US and Korea. The benchmark consists of 671 handwritten questions, each accompanied by two answers from each nation. Our evaluation of 42 different LLMs, each tested with nine unique prompts, reveals that most models inherently respond with an American bias. Additionally, we observe that prompting, and instruction-tuning, are all practical measures to enhance cultural alignment.

ACL 2024

Assessing LLMs&#39; Ability to Navigate Cultural Knowledge Conflicts

cultural knowledge conflict

cultural alignment

Large language models (LLMs) often face knowledge conflicts when addressing questions that can yield different answers based on the cultural or regional context. In this work, we create the **QARV** benchmark to evaluate LLM's proficiency in managing these conflicts between two distinct countries, the US and Korea. The benchmark consists of 671 handwritten questions, each accompanied by two answers from each nation. Our evaluation of 42 different LLMs, each tested with nine unique prompts, reveals that most models inherently respond with an American bias. Additionally, we observe that prompting, and instruction-tuning, are all practical measures to enhance cultural alignment.

Assessing LLMs' Ability to Navigate Cultural Knowledge Conflicts

workshop paper

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

Cultural differences in common ground may result in pragmatic failure and misunderstandings. We develop our method Rational Speech Acts for Cross-Cultural Communication (RSA+C3) to resolve cross-cultural differences in common ground. To measure the success of our method, we study RSA+C3 in the collaborative referential game of Codenames Duet and show that our method successfully improves collaboration between simulated players of different cultures. Our contributions are threefold: (1) creating Codenames players using contrastive learning of an embedding space and LLM prompting that are aligned with human patterns of play, (2) studying culturally induced differences in common ground reflected in our trained models, and (3) demonstrating that our method RSA+C3 can ease cross-cultural communication in gameplay by inferring sociocultural context from interaction.

RSA+C3: Cross-Cultural Communication using RSA in Codenames

Large language models (LLMs) show limitations in comprehending everyday cultural commonsense knowledge in diverse regions, especially in non-English languages, as these are not necessarily written explicitly online. Existing works, however, mainly focus on a single language or rely heavily on online data sources like Wikipedia, which are often included in the training data. To address this, we present BLEnD, a multilingual socio-cultural commonsense benchmark with 13K question samples from carefully hand-crafted templates. The benchmark covers cultural aspects across 9 countries and regions (Azerbaijan, China, Greece, Indonesia, Mexico, Spain, UK, US, and West Java) in 7 languages (Azerbaijani, Chinese, English, Greek, Indonesian, Spanish, and Sundanese). We show that LLMs perform better in high-resource cultures, with a maximum 27.9%p difference in GPT-4 under a short-answer question format. Furthermore, they perform better for mid-to-high-resource cultures in their local languages, while for low-resource cultures, they perform better in English.

BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain—a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.

Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge

Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in \~4.5K questions per language (\~9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

We present a research agenda focused on efficiently extracting, assuring quality, and consolidating textual company sustainability information to address urgent climate change decision-making needs. Starting from the goal to create integrated FAIR (Findable, Accessible, Interoperable, Reusable) climate-related data, we identify research needs pertaining to the technical aspects of information extraction as well as to the design of the integrated sustainability datasets that we seek to compile. Regarding extraction, we leverage technological advancements, particularly in large language models (LLMs) and Retrieval-Augmented Generation (RAG) pipelines, to unlock the underutilized potential of unstructured textual information contained in corporate sustainability reports. In applying these techniques, we review key challenges, which include the retrieval and extraction of CO 
emission values from PDF documents, especially from unstructured tables and graphs therein, and the validation of automatically extracted data through comparisons with human-annotated values. We also review how existing use cases and practices in climate risk analytics relate to choices of what textual information should be extracted and how it could be linked to existing structured data.

Informing climate risk analysis using textual information - A research agenda

Climate adaptation in the agricultural sector necessitates tools that equip farmers and farm advisors with relevant and trustworthy information to help increase their resiliency to climate change. We introduce {\em My Climate Advisor}, a question-answering (QA) prototype that synthesises information from different data sources, such as peer-reviewed scientific literature and high-quality, industry-relevant grey literature to generate answers, with references, to a given user's question. Our prototype uses open-source generative models for data privacy and intellectual property protection, and retrieval augmented generation for answer generation, grounding and provenance. While there are standard evaluation metrics for QA systems, no existing evaluation framework suits our LLM-based QA application in the climate adaptation domain. We design an evaluation framework with seven metrics based on the requirements of the domain experts to judge the generated answers from 12 different LLM-based models. Our initial evaluations through a user study via domain experts show promising usability results.

My Climate Advisor: An Application of NLP in Climate Adaptation for Agriculture

This paper presents the ClimateSent-GAT Model, a novel approach that combines Graph Attention Networks (GATs) with natural language processing techniques to accurately identify and predict disagreements within Reddit comment-reply pairs. Our model classifies disagreements into three categories: agree, disagree, and neutral. Leveraging the inherent graph structure of Reddit comment-reply pairs, the model significantly outperforms existing benchmarks by capturing complex interaction patterns and sentiment dynamics. This research advances graph-based NLP methodologies and provides actionable insights for policymakers and educators in climate science communication.

Decoding Climate Disagreement: A Graph Neural Network-Based Approach to Understanding Social Media Dynamics

This paper introduces and evaluates ChatNetZero, a large-language model (LLM) chatbot developed through Retrieval-Augmented Generation (RAG), which uses generative AI to produce answers grounded in verified, climate-domain specific information. We describe ChatNetZero's design, particularly the innovation of anti-hallucination and reference modules designed to enhance the accuracy and credibility of generated responses. To evaluate ChatNetZero's performance against other LLMs, including GPT-4, Gemini, Coral, and ChatClimate, we conduct two types of validation: comparing LLMs' generated responses to original source documents to verify their factual accuracy, and employing an expert survey to evaluate the overall quality, accuracy and relevance of each response. We find that while ChatNetZero responses show higher factual accuracy when compared to original source data, experts surveyed prefer lengthier responses that provide more context. Our results highlight the importance of prioritizing information presentation in the design of domain-specific LLMs to ensure that scientific information is effectively communicated, especially as even expert audiences find it challenging to assess the credibility of AI-generated content.

Evaluating ChatNetZero, an LLM-Chatbot to Demystify Climate Pledges

To better understand how extreme climate events impact society, we need to increase the availability of accurate and comprehensive information about these impacts. We propose a method for building large-scale databases of climate extreme impacts from online textual sources, using LLMs for information extraction in combination with more traditional NLP techniques to improve accuracy and consistency. We evaluate the method against a small benchmark database created by human experts and find that extraction accuracy varies for different types of information. We compare three different LLMs and find that, while the commercial GPT-4 model gives the best performance overall, the open-source models Mistral and Mixtral are competitive for some types of information.

Using LLMs to Build a Database of Climate Extreme Impacts

Climate communication is often seen by the NLP community as an opportunity for machine translation, applied to ever smaller languages. However, over 90% the world's linguistic diversity comes from languages with `primary orality' and mostly spoken in non-Western oral societies. A case in point is the Aboriginal communities of Northern Australia, where we have been conducting workshops on climate communication, revealing shortcomings in existing communication practices along with new opportunities for improving intercultural communication. We present a case study of climate communication in an oral society, including the voices of many local people, and draw several lessons for the research program of NLP in the climate space.

Assessing LLMs' Ability to Navigate Cultural Knowledge Conflicts

Downloads

Next from ACL 2024

RSA+C3: Cross-Cultural Communication using RSA in Codenames

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES