Thailand

Integrating Large Language Models (LLMs) into various global cultures presents a cultural challenge: they must respect social norms, and avoid transgressing cultural boundaries. This calls for LLMs to be able to $\textit{adapt}$ their outputs to diverse cultural norms. However, the extent of this $\textit{cultural adaptability}$ remains unclear. To this end, we introduce $NormAd$, a dataset of 2.6k stories that represent social and cultural norms from 75 countries to assess the ability of LLMs to adapt to different granular contexts. Evaluation on $NormAd$ suggests that LLMs struggle with cultural reasoning across all contextual granularities -- Mistral-7b-Instruct, one of the top performing models, achieves only 81.8\% accuracy versus 95.6\% achieved by humans. Additionally, LLMs show stronger adaptability to English-centric cultures over those from the Global South, and find it considerably easier to assess the social acceptability of stories that adhere to cultural norms than those that deviate from them. Our benchmark emphasizes the potential to make models more equitable towards global audiences.

ACL 2024

$NormAd$: A Benchmark for Measuring the Cultural Adaptability of Large Language Models

contexts

adaptability

culture

workshop paper

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by systematically training six LLMs of identical size (2.6B parameters) and architecture: five monolingual models (English, German, French, Italian, and Spanish) and one multilingual model trained on an equal distribution of data across these languages, all using publicly available data. To ensure robust evaluation, standard bias benchmarks were automatically translated into the five target languages and verified for both translation quality and bias preservation by human annotators. Our results consistently demonstrate that multilingual training effectively mitigates bias. Moreover, we observe that multilingual models achieve not only lower bias but also superior prediction accuracy when compared to monolingual models with the same amount of training data, model architecture, and size.

Do Multilingual Large Language Models Mitigate Stereotype Bias?

Large language models (LLMs) often face knowledge conflicts when addressing questions that can yield different answers based on the cultural or regional context. In this work, we create the **QARV** benchmark to evaluate LLM's proficiency in managing these conflicts between two distinct countries, the US and Korea. The benchmark consists of 671 handwritten questions, each accompanied by two answers from each nation. Our evaluation of 42 different LLMs, each tested with nine unique prompts, reveals that most models inherently respond with an American bias. Additionally, we observe that prompting, and instruction-tuning, are all practical measures to enhance cultural alignment.

Assessing LLMs' Ability to Navigate Cultural Knowledge Conflicts

Cultural differences in common ground may result in pragmatic failure and misunderstandings. We develop our method Rational Speech Acts for Cross-Cultural Communication (RSA+C3) to resolve cross-cultural differences in common ground. To measure the success of our method, we study RSA+C3 in the collaborative referential game of Codenames Duet and show that our method successfully improves collaboration between simulated players of different cultures. Our contributions are threefold: (1) creating Codenames players using contrastive learning of an embedding space and LLM prompting that are aligned with human patterns of play, (2) studying culturally induced differences in common ground reflected in our trained models, and (3) demonstrating that our method RSA+C3 can ease cross-cultural communication in gameplay by inferring sociocultural context from interaction.

RSA+C3: Cross-Cultural Communication using RSA in Codenames

Large language models (LLMs) show limitations in comprehending everyday cultural commonsense knowledge in diverse regions, especially in non-English languages, as these are not necessarily written explicitly online. Existing works, however, mainly focus on a single language or rely heavily on online data sources like Wikipedia, which are often included in the training data. To address this, we present BLEnD, a multilingual socio-cultural commonsense benchmark with 13K question samples from carefully hand-crafted templates. The benchmark covers cultural aspects across 9 countries and regions (Azerbaijan, China, Greece, Indonesia, Mexico, Spain, UK, US, and West Java) in 7 languages (Azerbaijani, Chinese, English, Greek, Indonesian, Spanish, and Sundanese). We show that LLMs perform better in high-resource cultures, with a maximum 27.9%p difference in GPT-4 under a short-answer question format. Furthermore, they perform better for mid-to-high-resource cultures in their local languages, while for low-resource cultures, they perform better in English.

BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain—a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.

Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge

Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in \~4.5K questions per language (\~9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

We present a research agenda focused on efficiently extracting, assuring quality, and consolidating textual company sustainability information to address urgent climate change decision-making needs. Starting from the goal to create integrated FAIR (Findable, Accessible, Interoperable, Reusable) climate-related data, we identify research needs pertaining to the technical aspects of information extraction as well as to the design of the integrated sustainability datasets that we seek to compile. Regarding extraction, we leverage technological advancements, particularly in large language models (LLMs) and Retrieval-Augmented Generation (RAG) pipelines, to unlock the underutilized potential of unstructured textual information contained in corporate sustainability reports. In applying these techniques, we review key challenges, which include the retrieval and extraction of CO 
emission values from PDF documents, especially from unstructured tables and graphs therein, and the validation of automatically extracted data through comparisons with human-annotated values. We also review how existing use cases and practices in climate risk analytics relate to choices of what textual information should be extracted and how it could be linked to existing structured data.

Informing climate risk analysis using textual information - A research agenda

Climate adaptation in the agricultural sector necessitates tools that equip farmers and farm advisors with relevant and trustworthy information to help increase their resiliency to climate change. We introduce {\em My Climate Advisor}, a question-answering (QA) prototype that synthesises information from different data sources, such as peer-reviewed scientific literature and high-quality, industry-relevant grey literature to generate answers, with references, to a given user's question. Our prototype uses open-source generative models for data privacy and intellectual property protection, and retrieval augmented generation for answer generation, grounding and provenance. While there are standard evaluation metrics for QA systems, no existing evaluation framework suits our LLM-based QA application in the climate adaptation domain. We design an evaluation framework with seven metrics based on the requirements of the domain experts to judge the generated answers from 12 different LLM-based models. Our initial evaluations through a user study via domain experts show promising usability results.

My Climate Advisor: An Application of NLP in Climate Adaptation for Agriculture

This paper presents the ClimateSent-GAT Model, a novel approach that combines Graph Attention Networks (GATs) with natural language processing techniques to accurately identify and predict disagreements within Reddit comment-reply pairs. Our model classifies disagreements into three categories: agree, disagree, and neutral. Leveraging the inherent graph structure of Reddit comment-reply pairs, the model significantly outperforms existing benchmarks by capturing complex interaction patterns and sentiment dynamics. This research advances graph-based NLP methodologies and provides actionable insights for policymakers and educators in climate science communication.

Decoding Climate Disagreement: A Graph Neural Network-Based Approach to Understanding Social Media Dynamics

This paper introduces and evaluates ChatNetZero, a large-language model (LLM) chatbot developed through Retrieval-Augmented Generation (RAG), which uses generative AI to produce answers grounded in verified, climate-domain specific information. We describe ChatNetZero's design, particularly the innovation of anti-hallucination and reference modules designed to enhance the accuracy and credibility of generated responses. To evaluate ChatNetZero's performance against other LLMs, including GPT-4, Gemini, Coral, and ChatClimate, we conduct two types of validation: comparing LLMs' generated responses to original source documents to verify their factual accuracy, and employing an expert survey to evaluate the overall quality, accuracy and relevance of each response. We find that while ChatNetZero responses show higher factual accuracy when compared to original source data, experts surveyed prefer lengthier responses that provide more context. Our results highlight the importance of prioritizing information presentation in the design of domain-specific LLMs to ensure that scientific information is effectively communicated, especially as even expert audiences find it challenging to assess the credibility of AI-generated content.

Premium content

$NormAd$: A Benchmark for Measuring the Cultural Adaptability of Large Language Models

Downloads

Next from ACL 2024

Do Multilingual Large Language Models Mitigate Stereotype Bias?

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES