Lecture image placeholder

Premium content

Access to this content requires a subscription. You must be a premium user to view this content.

Monthly subscription - $9.99Pay per view - $4.99Access through your institutionLogin with Underline account
Need help?
Contact us
Lecture placeholder background
VIDEO DOI: https://doi.org/10.48448/9v59-jq76

workshop paper

ACL 2024

August 16, 2024

Bangkok, Thailand

Assessing LLMs' Ability to Navigate Cultural Knowledge Conflicts

keywords:

cultural knowledge conflict

cultural alignment

llm

Large language models (LLMs) often face knowledge conflicts when addressing questions that can yield different answers based on the cultural or regional context. In this work, we create the QARV benchmark to evaluate LLM's proficiency in managing these conflicts between two distinct countries, the US and Korea. The benchmark consists of 671 handwritten questions, each accompanied by two answers from each nation. Our evaluation of 42 different LLMs, each tested with nine unique prompts, reveals that most models inherently respond with an American bias. Additionally, we observe that prompting, and instruction-tuning, are all practical measures to enhance cultural alignment.

Downloads

SlidesTranscript English (automatic)

Next from ACL 2024

RSA+C3: Cross-Cultural Communication using RSA in Codenames
workshop paper

RSA+C3: Cross-Cultural Communication using RSA in Codenames

ACL 2024

Isadora White

16 August 2024

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)

PRESENTATIONS

  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved