United States

In this study, we explore the application of large language models (LLMs) to analyze narrative discourse within the framework established by the field of narratology. We develop a set of elementary narrative features derived from prior theoretical work that focus on core dimensions of narrative, including time, setting, and perspective. Through experiments with GPT-4 and fine-tuned open-source models like Llama3, we demonstrate the models&#39; ability to annotate narrative passages with reasonable levels of agreement with human annotators. Leveraging a dataset of human-annotated passages spanning 18 distinct narrative and non-narrative genres, our work provides empirical support for the deictic theory of narrative communication. This theory posits that a fundamental function of storytelling is the focalization of attention on distant human experiences to facilitate social coordination. We conclude with a discussion of the possibilities for LLM-driven narrative discourse understanding.

EMNLP 2024

Using Large Language Models for Understanding Narrative Discourse

llms

fiction

narrative theory

discourse

In this study, we explore the application of large language models (LLMs) to analyze narrative discourse within the framework established by the field of narratology. We develop a set of elementary narrative features derived from prior theoretical work that focus on core dimensions of narrative, including time, setting, and perspective. Through experiments with GPT-4 and fine-tuned open-source models like Llama3, we demonstrate the models' ability to annotate narrative passages with reasonable levels of agreement with human annotators. Leveraging a dataset of human-annotated passages spanning 18 distinct narrative and non-narrative genres, our work provides empirical support for the deictic theory of narrative communication. This theory posits that a fundamental function of storytelling is the focalization of attention on distant human experiences to facilitate social coordination. We conclude with a discussion of the possibilities for LLM-driven narrative discourse understanding.

## Welcome to EMNLP 2024! 
We are excited to welcome you to one of the most prominent conferences in the field of Natural Language Processing. This year, EMNLP 2024 is being held in a hybrid format,
offering both virtual and in-person participation in beautiful Miami. Due to a record-breaking number of submissions, we've expanded the total number of accepted papers to accommodate more cutting-edge research from around the globe.
### [Conference Handbook](https://drive.google.com/file/d/1WPROgxjLAC96AJL7Ugy0tEnYm7dkrbHt/view?usp=sharing)

You are required to register for this event. **Please register [here](https://2024.emnlp.org/registration/).** The EMNLP 2024 event page on Underline will be open to public one week prior to the event.

Please register!

EMNLP 2024 will take place in Miami, Florida from Nov 12th to Nov 16th, 2024, at the Hyatt Regency Miami Hote and on Underline for remote participants.

Narrative is widely recognized as a powerful tool for structuring information and facilitating comprehension of complex ideas in various domains, such as science communication. This paper investigates whether incorporating narrative elements can assist Large Language Models (LLMs) in solving complex tasks more effectively. We propose a novel approach, Story of Thought (SoT), integrating narrative structures into prompting techniques for problem-solving tasks. This approach involves constructing narratives around problem statements and creating a framework to identify and organize relevant information. We hypothesize that this narrative-based information curation process enhances problem comprehension by contextualizing critical information and highlighting causal relationships within the problem space. Our experimental results show that the SoT approach consistently surpasses Chain of Thought (CoT) and Analogical Reasoning in GPQA tasks, achieving higher accuracy and better solutions in physics, chemistry, and biology problem-solving tasks with all tested OpenAI, Meta, and Mistral LLMs.

Can Stories Help LLMs Reason? Curating Information Space Through Narrative

Stories are a rich representation of the beliefs, practices, and languages of communities, and a source of cultural information for researchers in the social sciences and humanities. Characters – their personalities and behaviour – are a key aspect of this process. Large Language Models (LLMs) are becoming increasingly popular tools for creative text generation. Our work creates the GPT-Writing Prompts dataset, which pairs 97,219 prompts and human-written stories from the Reddit WritingPrompts dataset with corresponding GPT-3.5 generated stories. We quantify the emotional and descriptive features of protagonists in stories along six lexical dimensions: appearance, intellect, power, valence, arousal, dominance. We also quantify biases in character portrayal based on protagonist gender in both human-written and GPT-generated stories.

The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Stories

Recent advancements in NLP and language modeling have significantly enhanced LLMs’ ability to extract personal identifiers in text beyond the writing style, including names, locations, and cultural references to infer the writer’s identity. This constitutes an attack on privacy of individuals and could have fatal consequences for some personal narratives (e.g. domestic violence). In this work, we investigate domestic violence narratives in a Reddit community (r/do- mesticviolence), with the aim of identifying common patterns across a large collection of these stories. Our ultimate goal is to leverage the shared structure to develop a privacy preserving framework which assesses the safety of sharing a story. We propose a binary classification approach that utilizes distance vectors as features to classify a story as either ”similar to other domestic violence stories” or ”not similar”. This could serve as a first step toward more sophisticated methods for anonymizing the structure of sensitive narratives.

Is It Safe to Tell Your Story? Towards Achieving Privacy for Sensitive Narratives

This work presents the task of Zero-shot Trajectory Mapping, which focuses on the spatial dimension of narratives. The task consists of two parts: (1) creating a ``map'' with all the locations mentioned in a set of texts, and (2) extracting a trajectory from a single testimony and positioning it within the map. Following recent advances in context length capabilities of large language models, we propose a pipeline for this task in a completely unsupervised manner, without requiring any type of labels. We test the pipeline on a set on 402 Holocaust testimonies and present the resulting map and samples of the trajectory. We find that current long-range models succeed in generating meaningful maps and trajectories. We show how the trajectories can be used to align parallel parts of different testimonies. We conclude that LLMs can serve as beneficial tools for Holocaust research and narrative analysis.

Zero-shot Trajectory Mapping in Holocaust Testimonies

To understand how stories are structured, we would like to be able to analyze the architecture of narratives. This article reviews and compares existing annotation guidelines for scene and narrative level annotation. We propose new guidelines, based on existing ones, and show how these can be effectively extended from general-purpose to specialized contexts, such as mystery novels which feature unique narrative elements like red herrings and plot twists. This provides a controlled environment for examining genre-specific event structuring. Additionally, we present a newly annotated genre-specific dataset of mystery novels, offering valuable resources for training and evaluating models in narrative understanding. This study aims to enhance annotation practices and advance the development of computational models for narrative analysis.

Annotating Mystery Novels: Guidelines and Adaptations

Humans are naturally drawn to narratives, using them to make sense of their lives and engage with information. Even brief social media posts often take narrative forms to capture attention. This underscores the need for advanced computational methods to analyze narratives effectively, recognizing both their potential and limitations. Our study compares traditional techniques like keyword-based measurements and topic modeling with the latest concept-frame-focused narrative triplet extraction method. Using a dataset with competing narratives from two communities, we evaluate these methods' accuracy and objectivity with a Large Language Model. This research offers innovative insights for operationalizing "narrative" in natural language processing, bridging the gap between computer science and social science in narrative understanding.

Evaluating and Enhancing Methods for Narrative Understanding

Evaluating the creativity of large language models (LLMs) in story writing is difficult because LLM-generated stories could seemingly look creative but be very similar to some existing stories in their huge and proprietary training corpus. To overcome this challenge, we introduce a novel benchmark dataset with varying levels of prompt specificity: CS4 (Comparing the Skill of Creating Stories by Controlling the Synthesized Constraint Specificity). By increasing the number of requirements/constraints in the prompt, we can increase the prompt specificity and hinder LLMs from retelling high-quality narratives in their training data. Consequently, CS4 empowers us to indirectly measure the LLMs' creativity without human annotations. Our experiments on LLaMA, Gemma, and Mistral not only highlight the creativity challenges LLMs face when dealing with highly specific prompts but also reveal that different LLMs perform very differently under different numbers of constraints and achieve different balances between the model's instruction-following ability and narrative coherence. Additionally, our experiments on OLMo suggest that Learning from Human Feedback (LHF) can help LLMs select better stories from their training data but has limited influence in boosting LLMs' ability to produce creative stories that are unseen in the training corpora.

CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints

We present a novel approach to classify causal micro-narratives from text. These narratives are sentence-level explanations of the cause(s) and/or effect(s) of a target subject. The approach requires only a subject-specific ontology of causes and effects, and we demonstrate it with an application to inflation narratives. Using a human-annotated dataset spanning historical and contemporary US news articles for training, we evaluate several large language models (LLMs) on this multi-label classification task. The best-performing model---a fine-tuned Llama 3.1 8B---achieves F1 scores of 0.87 on narrative detection and 0.71 on narrative classification. Comprehensive error analysis reveals challenges arising from linguistic ambiguity and highlights how model errors often mirror human annotator disagreements. This research establishes a framework for extracting causal micro-narratives from real-world data, with wide-ranging applications to social science research.

Causal Micro-Narratives

This work analyzes character development by capturing both their religious behaviors and their expressed belief systems. We propose a computational analysis of the interplay of internal and external changes the protagonist undergoes within a narrative. Our framework employs a two-dimensional analysis, one dimension captures the protagonist’s world of beliefs, and the other describes the practices that may express their inner life. As a test case, we view Holocaust testimony transcripts, each telling the story of an individual. We focus on exploring the protagonist’s religious trajectory, examining the evolution of religious beliefs and practices throughout the narrative. The trajectories are extracted by prompting and fine-tuning LLMs. We cluster the resulting trajectories in the dataset, identifying common sequences in the data. Our findings highlight multiple common structures of religious activity across the narratives; most present a constant disposition toward religion, serving as valuable material for historical and sociological research.

Computational Analysis of Religious Journeys in Holocaust Testimonies

The limited generalization of coreference resolution (CR) models has been a major bottleneck in the task’s broad application. Prior work has identified annotation differences, especially for mention detection, as one of the main reasons for the generalization gap and proposed using additional annotated target domain data. Rather than relying on this additional annotation, we propose an alternative referential task, Major Entity Identification (MEI), where we: (a) assume the target entities to be specified in the input, and (b) limit the task to only the frequent entities. Through extensive experiments, we demonstrate that MEI models generalize well across domains on multiple datasets with supervised models and LLM-based few-shot prompting. Additionally, MEI fits the classification framework, which enables the use of robust and intuitive classification-based metrics. Finally, MEI is also of practical use as it allows a user to search for all mentions of a particular entity or a group of entities of interest

Premium content

Using Large Language Models for Understanding Narrative Discourse

Downloads

Next from EMNLP 2024

Can Stories Help LLMs Reason? Curating Information Space Through Narrative

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES