United States

Stories are a rich representation of the beliefs, practices, and languages of communities, and a source of cultural information for researchers in the social sciences and humanities. Characters – their personalities and behaviour – are a key aspect of this process. Large Language Models (LLMs) are becoming increasingly popular tools for creative text generation. Our work creates the GPT-Writing Prompts dataset, which pairs 97,219 prompts and human-written stories from the Reddit WritingPrompts dataset with corresponding GPT-3.5 generated stories. We quantify the emotional and descriptive features of protagonists in stories along six lexical dimensions: appearance, intellect, power, valence, arousal, dominance. We also quantify biases in character portrayal based on protagonist gender in both human-written and GPT-generated stories.

EMNLP 2024

The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Stories

character portrayal

human vs llm

protagonist

gpt-writing prompts dataset

generated stories

characters

stories

gender bias

large language models

## Welcome to EMNLP 2024! 
We are excited to welcome you to one of the most prominent conferences in the field of Natural Language Processing. This year, EMNLP 2024 is being held in a hybrid format,
offering both virtual and in-person participation in beautiful Miami. Due to a record-breaking number of submissions, we've expanded the total number of accepted papers to accommodate more cutting-edge research from around the globe.
### [Conference Handbook](https://drive.google.com/file/d/1WPROgxjLAC96AJL7Ugy0tEnYm7dkrbHt/view?usp=sharing)

You are required to register for this event. **Please register [here](https://2024.emnlp.org/registration/).** The EMNLP 2024 event page on Underline will be open to public one week prior to the event.

Please register!

EMNLP 2024 will take place in Miami, Florida from Nov 12th to Nov 16th, 2024, at the Hyatt Regency Miami Hote and on Underline for remote participants.

Recent advancements in NLP and language modeling have significantly enhanced LLMs’ ability to extract personal identifiers in text beyond the writing style, including names, locations, and cultural references to infer the writer’s identity. This constitutes an attack on privacy of individuals and could have fatal consequences for some personal narratives (e.g. domestic violence). In this work, we investigate domestic violence narratives in a Reddit community (r/do- mesticviolence), with the aim of identifying common patterns across a large collection of these stories. Our ultimate goal is to leverage the shared structure to develop a privacy preserving framework which assesses the safety of sharing a story. We propose a binary classification approach that utilizes distance vectors as features to classify a story as either ”similar to other domestic violence stories” or ”not similar”. This could serve as a first step toward more sophisticated methods for anonymizing the structure of sensitive narratives.

Is It Safe to Tell Your Story? Towards Achieving Privacy for Sensitive Narratives

This work presents the task of Zero-shot Trajectory Mapping, which focuses on the spatial dimension of narratives. The task consists of two parts: (1) creating a ``map'' with all the locations mentioned in a set of texts, and (2) extracting a trajectory from a single testimony and positioning it within the map. Following recent advances in context length capabilities of large language models, we propose a pipeline for this task in a completely unsupervised manner, without requiring any type of labels. We test the pipeline on a set on 402 Holocaust testimonies and present the resulting map and samples of the trajectory. We find that current long-range models succeed in generating meaningful maps and trajectories. We show how the trajectories can be used to align parallel parts of different testimonies. We conclude that LLMs can serve as beneficial tools for Holocaust research and narrative analysis.

Zero-shot Trajectory Mapping in Holocaust Testimonies

To understand how stories are structured, we would like to be able to analyze the architecture of narratives. This article reviews and compares existing annotation guidelines for scene and narrative level annotation. We propose new guidelines, based on existing ones, and show how these can be effectively extended from general-purpose to specialized contexts, such as mystery novels which feature unique narrative elements like red herrings and plot twists. This provides a controlled environment for examining genre-specific event structuring. Additionally, we present a newly annotated genre-specific dataset of mystery novels, offering valuable resources for training and evaluating models in narrative understanding. This study aims to enhance annotation practices and advance the development of computational models for narrative analysis.

Annotating Mystery Novels: Guidelines and Adaptations

Humans are naturally drawn to narratives, using them to make sense of their lives and engage with information. Even brief social media posts often take narrative forms to capture attention. This underscores the need for advanced computational methods to analyze narratives effectively, recognizing both their potential and limitations. Our study compares traditional techniques like keyword-based measurements and topic modeling with the latest concept-frame-focused narrative triplet extraction method. Using a dataset with competing narratives from two communities, we evaluate these methods' accuracy and objectivity with a Large Language Model. This research offers innovative insights for operationalizing "narrative" in natural language processing, bridging the gap between computer science and social science in narrative understanding.

Evaluating and Enhancing Methods for Narrative Understanding

Evaluating the creativity of large language models (LLMs) in story writing is difficult because LLM-generated stories could seemingly look creative but be very similar to some existing stories in their huge and proprietary training corpus. To overcome this challenge, we introduce a novel benchmark dataset with varying levels of prompt specificity: CS4 (Comparing the Skill of Creating Stories by Controlling the Synthesized Constraint Specificity). By increasing the number of requirements/constraints in the prompt, we can increase the prompt specificity and hinder LLMs from retelling high-quality narratives in their training data. Consequently, CS4 empowers us to indirectly measure the LLMs' creativity without human annotations. Our experiments on LLaMA, Gemma, and Mistral not only highlight the creativity challenges LLMs face when dealing with highly specific prompts but also reveal that different LLMs perform very differently under different numbers of constraints and achieve different balances between the model's instruction-following ability and narrative coherence. Additionally, our experiments on OLMo suggest that Learning from Human Feedback (LHF) can help LLMs select better stories from their training data but has limited influence in boosting LLMs' ability to produce creative stories that are unseen in the training corpora.

CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints

We present a novel approach to classify causal micro-narratives from text. These narratives are sentence-level explanations of the cause(s) and/or effect(s) of a target subject. The approach requires only a subject-specific ontology of causes and effects, and we demonstrate it with an application to inflation narratives. Using a human-annotated dataset spanning historical and contemporary US news articles for training, we evaluate several large language models (LLMs) on this multi-label classification task. The best-performing model---a fine-tuned Llama 3.1 8B---achieves F1 scores of 0.87 on narrative detection and 0.71 on narrative classification. Comprehensive error analysis reveals challenges arising from linguistic ambiguity and highlights how model errors often mirror human annotator disagreements. This research establishes a framework for extracting causal micro-narratives from real-world data, with wide-ranging applications to social science research.

Causal Micro-Narratives

This work analyzes character development by capturing both their religious behaviors and their expressed belief systems. We propose a computational analysis of the interplay of internal and external changes the protagonist undergoes within a narrative. Our framework employs a two-dimensional analysis, one dimension captures the protagonist’s world of beliefs, and the other describes the practices that may express their inner life. As a test case, we view Holocaust testimony transcripts, each telling the story of an individual. We focus on exploring the protagonist’s religious trajectory, examining the evolution of religious beliefs and practices throughout the narrative. The trajectories are extracted by prompting and fine-tuning LLMs. We cluster the resulting trajectories in the dataset, identifying common sequences in the data. Our findings highlight multiple common structures of religious activity across the narratives; most present a constant disposition toward religion, serving as valuable material for historical and sociological research.

Computational Analysis of Religious Journeys in Holocaust Testimonies

The limited generalization of coreference resolution (CR) models has been a major bottleneck in the task’s broad application. Prior work has identified annotation differences, especially for mention detection, as one of the main reasons for the generalization gap and proposed using additional annotated target domain data. Rather than relying on this additional annotation, we propose an alternative referential task, Major Entity Identification (MEI), where we: (a) assume the target entities to be specified in the input, and (b) limit the task to only the frequent entities. Through extensive experiments, we demonstrate that MEI models generalize well across domains on multiple datasets with supervised models and LLM-based few-shot prompting. Additionally, MEI fits the classification framework, which enables the use of robust and intuitive classification-based metrics. Finally, MEI is also of practical use as it allows a user to search for all mentions of a particular entity or a group of entities of interest

Major Entity Identification: A Generalizable Alternative to Coreference Resolution

From a communications perspective, a frame defines the packaging of the language used in such a way as to encourage certain interpretations and to discourage others. For example, a news article can frame immigration as either a boost or a drain on the economy, and thus communicate very different interpretations of the same phenomenon. In this work, we argue that to explain framing devices we have to look at the way narratives are constructed. As a first step in this direction, we propose a framework that extracts events and their relations to other events, and groups them into high-level narratives that help explain frames in news articles. We show that our framework can be used to analyze framing in U.S. news for two different domains: immigration and gun control.

Media Framing through the Lens of Event-Centric Narratives

We investigate how NLP can help annotate the structure and complexity of oral narrative texts elicited via the Multilingual Assessment Instrument for Narratives (MAIN). MAIN is a theory-based tool designed to evaluate the narrative abilities of children who are learning one or more languages from birth or early in their development. It provides a standardized way to measure how well children can comprehend and produce stories across different languages and referential norms for children between 3 and 12 years old. MAIN has been adapted to over ninety languages and is used in over 65 countries. The MAIN analysis focuses on story structure and story complexity which are typically evaluated manually based on scoring sheets. We here investigate the automation of this process using BERT-based classification which already yields promising results.

Premium content

The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Stories

Next from EMNLP 2024

Is It Safe to Tell Your Story? Towards Achieving Privacy for Sensitive Narratives

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES