Malta

Identifying direct speech in literary fiction is challenging for cases that do not mark speech segments with quotation marks. Such efforts have previously been based either on smaller manually annotated gold data or larger automatically annotated silver data, extracted from works with quotation marks. However, no comparison has so far been made between the performance of these two types of training data. In this work, we address this gap. We further explore the effect of different types of typographical speech marking and of using evaluation metrics of different granularity. We perform experiments on Swedish literary texts and find that using gold and silver data has different strengths, and gives different results on span-level and token-level metrics. If the training data contains some data that matches the speech marking of the target, that is generally sufficient for achieving good results, but it does not seem to hurt if the training data also contains other markings.

EACL 2024

Direct Speech Identification in Swedish Literature – an Exploration of Training Data Type, Typographical Markers, and Evaluation Granularity

speech tags

digital literary studies

direct speech identification

workshop paper

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

Please register!

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

We present a novel corpus consisting of orthographically variant words found in works of 19th century U.S. literature annotated with their corresponding "standard" word pair. We train a set of neural edit distance models to pair these variants with their standard forms, and compare the performance of these models to the performance of a set of neural edit distance models trained on a corpus of orthographic errors made by L2 English learners. Finally, we analyze the relative performance of these models in the light of different negative training sample generation strategies, and offer concluding remarks on the unique challenge literary orthographic variation poses to string pairing methodologies.

Pairing Orthographically Variant Literary Words to Standard Equivalents Using Neural Edit Distance Models

Coreference annotation and resolution is a vital component of computational literary studies. However, it has previously been difficult to build high quality systems for fiction. Coreference requires complicated structured outputs, and literary text involves subtle inferences and highly varied language. New language-model-based seq2seq systems present the opportunity to solve both these problems by learning to directly generate a copy of an input sentence with markdown-like annotations. We create, evaluate, and release several trained models for coreference, as well as a workflow for training new models.

[Lions: 1] and [Tigers: 2] and [Bears: 3], Oh My! Literary Coreference Annotation with LLMs

The automatic classification of stage directions is a little explored topic in computational drama analysis, in spite of their relevance for plays' structural and stylistic analysis. With a view to start assessing good practices for the automatic annotation of this textual element, we developed a 13-class stage direction typology, based on annotations in the FreDraCor corpus (French-language plays), but abstracting away from their huge variability while still providing classes useful for literary research. We fine-tuned transformers-based models to classify against the typology, gradually decreasing the corpus size used for fine tuning, to compare model efficiency with reduced training data. A result comparison speaks in favour of distilled monolingual models for this task, and, unlike earlier research on German, shows no negative effects of model case-sensitivity. The results have practical relevance for computational literary studies, as comparing classification results with complementary stage direction typologies, limiting the amount of manual annotation needed to apply them, would be helpful towards a systematic study of this important textual element.

Stage Direction Classification in French Theater: Transfer Learning Experiments

Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as conversational
agents. These models, such as GPT-4, can not only master language but also solve
complex tasks in areas like mathematics, coding, medicine, and law. However, there
is still a lack of comprehensive understanding regarding their full potential,
primarily due to the black-box nature of many models and lack of holistic evaluation.
To address these challenges, we present InstructEval, a more comprehensive evaluation
suite designed specifically for instruction-tuned large language models. Unlike
previous works, our evaluation involves a rigorous assessment of models based
on problem-solving, writing ability, and alignment to human values. We take a
holistic approach to analyze various factors affecting model performance, including
the pretraining foundation, instruction-tuning data, and training methods. Our
findings reveal that the quality of instruction data is a crucial factor in scaling
model performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment.

InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

Most adults can complete a sequence of steps to achieve a certain goal,
such as making a sandwich or repairing a bicycle tire. In completing these goal-oriented
tasks, or simply tasks in this paper, one must use sequential reasoning to understand
the relationship between the sequence of steps and the goal. LLMs have shown
impressive capabilities across various natural language understanding tasks. However,
prior work has mainly

focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); how well
LLMs can perform on more complex reasoning tasks like sequential reasoning is
not clear. In this paper, we address this gap and conduct a comprehensive evaluation
of how well LLMs are able to conduct this reasoning for tasks and how they scale
w.r.t multiple dimensions(e.g. adaptive prompting strategies, number of in-context
examples, varying complexity of the sequential task). Our findings reveal that
while Chain of Thought (CoT) prompting can significantly enhance LLMs'' sequential
reasoning in certain scenarios, it can also be detrimental in others, whereas
Tree of Thoughts (ToT) reasoning is less effective for this type of task. Additionally,
we discover that an increase in model size or in-context examples does not consistently
lead to improved performance.

Can Large Language Models Reason About Goal-Oriented Tasks?

No two authors write alike. Personal flourishes invoked in written narratives,
from lexicon to rhetorical devices, imply a particular author—what literary theorists
label the implied or virtual author; distinct from the real author or narrator
of a text. Early large language models trained on unfiltered training sets drawn
from a variety of discordant sources yielded incoherent personalities, problematic
for conversational tasks but proving useful for sampling literature from multiple
perspectives. Successes in alignment research in recent years have allowed researchers
to impose subjectively consistent personae on language models via instruction
tuning and reinforcement learning from human feedback (RLHF), but whether aligned
models retain the ability to model an arbitrary virtual author has received little
scrutiny. By studying 4,374 stories sampled from three OpenAI language models,
we show successive versions of GPT-3 suffer from increasing degrees of “mode collapse”
whereby overfitting the model during alignment constrains it from generalizing
over authorship: models suffering from mode collapse become unable to assume a
multiplicity of perspectives. Our method and results are significant for researchers
seeking to employ language models in sociological simulations.

Detecting Mode Collapse in Language Models via Narration

The extraction of structured data from websites is critical for numerous Artificial Intelligence applications, but modern web design increasingly stores information visually in images rather than in text. This shift calls into question the optimal technique, as language-only models fail without textual cues while new multimodal models like GPT-4 promise image understanding abilities. We conduct the first rigorous comparison between text-based and vision-based models for extracting event metadata harvested from comic convention websites. Surprisingly, our results between GPT-4 Vision and GPT-4 Text uncover a significant accuracy advantage for vision-based methods in an applies-to-apples setting, indicating that vision models may be outpacing language-alone techniques in the task of information extraction from websites. We release our dataset and provide a qualitative analysis to guide further research in multi-modal models for web information extraction.

The Future of Web Data Mining: Insights from Multimodal and Code-based Extraction Methods

Freedom of Information Act (FOIA) legislation grants citizens the right to request information from various levels of the government, and aims to promote the transparency of governmental agencies. However, the processing of these requests is often met with delays, due to the inherent complexity of gathering the required documents. To obtain accurate estimates of the processing times of requests, and to identify bottlenecks in the process, this research proposes a pipeline to automatically extract these timelines from decision letters of Dutch FOIA requests. These decision letters are responses to requests, and contain an overview of the process, including when the request was received, and possible communication between the requester and the relevant agency. The proposed pipeline can extract dates with an accuracy of .94, extract event phrases with a mean ROUGE- L F1 score of .80 and can classify events with a macro F1 score of .79. Out of the 50 decision letters used for testing (each letter containing one timeline), the model correctly classified 10 of the timelines completely correct, with an average of 3.1 mistakes per decision letter.

Timeline Extraction from Decision Letters Using ChatGPT

We describe a new weakly supervised method for sentence-level event detection, based exclusively on linear prototype patterns like "people got sick" or "a roadside bomb killed people". We propose a new BERT based algorithm for approximate pattern matching to identify event phrases, semantically similar to these prototypes. To the best of our knowledge, a similar approach has not been used in the context of event detection. We experimented with two event corpora in the area of disease outbreaks and terrorism and we achieved promising results in sentence level event identification: 0.78 F1 score for new disease cases detection and 0.68 F1 in detecting terrorist attacks. Results were in line with some state-of-the-art systems.

Leveraging Approximate Pattern Matching with BERT for Event Detection

There is a large and growing body of literature on datasets created to facilitate the study of socio-political events of conflict and unrest. However, the datasets, and the approaches taken to create them, vary a lot depending on the type of research they are intended to support. For example, while scholars from natural language processing (NLP) tend to focus on annotating specific spans of text indicating various components of an event, scholars from the disciplines of political science and conflict studies tend to focus on creating databases that code an abstract but structured representation of the event, less tied to a specific source text. The survey presented in this paper aims to map out the current landscape of available event datasets within the domain of social and political conflict and unrest ‚Äì both from the NLP and political science communities ‚Äì offering a unified view of the work done across different disciplines.

Premium content

Direct Speech Identification in Swedish Literature – an Exploration of Training Data Type, Typographical Markers, and Evaluation Granularity

Downloads

Next from EACL 2024

Pairing Orthographically Variant Literary Words to Standard Equivalents Using Neural Edit Distance Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES