Malta

Digital archive collections that have been contributed by communities, known as community-generated digital content (CGDC), are important sources of historical and cultural knowledge. However, CGDC items are not easily searchable due to semantic information being obscured within their textual metadata. In this paper, we investigate the extent to which state-of-the-art, general-domain entity linking (EL) models (i.e., BLINK, EPGEL and mGENRE) can map named entities mentioned in CGDC textual metadata, to Wikidata entities. We evaluate and compare their performance on an annotated dataset of CGDC textual metadata and provide some error analysis, in the way of informing future studies aimed at enriching CGDC metadata using entity linking methods.

EACL 2024

Enriching the Metadata of Community-Generated Digital Content through Entity Linking: An Evaluative Comparison of State-of-the-Art Models

community-generated digital content (cgdc) entity linking (el) textual metadata named entity recognition (ner) knowledge base enrichment

workshop paper

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

Please register!

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

The application of text mining methods is gaining increasing attention not only in the Digital Humanities (DH) and Computational Social Sciences (CSS) in general, but also in vocational education and training (VET) research. The use of algorithms promises to enable research on corpora that are simply too large for manual methods. The challenge is that concepts such as occupations or skills, which play an important role in VET research, are abstract in principle, but algorithms require concrete instructions for action (in terms of rules or annotated examples), which have to be negotiated during the process of operationalisation. Using the concept of occupations, we will compare two approaches for automatically identifying the concept in a large corpus. We will take debates from the German Bundestag and we will compare a dictionary-based approach with a BERT fine-tuning, two methods that are widely used in the context of DH and CSS.

Recognising Occupational Titles in German Parliamentary Debates

We present a novel combination of dynamic embedded topic models and change-point detection to explore diachronic change of lexical semantic modality in classical and early Christian Latin. We demonstrate several methods for finding and characterizing patterns in the output, and relating them to traditional scholarship in Comparative Literature and Classics. This simple approach to unsupervised models of semantic change can be applied to any suitable corpus, and we conclude with future directions and refinements aiming to allow noisier, less-curated materials to meet that threshold.

Dynamic embedded topic models and change-point detection for exploring literary-historical hypotheses

Many collections of digitized newspapers suffer from poor OCR quality, which impacts readability, information retrieval, and analysis of the material. Errors in OCR output can be reduced by applying machine translation models to "translate" it into a corrected version. Although transformer models show promising results in post-OCR correction and related tasks in other languages, they have not yet been explored for correcting OCR errors in Swedish texts. This paper presents a post-OCR correction model for Swedish 19th to 21th century newspapers based on the pre-trained transformer model ByT5. Three versions of the model were trained on different mixes of training data. The best model, which achieved a 36% reduction in CER, is made freely available and will be integrated into the automatic processing pipeline of Språkbanken Text, a Swedish language technology infrastructure containing modern and historical written data.

Post-OCR Correction of Digitized Swedish Newspapers with ByT5

In this paper we present the Kronieken Corpus, a new digital collection of 204 chronicles written in Dutch/Flemish between 1500 and 1850, which have been scanned, transcribed and annotated with named entities, dates, pages and a smaller part with sources and attributions. The texts belong to 308 physical volumes and contain between 23 and 24 million words. 107 chronicles, or 178 chronicle volumes, collected from 39 different archives and libraries in The Netherlands and Belgium and transcribed by volunteers had never been transcribed or published before. The result is a unique enriched historical text corpus of original hand-written, non-canonical and non-fiction text by lay people from the early modern period.

The Kronieken Corpus: an Annotated Collection of Dutch/Flemish Chronicles from 1500-1850

Identifying direct speech in literary fiction is challenging for cases that do not mark speech segments with quotation marks. Such efforts have previously been based either on smaller manually annotated gold data or larger automatically annotated silver data, extracted from works with quotation marks. However, no comparison has so far been made between the performance of these two types of training data. In this work, we address this gap. We further explore the effect of different types of typographical speech marking and of using evaluation metrics of different granularity. We perform experiments on Swedish literary texts and find that using gold and silver data has different strengths, and gives different results on span-level and token-level metrics. If the training data contains some data that matches the speech marking of the target, that is generally sufficient for achieving good results, but it does not seem to hurt if the training data also contains other markings.

Direct Speech Identification in Swedish Literature – an Exploration of Training Data Type, Typographical Markers, and Evaluation Granularity

We present a novel corpus consisting of orthographically variant words found in works of 19th century U.S. literature annotated with their corresponding "standard" word pair. We train a set of neural edit distance models to pair these variants with their standard forms, and compare the performance of these models to the performance of a set of neural edit distance models trained on a corpus of orthographic errors made by L2 English learners. Finally, we analyze the relative performance of these models in the light of different negative training sample generation strategies, and offer concluding remarks on the unique challenge literary orthographic variation poses to string pairing methodologies.

Pairing Orthographically Variant Literary Words to Standard Equivalents Using Neural Edit Distance Models

Coreference annotation and resolution is a vital component of computational literary studies. However, it has previously been difficult to build high quality systems for fiction. Coreference requires complicated structured outputs, and literary text involves subtle inferences and highly varied language. New language-model-based seq2seq systems present the opportunity to solve both these problems by learning to directly generate a copy of an input sentence with markdown-like annotations. We create, evaluate, and release several trained models for coreference, as well as a workflow for training new models.

[Lions: 1] and [Tigers: 2] and [Bears: 3], Oh My! Literary Coreference Annotation with LLMs

The automatic classification of stage directions is a little explored topic in computational drama analysis, in spite of their relevance for plays' structural and stylistic analysis. With a view to start assessing good practices for the automatic annotation of this textual element, we developed a 13-class stage direction typology, based on annotations in the FreDraCor corpus (French-language plays), but abstracting away from their huge variability while still providing classes useful for literary research. We fine-tuned transformers-based models to classify against the typology, gradually decreasing the corpus size used for fine tuning, to compare model efficiency with reduced training data. A result comparison speaks in favour of distilled monolingual models for this task, and, unlike earlier research on German, shows no negative effects of model case-sensitivity. The results have practical relevance for computational literary studies, as comparing classification results with complementary stage direction typologies, limiting the amount of manual annotation needed to apply them, would be helpful towards a systematic study of this important textual element.

Stage Direction Classification in French Theater: Transfer Learning Experiments

Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as conversational
agents. These models, such as GPT-4, can not only master language but also solve
complex tasks in areas like mathematics, coding, medicine, and law. However, there
is still a lack of comprehensive understanding regarding their full potential,
primarily due to the black-box nature of many models and lack of holistic evaluation.
To address these challenges, we present InstructEval, a more comprehensive evaluation
suite designed specifically for instruction-tuned large language models. Unlike
previous works, our evaluation involves a rigorous assessment of models based
on problem-solving, writing ability, and alignment to human values. We take a
holistic approach to analyze various factors affecting model performance, including
the pretraining foundation, instruction-tuning data, and training methods. Our
findings reveal that the quality of instruction data is a crucial factor in scaling
model performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment.

InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

Most adults can complete a sequence of steps to achieve a certain goal,
such as making a sandwich or repairing a bicycle tire. In completing these goal-oriented
tasks, or simply tasks in this paper, one must use sequential reasoning to understand
the relationship between the sequence of steps and the goal. LLMs have shown
impressive capabilities across various natural language understanding tasks. However,
prior work has mainly

focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); how well
LLMs can perform on more complex reasoning tasks like sequential reasoning is
not clear. In this paper, we address this gap and conduct a comprehensive evaluation
of how well LLMs are able to conduct this reasoning for tasks and how they scale
w.r.t multiple dimensions(e.g. adaptive prompting strategies, number of in-context
examples, varying complexity of the sequential task). Our findings reveal that
while Chain of Thought (CoT) prompting can significantly enhance LLMs'' sequential
reasoning in certain scenarios, it can also be detrimental in others, whereas
Tree of Thoughts (ToT) reasoning is less effective for this type of task. Additionally,
we discover that an increase in model size or in-context examples does not consistently
lead to improved performance.

Premium content

Enriching the Metadata of Community-Generated Digital Content through Entity Linking: An Evaluative Comparison of State-of-the-Art Models

Downloads

Next from EACL 2024

Recognising Occupational Titles in German Parliamentary Debates

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES