Throughout history, pictorial record-keeping has been used to document events, stories, and concepts.  A popular example of this is the Tzolk'in Maya calendar. The pre-Columbian Mixtec society also recorded many works through graphical media called codices that depict both stories and real events. Mixtec codices are unique because the depicted scenes are highly structured within and across documents. As a first effort toward translation, we created two binary classification tasks over Mixtec codices, namely, gender and pose. The composition of figures within a codex is essential for understanding the codex's narrative. We labeled a dataset with around 1300 figures drawn from three codices of varying qualities. We finetuned the Visual Geometry Group 16 (VGG-16) and Vision Transformer 16 (ViT-16) models, measured their performance, and compared learned features with expert opinions found in literature.  The results show that when finetuned, both VGG and ViT perform well, with the transformer-based architecture (ViT) outperforming the CNN-based architecture (VGG) at higher learning rates.  We are releasing this work to allow collaboration with the Mixtec community and domain scientists.

Analyzing Finetuned Vision Models for Mixtec Codex Interpretation

In Chinese studies, understanding the nuanced traits of historical figures, often not explicitly evident in biographical data, has been a key interest. 
However, identifying these traits can be challenging due to the need for domain expertise, specialist knowledge, and context-specific insights, making the process time-consuming and difficult to scale. 
Our focus on studying officials from China's Ming Dynasty is no exception.
To tackle this challenge, we propose MingOfficial, a large-scale multi-modal dataset consisting of both structured (career records, annotated personnel types) and text (historical texts) data for $9,376$ officials.
We further couple the dataset with a a graph neural network (GNN) to combine both modalities in order to allow investigation of social structures and provide features to boost down-stream tasks.
Experiments show that our proposed MingOfficial could enable exploratory analysis of official identities, and also significantly boost performance in tasks such as identifying nuance identities (e.g.\ civil officials holding military power) from $24.6\%$ to $98.2\%$ F$_1$ score in hold-out test set. By making MingOfficial publicly available (see main text for the URL) as both a dataset and an interactive tool, we aim to stimulate further research into the role of social context and representation learning in identifying individual characteristics, and hope to provide inspiration for computational approaches in other fields beyond Chinese studies.

MingOfficial: A Ming Official Career Dataset and a Historical Context-Aware Representation Learning Framework | VIDEO

The digitisation of historical documents has provided historians with unprecedented research opportunities. Yet, the conventional approach to analysing historical documents involves converting them from images to text using OCR, a process that overlooks the potential benefits of treating them as images and introduces high levels of noise. To bridge this gap, we take advantage of recent advancements in pixel-based language models trained to reconstruct masked patches of pixels instead of predicting token distributions. Due to the scarcity of real historical scans, we propose a novel method for generating synthetic scans to resemble real historical documents. We then pre-train our model, PHD, on a combination of synthetic scans and real historical newspapers from the 1700-1900 period. Through our experiments, we demonstrate that PHD exhibits high proficiency in reconstructing masked image patches and provide evidence of our model's noteworthy language understanding capabilities. Notably, we successfully apply our model to a historical QA task, highlighting its usefulness in this domain.

PHD: Pixel-Based Language Modeling of Historical Documents

Large Language Models (LLMs) have gained significant popularity for their impressive performance across diverse fields. However, LLMs are prone to hallucinate untruthful or nonsensical outputs that fail to meet user expectations in many real-world applications. Existing works for detecting hallucinations in LLMs either rely on external knowledge for reference retrieval or require sampling multiple responses from the LLM for consistency verification, making these methods costly and inefficient. In this paper, we propose a novel reference-free, uncertainty-based method for detecting hallucinations in LLMs. Our approach imitates human focus in factuality checking from three aspects: 1) focus on the most informative and important keywords in the given text; 2) focus on the unreliable tokens in historical context which may lead to a cascade of hallucinations; and 3) focus on the token properties such as token type and token frequency. Experimental results on relevant datasets demonstrate the effectiveness of our proposed method, which achieves state-of-the-art performance across all the evaluation metrics and eliminates the need for additional information.

Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus

The identification of cognates is a fundamental process in historical linguistics, on which any further research is based. Even though there are several cognate databases for Romance languages, they are rather scattered, incomplete, noisy, contain unreliable information, or have uncertain availability.
In this paper we introduce a comprehensive database of Romance cognates and borrowings based on the etymological information provided by the dictionaries. We extract pairs of cognates between any two Romance languages by parsing electronic dictionaries of Romanian, Italian, Spanish, Portuguese and French. Based on this resource, we propose a strong benchmark for the automatic detection of cognates, by applying machine learning and deep learning based methods on any two pairs of Romance languages. We find that automatic identification of cognates is possible with accuracy averaging around 94% for the more difficult task formulations.

RoBoCoP: A Comprehensive ROmance BOrrowing COgnate Package and Benchmark for Multilingual Cognate Identification

Phonological reconstruction is one of the central problems in historical linguistics where a proto-word of an ancestral language is determined from the observed cognate words of daughter languages. Computational approaches to historical linguistics attempt to automate the task by learning models on available linguistic data. Several ideas and techniques drawn from computational biology have been successfully applied in this area of computational historical linguistics. Following these lines, we adapt MSA Transformer, a protein language model, to the problem of automated phonological reconstruction. MSA Transformer trains on multiple sequence alignments as input and is, thus, apt for application on aligned cognate words. We, hence, name our model as Cognate Transformer. We also apply the model on another associated task, namely, cognate reflex prediction where a reflex word in a daughter language is predicted based on cognate words from other daughter languages. We show that our model outperforms the existing models on both the tasks, especially when it is pre-trained on masked word prediction task.

Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction | VIDEO

 Self-domestication refers to a recent hypothesis about how our species emerged. It claims that the human distinctiveness is, to a large extent, the outcome of an evolutionary process similar to animal domestication. This includes our enhanced social cognition, increased cooperation, and extended social networks, and ultimately, our advanced technology and sophisticated culture. Language is perhaps the most distinctive of human features. In this talk, I will present a model of language evolution under the effects of self-domestication forces. At the basis of this model one can find the idea that self-domestication, through its impact on our behavior, facilitated the creation of the cultural niche that enables the sophistication of language structure and use via a cultural mechanism. Some cognitive changes to which self-domestication might have also aided, like our increased cognitive fluidity, could have also contributed to make human language and human pragmatics more complex. This evolutionary model ties different stages in the evolution of languages with changes in the management of aggression (at the core of the self-domestication hypothesis), with both aspects being engaged in a mutually reinforcing feedback loop, as both depend on, and impact on a common neurobiological substrate. The model argues for a strong continuity between human language and the cognitive abilities and behaviors of other species, but it also grants cultural evolution a crucial role in language evolution.

The evolution of language under self-domestication forces: a model

**Author(s):** Theofanopoulou, C.<br><br> It has been long thought that the kind of social interaction modern humans display should have been a key feature in the evolution of language (Tomasello et al., 2003). At the same time, our increased prosociality has independently been hypothesized to be very similar to the one we witness in domesticated species (e.g., Thomas, 2014). In this talk, I will connect the dots between several studies I have run in the past years around the topic of human self-domestication and its relevance to speech evolution. I will present genomic evidence suggesting that modern humans provide a good case for self-domestication, based on the significant convergence in signs of positive selection found in the genomes of domesticated species and modern humans (Theofanopoulou et al. 2017). I will show evidence supporting that the oxytocin-vasotocin system is a good candidate for the neurobiology underlying the vocal complexity differences we observe in domesticated vs. wild avian species (Tobari, Theofanopoulou et al. 2021), and propose how these findings can shed light on the evolution of human speech. Lastly, I will present modern human specific alleles we identified in the oxytocin and vasotocin receptors, when compared to archaic humans and non-human primates (Theofanopoulou et al. 2022) and will explain how these sites could have impacted the evolution of human prosociality via self-domestication.

Human self-domestication and speech evolution through the lens of the oxytocin and vasotocin gene family

Recent experimental and computational modelling work has found that languages are shaped by the referential context in which they operate. Wray and Grace (2007) argue that even compositionality, traditionally regarded as a universal and fundamental feature of human languages, may have only culturally evolved in response to changing social contexts. But how can the referential contexts of individual interactions come to shape the level of compositionality in the language of an entire community? To explore this question, we propose an iterated hierarchical Bayesian model that shows how partner-specific linguistic innovations can be generalized as community-wide features via a context-sensitive pathway. Our simulations show that the degree of compositionality that evolves in the language of a community depends on the communicative needs of its members, but also on the degree of user uncertainty over the nature of those needs, and on the level of heterogeneity in the community's needs.

The complexity of a language is shaped by the communicative needs of its users and by the hierarchical nature of their social inferences

As efficient systems of communication, languages are usually expected to map meanings to forms in a one-to-one way, using for example the same affix form (e.g., -s in English) every time a particular meaning is intended (e.g., plural number), and placing affixes with the same meaning consistently in the same position (e.g., always suffixal). Forms and positional rules extending over contexts with a common meaning (e.g., plural in 1PL, 2PL, 3PL) are thus considered natural, and those extending over contexts with no consistent common meaning (e.g., 1PL and 3SG) are considered unnatural. Natural patterns are most common cross-linguistically, and most learnable in experiments; however, little is yet know about differences between unnatural classes. In this study we explore syncretism (i.e., use of the same form in different functions) and affix position in the domain of person and number agreement in verbs, both cross-linguistically and in artificial language learning experiments. Results from the two approaches and both phenomena converge in finding a gradient of (un)naturalness. Rather than a dichotomous natural/unnatural distinction, we found that both cross-linguistic frequency and learnability are proportional to the amount of shared feature values among the contexts requiring the same form or position. We argue that a cognitive bias towards similarity-based structure explains our experimental results and could be driving the patterns observed in natural languages.

A Naturalness Gradient Shapes the Learnability and Cross-Linguistic Distribution of Morphological Paradigms

Typological data shows a tendency for languages to exhibit harmonic (i.e. consistent) ordering between heads and dependents. However, some categories seem to contradict this tendency. Here we investigate one such case, the order of the noun with respect to two dependents—adjectives, which tend to follow the noun and genitives which precede. We report two silent gesture experiments examining (i) whether there are cognitive biases favouring postnominal adjective and prenominal genitive order in a single trial judgement task, and (ii) if those preferences continue to influence order when participants learn a complete word order system. Our results shed light on how biases for individual categories of elements interact with biases that affect the wider linguistic system. While participants strongly prefer postnominal adjectives and prenominal genitives when these are judged in isolation, when they learn a system of ordering, these biases are obscured and (at least in some cases) harmony emerges.

The Influence of Category-specific and System-wide Preferences on Cross-Linguistic Word Order Patterns

Words in human languages cluster together in phonological neighborhoods more closely than would be expected by chance. But why? One explanation is that large neighborhoods are directly selected for, possibly because they scaffold word learning and production. But it's also possible that they emerge as a byproduct of other constraints or selection pressures operating over real lexica. We advance one such selection pressure as a candidate explanation. A pressure to avoid overloading unique wordforms with homophones may lead to clusters of words that are not identical but similar. Using simulated baselines, we test the viability of this alternative account. We find that a pressure against loading too many meanings on unique wordforms––paired with the phonotactics of a target language––produces lexica with neighborhoods that are at least as large on average as those in real lexica. This does not rule out the possibility of a pro-neighborhood pressure, but it does demonstrate the viability of a parsimonious alternative account based on a pressure against homonymy for which there is independent evidence.

Underline digital video library

Analyzing Finetuned Vision Models for Mixtec Codex Interpretation

MingOfficial: A Ming Official Career Dataset and a Historical Context-Aware Representation Learning Framework | VIDEO

PHD: Pixel-Based Language Modeling of Historical Documents

Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus

RoBoCoP: A Comprehensive ROmance BOrrowing COgnate Package and Benchmark for Multilingual Cognate Identification

Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction | VIDEO

The evolution of language under self-domestication forces: a model

Human self-domestication and speech evolution through the lens of the oxytocin and vasotocin gene family

The complexity of a language is shaped by the communicative needs of its users and by the hierarchical nature of their social inferences

A Naturalness Gradient Shapes the Learnability and Cross-Linguistic Distribution of Morphological Paradigms

The Influence of Category-specific and System-wide Preferences on Cross-Linguistic Word Order Patterns

Can a pressure against homophones explain phonological neighborhoods?

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES