Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as \textbf{t}ask and \textbf{a}rgument \textbf{g}rounding for \textbf{E}mbodied agents (\system). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.

tagE: Enabling an Embodied Agent to Understand Human Instructions​

A key contribution to being a successful researcher in natural language processing, as in any area, is having a clear overarching vision of what your body of research is trying to accomplish. Using my own 40-year career as an example, I will attempt to provide general advice on formulating and pursuing a coherent research vision. In particular, I will focus on formulating a unique, personal objective that exploits your specific talents, knowledge, and passions, and that is distinct from the current popular trends in the field. I will also focus on formulating a vision that bridges existing fields of study to produce an overarching agenda that unifies previously disparate ideas.

The Vision Thing: Finding and Pursuing your Research Passion

Transliteration is very important in the Indian
language context due to the usage of multi-
ple scripts and the widespread use of roman-
ized inputs. However, few training and evalu-
ation sets are publicly available. We introduce
Aksharantar1, the largest publicly available
transliteration dataset for Indian languages cre-
ated by mining from monolingual and paral-
lel corpora, as well as collecting data from hu-
man annotators. The dataset contains 26 mil-
lion transliteration pairs for 21 Indic lan-
guages from 3 language families using 12
scripts. Aksharantar is 21 times larger than ex-
isting datasets and is the first publicly available
dataset for 7 languages and 1 language fam-
ily. We also introduce the Aksharantar testset
comprising 103k word pairs spanning 19 lan-
guages that enables a fine-grained analysis of
transliteration models on native origin words,
foreign words, frequent words, and rare words.
Using the training set, we trained IndicXlit,
a multilingual transliteration model that im-
proves accuracy by 15% on the Dakshina test
set, and establishes strong baselines on the Ak-
sharantar testset introduced in this work. The
models, mining scripts, transliteration guide-
lines, and datasets are available at https://
github.com/AI4Bharat/IndicXlit under
open-source licenses.

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

**Abstract:**

Social learning is a key driver of the success of the human species (Henrich, 2015), enabling the preservation of technologies that individuals could not possibly have acquired on their own (Tennie, Call, & Tomasello, 2009). Presently, we are witnessing the rise of intelligent machines – “AI systems” – that are capable of making discoveries beyond the human horizon (Wang et al., 2023). This raises the question of whether social learning enables humans to also adopt and preserve inconceivable discoveries made by intelligent machines. In our behavioral experiment, participants’ success in a strategic task hinged on adopting a solution that humans struggle to discover due to a characteristic bias to avoid large losses. We contrast populations of humans transmitting solutions over multiple generations on their own with populations that had the opportunity to learn from AI agents in their first generation. Our findings show that an algorithmic model succeeds in discovering the adaptive strategy where humans fail on their own. Most importantly, humans are able to adopt this machine-introduced strategy and preserve it over multiple generations. Taken together, our results showcase the potential of intelligent machines to transform human cultural evolution, and an opportunity for designing them towards benefitting human progress.

**Speaker's social media:**

Twitter: @ThomasF_Mueller
Bluesky: @thomasfmueller.bsky.social

LIVE - Humans can learn and culturally preserve inconceivable strategies discovered by intelligent machines

**Abstract:**

This project looks at whether cooperation and storytelling have co-evolved. Debates continue over the extent to which widespread cooperation in humans can be explained by the mechanisms that maintain it in other species, such as kin selection and reciprocity. Storytelling may help reinforce large-scale cooperation by transmitting prosocial norms and acting as a social ‘flight simulator’. Previous evidence from transmission chain experiments suggests a bias towards social information in narratives, but the co-evolution of storytelling and cooperation remains understudied.  

This project looked at how prosocial behaviours were viewed within folktales and if prosocial moral messages were more widespread in folktales. To determine this, 452 people read 300 Indo-European folktales and commented on the moral messaging and moral behaviours found within them. It was found that cooperative behaviours such as altruism, reciprocity, helping family were viewed significantly more positively while individualistic behaviours were found to be viewed significantly more negatively. Significantly more stories were rated as having a prosocial main moral message than an individualistic one. This project aims to undertake future projects using phylogenetic comparative methods and experimental research, investigating whether prosociality in stories affects transmission fidelity and cooperative behaviour itself.


**Speaker's social media:**

em_ilyjxx 
emilymljeffries.bsky.social

The Moral of the Story - Investigating the co-evolution of storytelling and Human Cooperation via a survey of moral attitudes to Indo-European folktales

**Abstract:**

Language provides humans a remarkably flexible and abstract way to transmit information. Here, we tested the hypothesis that language allows cultural evolution to improve our representation – or model – of the world. To do so, we investigated the role of language in cultural evolution from a reinforcement learning perspective that proposes that adaptative behavior is based on learning from rewards and punishments. Using a combination of experiments, computational models, and large language models, we aimed to uncover the processes through which language fuels cultural evolution. We implemented a sequential decision-making task, requiring a mental model of the task structure within an online transmission chain design (n = 1000, 10 generations). Trough an evolutionary selection algorithm, best performing participant were sampled and provided written advice about their model of the task for the next generation. Consistent with the idea that language is important for cultural evolution, we observe an increase in performance and model knowledge over generations. Analysis using large-language models reveal an intergeneration improvement in the model knowledge communicated via written advice. Overall, these results suggest that language allows CE to grow better representations of the world by successive combination and refinement of individual world models.


**Speaker's social media:**

Twitter: @BlubluDoScience

LIVE - Language allows cultural evolution to improve world models

As the media landscape evolves, the study of news framing, key to shaping public opinion, has expanded. Our study presents a methodology for framing analysis in Spanish media using NLP to uncover the cultural and political factors shaping narratives. This method enhances linguistic inclusivity in framing studies and highlights Spanish media's role in public opinion, showing NLP's utility in diverse linguistic contexts and contributing to a more comprehensive global media analysis.

Cross-Linguistic Framing Analysis: Unveiling Political and Cultural Narratives in Spanish-Language News

Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations, but is expensive, even if a sampling-based approximation is used. Besides requiring a large number of sampled sequences, it requires the pairwise calculation of a utility metric, which has quadratic complexity. In this paper, we propose to approximate pairwise metric scores with scores calculated against aggregated reference representations. This changes the complexity of utility estimation from $O(n^2)$ to $O(n)$, while empirically preserving most of the quality gains of MBR decoding. We release our source code.

Linear-time Minimum Bayes Risk Decoding with Reference Aggregation

Recent watermarked generation algorithms inject detectable signatures during language generation to facilitate post-hoc detection. While token-level watermarks are vulnerable to paraphrase attacks, SemStamp (Hou et al., 2023) applies watermark on the semantic representation of sentences and demonstrates promising robustness. SemStamp employs locality-sensitive hashing (LSH) to partition the semantic space with arbitrary hyperplanes, which results in a suboptimal tradeoff between robustness and speed. We propose k-SemStamp, a simple yet effective enhancement of SemStamp, utilizing k-means clustering as an alternative of LSH to partition the embedding space with awareness of inherent semantic structure. Experimental results indicate that k-SemStamp saliently improves its robustness and sampling efficiency while preserving the generation quality, advancing a more effective tool for machine-generated text detection.

k-SemStamp: A Clustering-Based Semantic Watermark for Detection of Machine-Generated Text

The principle of continual relation extraction (CRE) involves adapting to emerging novel relations while preserving old knowledge. Existing CRE approaches excel in preserving old knowledge but falter when confronted with contaminated data streams, likely due to an artificial assumption of no annotation errors. Recognizing the prevalence of noisy labels in real-world datasets, we introduce a more practical learning scenario, termed as \textit{noisy-CRE}. In response to this challenge, we propose a noise-resistant contrastive framework called Noise-guided Attack in Contrastive Learning (NaCL), aimed at learning incremental corrupted relations. Diverging from conventional approaches like sample discarding or relabeling in the presence of noisy labels, NaCL takes a transformative route by modifying the feature space through targeted attack. This attack aims to align the feature space with the provided, albeit inaccurate, labels, thereby enhancing contrastive representations. Extensive empirical validations demonstrate the consistent performance improvement of NaCL with increasing noise rates, surpassing state-of-the-art methods.

Enhancing Contrastive Learning with Noise-Guided Attack: Towards Continual Relation Extraction in the Wild

Hateful memes detection is a challenging multimodal understanding task that requires comprehensive learning of vision, language, and cross-modal interactions. Previous research has focused on developing effective fusion strategies for integrating hate information from different modalities. However, these methods excessively rely on cross-modal fusion features, ignoring the modality uncertainty caused by the contribution degree of each modality to hate sentiment and the modality imbalance caused by the dominant modality suppressing the optimization of another modality. To this end, this paper proposes an Uncertainty-guided Modal Rebalance (UMR) framework for hateful memes detection. The uncertainty of each meme is explicitly formulated by designing stochastic representation drawn from a Gaussian distribution for aggregating cross-modal features with unimodal features adaptively. The modality imbalance is alleviated by improving cosine loss from the perspectives of inter-modal feature and weight vectors constraints. In this way, the suppressed unimodal representation ability in multimodal models would be unleashed, while the learning of modality contribution would be further promoted. Extensive experimental results demonstrate that the proposed UMR produces the state-of-the-art performance on four widely-used datasets.

Uncertainty-Guided Modal Rebalance for Hateful Memes Detection

Ensuring that online discussions are civil and productive is a major challenge for social media platforms. Such platforms usually rely both on users and on automated detection tools to flag inappropriate arguments of other users, which moderators then review. However, this kind of post-hoc moderation is expensive and time-consuming, and moderators are often overwhelmed by the amount and severity of flagged content. Instead, a promising alternative is to prevent negative behavior during content creation. This paper studies how inappropriate language in arguments can be computationally mitigated. We propose a reinforcement learning-based rewriting approach that balances content preservation and appropriateness based on existing classifiers, prompting an instruction-finetuned large language model (LLM) as our initial policy. Unlike related style transfer tasks, rewriting inappropriate arguments allows deleting and adding content permanently. It is therefore tackled on document level rather than sentence level. We evaluate different weighting schemes for the reward function in both absolute and relative human assessment studies. Systematic experiments on non-parallel data provide evidence that our approach can mitigate the inappropriateness of arguments while largely preserving their content. It significantly outperforms competitive baselines, including few-shot learning, prompting, and humans.

Underline digital video library

tagE: Enabling an Embodied Agent to Understand Human Instructions

The Vision Thing: Finding and Pursuing your Research Passion

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

LIVE - Humans can learn and culturally preserve inconceivable strategies discovered by intelligent machines

The Moral of the Story - Investigating the co-evolution of storytelling and Human Cooperation via a survey of moral attitudes to Indo-European folktales

LIVE - Language allows cultural evolution to improve world models

Cross-Linguistic Framing Analysis: Unveiling Political and Cultural Narratives in Spanish-Language News

Linear-time Minimum Bayes Risk Decoding with Reference Aggregation

k-SemStamp: A Clustering-Based Semantic Watermark for Detection of Machine-Generated Text

Enhancing Contrastive Learning with Noise-Guided Attack: Towards Continual Relation Extraction in the Wild

Uncertainty-Guided Modal Rebalance for Hateful Memes Detection

LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Underline digital video library

tagE: Enabling an Embodied Agent to Understand Human Instructions​

The Vision Thing: Finding and Pursuing your Research Passion

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

LIVE - Humans can learn and culturally preserve inconceivable strategies discovered by intelligent machines

The Moral of the Story - Investigating the co-evolution of storytelling and Human Cooperation via a survey of moral attitudes to Indo-European folktales

LIVE - Language allows cultural evolution to improve world models

Cross-Linguistic Framing Analysis: Unveiling Political and Cultural Narratives in Spanish-Language News

Linear-time Minimum Bayes Risk Decoding with Reference Aggregation

k-SemStamp: A Clustering-Based Semantic Watermark for Detection of Machine-Generated Text

Enhancing Contrastive Learning with Noise-Guided Attack: Towards Continual Relation Extraction in the Wild

Uncertainty-Guided Modal Rebalance for Hateful Memes Detection

LLM-based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

tagE: Enabling an Embodied Agent to Understand Human Instructions