A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines.1 Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.

Visually Grounded Continual Language Learning with Selective Specialization

Large language models capture knowledge and abilities learned from their pre-training data, which is typically limited to text and code. We present approaches for *grounding* these LLMs to carry out tasks that involve visual perception and reasoning. Our methods leverage LLMs' abilities to generate (1) dialogue (2) questions and (3) code, putting these to use in a series of visual dialogue tasks where people interact with models in perceptually-complex settings.

Interacting with LLMs for Grounded Tasks

The prevalence and salience of LMs has changed the way many researchers in NLP think about our work, causing many issues for model evaluation. This presentation presents a retrospective of recent past approaches to modeling text, and discusses what has changed, and what the current evaluation challenges are. Next, it discusses common pitfalls that LM-based research falls into, highlighting several fallacies, and suggesting ways to avoid them. Finally, this presentation highlights the importance of meta-evaluation research (the evaluation of our evaluations) as one crucial component to overcoming LM evaluation challenges in the post LM-boom.

Evaluation after the LM boom: Frustrations, fallacies andthe future

Oral Presentations of Contextual Bangla Neural Stemmer: Finding Contextualized Root Word Representations for Bangla Words

Oral Presentation of CNS Model-Paper ID: 68 | VIDEO

Identifying intents from dialogue utterances forms an integral component of task-oriented dialogue systems. Intent-related tasks are typically formulated either as a classification task, where the utterances are classified into predefined categories or as a clustering task when new and previously unknown intent categories need to be discovered from these utterances. Further, the intent classification may be modeled in a multiclass (MC) or multilabel (ML) setup. While typically these tasks are modeled as separate tasks, we propose INTENDD a unified approach leveraging a shared utterance encoding backbone. INTENDD uses an entirely unsupervised contrastive learning strategy for representation learning, where pseudo labels for the unlabeled utterances are generated based on their lexical features. Additionally, we introduce a two-step post-processing setup for the classification tasks using modified adsorption. Here, first, the residuals in the training data are propagated followed by smoothing the labels both modeled in a transductive setting. Through extensive evaluations on various benchmark datasets, we find that our approach consistently outperforms competitive baselines across all three tasks. On average, INTENDD reports percentage improvements of 2.32 %, 1.26 %, and 1.52 % in their respective metrics for few-shot MC, few-shot ML, and the intent discovery tasks respectively.

IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery

Transliteration is very important in the Indian
language context due to the usage of multi-
ple scripts and the widespread use of roman-
ized inputs. However, few training and evalu-
ation sets are publicly available. We introduce
Aksharantar1, the largest publicly available
transliteration dataset for Indian languages cre-
ated by mining from monolingual and paral-
lel corpora, as well as collecting data from hu-
man annotators. The dataset contains 26 mil-
lion transliteration pairs for 21 Indic lan-
guages from 3 language families using 12
scripts. Aksharantar is 21 times larger than ex-
isting datasets and is the first publicly available
dataset for 7 languages and 1 language fam-
ily. We also introduce the Aksharantar testset
comprising 103k word pairs spanning 19 lan-
guages that enables a fine-grained analysis of
transliteration models on native origin words,
foreign words, frequent words, and rare words.
Using the training set, we trained IndicXlit,
a multilingual transliteration model that im-
proves accuracy by 15% on the Dakshina test
set, and establishes strong baselines on the Ak-
sharantar testset introduced in this work. The
models, mining scripts, transliteration guide-
lines, and datasets are available at https://
github.com/AI4Bharat/IndicXlit under
open-source licenses.

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

The presentation titled "CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text" introduces a novel framework designed to enhance the processing and understanding of procedural text. CLMSM stands out by its unique approach of utilizing metadata from procedures, integrating a Contrastive Learning strategy for global context comprehension, and employing Masked Step Modeling to capture local context nuances. This framework addresses the inherent challenge in procedural text processing – the need for a temporal understanding and tracking of entities across steps, a critical aspect that conventional pre-training methods like Masked Language Modeling (MLM) often overlook.

In this presentation, we demonstrate how CLMSM significantly outperforms various baseline models in entity tracking tasks, underscoring the importance of in-domain pre-training for accurately learning temporal and entity-based information in procedures. The results are particularly notable in scenarios requiring the tracking of entities with implicit locations, highlighting CLMSM’s advanced knowledge retention capabilities. Additionally, we present insights into CLMSM's adaptability to out-of-domain tasks, further establishing its versatility. This framework not only sets a new standard in procedural text understanding but also opens avenues for future research, particularly in extending its applicability to other domains with procedural text, potentially unveiling new challenges and opportunities.

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Inverse Text Normalization (ITN) involves rewriting the verbalised form of text from spoken transcripts to its corresponding written form. The task inherently expects challenges in identifying ITN entries due to spelling variations in words arising out of dialects, transcription errors etc. Additionally, in Tamil, word boundaries between adjacent words in a sentence often get obscured due to Punarchi, i.e. phonetic transformation of these boundaries. Being morphologically rich, the words in Tamil show a high degree of agglutination due to inflection and clitics. The combination of such factors leads to a high degree of surfaceform variations, making scalability with pure rule-based approaches difficult. Instead, we experiment with fine-tuning three pre-trained neural LMs, consisting of a seq2seq model (s2s), a non-autoregressive text editor (NAR) and a sequence tagger + rules combination (tagger). While the tagger approach works best in a fully-supervised setting, s2s performs the best (98.05 F-Score) when augmented with additional data, via bootstrapping and data augmentation (DA&B). S2S reports a cumulative percentage improvement of 20.1 %, and statistically significant gains for all our models with DA&B. Compared to a fully supervised setup, bootstrapping alone reports a percentage improvement as high as 14.12 %, even with a small seed set of 324 ITN entries.

Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language | VIDEO

Open-domain dialogue system usually requires different sources of knowledge to generate more informative and evidential responses. However, existing knowledge-grounded dialogue systems either focus on a single knowledge source or overlook the dependency between multiple sources of knowledge, which may result in generating inconsistent or even paradoxical responses. To incorporate multiple knowledge sources and dependencies between them, we propose SAFARI, a novel framework that leverages the exceptional capabilities of large language models (LLMs) in planning, understanding, and incorporating under both supervised and unsupervised settings. Specifically, SAFARI decouples the knowledge grounding into multiple sources and response generation, which allows easy extension to various knowledge sources including the possibility of not using any sources. To study the problem, we construct a personalized knowledge-grounded dialogue dataset Knowledge Behind Persona (KBP), which is the first to consider the dependency between persona and implicit knowledge. Experimental results on the KBP dataset demonstrate that the SAFARI framework can effectively produce persona-consistent and knowledge-enhanced responses.

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as \textbf{t}ask and \textbf{a}rgument \textbf{g}rounding for \textbf{E}mbodied agents (\system). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.

tagE: Enabling an Embodied Agent to Understand Human Instructions​

Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including the ones unfavorable to classification performance. To overcome these challenges, we propose integrating two strategies: token pruning and token combining. Token pruning eliminates less important tokens in the attention mechanism’s key and value as they pass through the layers. Additionally, we adopt fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token’s importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model’s performance but also reduce its computational demands. Experiments with various datasets demonstrate superior performance compared to baseline models, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in F1 score. Additionally, memory cost is reduced to 0.61x, and a speedup of 1.64x is achieved.

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Emotion arcs capture how an individual (or a population) feels over time. They are widely used in industry and research; however, there is little work on evaluating the automatically generated arcs. This is because of the difficulty of establishing the true (gold) emotion arc. Our work, for the first time, systematically and quantitatively evaluates automatically generated emotion arcs. We also compare two common ways of generating emotion arcs: Machine-Learning (ML) models and Lexicon-Only (LexO) methods. By running experiments on 18 diverse datasets in 9 languages, we show that despite being markedly poor at instance level emotion classification, LexO methods are highly accurate at generating emotion arcs when aggregating information from hundreds of instances. We also show, through experiments on six indigenous African languages, as well as Arabic, and Spanish, that automatic translations of English emotion lexicons can be used to generate high-quality emotion arcs in less-resource languages. This opens up avenues for work on emotions in languages from around the world; which is crucial for commerce, public policy, and health research in service of speakers often left behind. Code and resources: https://github.com/dteodore/EmotionArcs

Underline digital video library

Visually Grounded Continual Language Learning with Selective Specialization

Interacting with LLMs for Grounded Tasks

Evaluation after the LM boom: Frustrations, fallacies andthe future

Oral Presentation of CNS Model-Paper ID: 68 | VIDEO

IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language | VIDEO

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

tagE: Enabling an Embodied Agent to Understand Human Instructions

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Underline digital video library

Visually Grounded Continual Language Learning with Selective Specialization

Interacting with LLMs for Grounded Tasks

Evaluation after the LM boom: Frustrations, fallacies andthe future

Oral Presentation of CNS Model-Paper ID: 68 | VIDEO

IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language | VIDEO

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

tagE: Enabling an Embodied Agent to Understand Human Instructions​

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

tagE: Enabling an Embodied Agent to Understand Human Instructions