Test of Time Awards

Presidential Address

Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. To overcome these limitations, we propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. CREATOR disentangles abstract tool creation and concrete decision execution, resulting in improved performance. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, CREATOR outperforms existing chain-of-thought, program-of-thought, and tool-using baselines. Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability. Further research demonstrates that leveraging LLMs as tool creators facilitates knowledge transfer, and LLMs exhibit varying levels of tool creation abilities, enabling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM's problem-solving paradigm, driving us closer to the next frontier of artificial intelligence.

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

One of the frequent points in the mainstream narrative about large language models is that they have “emergent properties” (sometimes even dangerous enough to be considered existential risk to mankind). However, there is much disagreement about even the very definition of such properties. If they are understood as a kind of generalization beyond training data - as something that a model does without being explicitly trained for it - I argue that we have not in fact established the existence of any such properties, and at the moment we do not even have the methodology for doing so.

A sanity check on emergent properties

Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including the ones unfavorable to classification performance. To overcome these challenges, we propose integrating two strategies: token pruning and token combining. Token pruning eliminates less important tokens in the attention mechanism’s key and value as they pass through the layers. Additionally, we adopt fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token’s importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model’s performance but also reduce its computational demands. Experiments with various datasets demonstrate superior performance compared to baseline models, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in F1 score. Additionally, memory cost is reduced to 0.61x, and a speedup of 1.64x is achieved.

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Many real-world tasks involve a mixed-initiative setup, wherein humans and AI systems collaboratively perform a task. While significant work has been conducted towards enabling humans to specify, through language, exactly how an agent should complete a task (i.e., low-level specification), prior work lacks on interpreting the high-level strategic intent of the human commanders. Parsing strategic intent from language will allow autonomous systems to independently operate according to the user's plan without frequent guidance or instruction. In this paper, we build a computational interface capable of translating unstructured language strategies into actionable intent in the form of goals and constraints. Leveraging a game environment, we collect a dataset of over 1000 examples, mapping language strategies to the corresponding goals and constraints, and show that our model, trained on this dataset, significantly outperforms human interpreters in inferring strategic intent (i.e., goals and constraints) from language (p < 0.05). Furthermore, we show that our model (125M parameters) significantly outperforms ChatGPT for this task (p < 0.05) in a low-data setting.

A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting | VIDEO

Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as \textbf{t}ask and \textbf{a}rgument \textbf{g}rounding for \textbf{E}mbodied agents (\system). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.

tagE: Enabling an Embodied Agent to Understand Human Instructions​

Transliteration is very important in the Indian
language context due to the usage of multi-
ple scripts and the widespread use of roman-
ized inputs. However, few training and evalu-
ation sets are publicly available. We introduce
Aksharantar1, the largest publicly available
transliteration dataset for Indian languages cre-
ated by mining from monolingual and paral-
lel corpora, as well as collecting data from hu-
man annotators. The dataset contains 26 mil-
lion transliteration pairs for 21 Indic lan-
guages from 3 language families using 12
scripts. Aksharantar is 21 times larger than ex-
isting datasets and is the first publicly available
dataset for 7 languages and 1 language fam-
ily. We also introduce the Aksharantar testset
comprising 103k word pairs spanning 19 lan-
guages that enables a fine-grained analysis of
transliteration models on native origin words,
foreign words, frequent words, and rare words.
Using the training set, we trained IndicXlit,
a multilingual transliteration model that im-
proves accuracy by 15% on the Dakshina test
set, and establishes strong baselines on the Ak-
sharantar testset introduced in this work. The
models, mining scripts, transliteration guide-
lines, and datasets are available at https://
github.com/AI4Bharat/IndicXlit under
open-source licenses.

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

Large language models (LLMs) have shown increasing capacity at planning and executing a high-level goal in a live computer environment (e.g. MiniWoB++). To perform a task, recent works often require a model to learn from trace examples of the task via either supervised learning or few/many-shot prompting. Without these trace examples, it remains a challenge how an agent can autonomously learn and improve its control on a computer, which limits the ability of an agent to perform a new task. We approach this problem with a zero-shot agent that requires no given expert traces. Our agent plans for executable actions on a partially observed environment, and iteratively progresses a task by identifying and learning from its mistakes via self-reflection and structured thought management. On the easy tasks of MiniWoB++, we show that our zero-shot agent often outperforms recent SoTAs, with more efficient reasoning. For tasks with more complexity, our reflective agent performs on par with prior best models, even though previous works had the advantages of accessing expert traces or additional screen information.

A Zero-Shot Language Agent for Computer Control with Structured Reflection

Natural language is an instrument of interaction, where interlocutors produce and comprehend language to relay information to accomplish their intents. Both opportunities and challenges arise from the interactive nature of language. The response of participants to the language they comprehend can form a strong learning signal: did I achieve my intent? In the first part, I will define a simple, and general framework to rapidly learn from interaction feedback, and show how to use this signal to learn to produce natural language instructions. I will then discuss the problem of language-conditioned reinforcement learning, where benchmark development has been hindered because computing rewards requires resolving language semantics. I will describe a new approach to address this challenge.

Natural Language Learning via Interaction

We present 1-PAGER the first system that answers a question and retrieves evidence using a single Transformer-based model and decoding process. 1-PAGER incrementally partitions the retrieval corpus using constrained decoding to select a document and answer string, and we show that this is competitive with comparable retrieve-and-read alternatives according to both retrieval and answer accuracy metrics. 1-PAGER also outperforms the equivalent ‘closed-book’ question answering model, by grounding predictions in an evidence corpus. While 1-PAGER is not yet on-par with more expensive systems that read many more documents before generating an answer, we argue that it provides an important step toward attributed generation by folding retrieval into the sequence-to-sequence paradigm that is currently dominant in NLP. We also show that the search paths used to partition the corpus are easy to read and understand, paving a way forward for interpretable neural retrieval.

1-Pager: One Pass Answer Generation and Evidence Retrieval

The presentation titled "CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text" introduces a novel framework designed to enhance the processing and understanding of procedural text. CLMSM stands out by its unique approach of utilizing metadata from procedures, integrating a Contrastive Learning strategy for global context comprehension, and employing Masked Step Modeling to capture local context nuances. This framework addresses the inherent challenge in procedural text processing – the need for a temporal understanding and tracking of entities across steps, a critical aspect that conventional pre-training methods like Masked Language Modeling (MLM) often overlook.

In this presentation, we demonstrate how CLMSM significantly outperforms various baseline models in entity tracking tasks, underscoring the importance of in-domain pre-training for accurately learning temporal and entity-based information in procedures. The results are particularly notable in scenarios requiring the tracking of entities with implicit locations, highlighting CLMSM’s advanced knowledge retention capabilities. Additionally, we present insights into CLMSM's adaptability to out-of-domain tasks, further establishing its versatility. This framework not only sets a new standard in procedural text understanding but also opens avenues for future research, particularly in extending its applicability to other domains with procedural text, potentially unveiling new challenges and opportunities.

Underline digital video library

Test of Time Awards

Presidential Address

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

A sanity check on emergent properties

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting | VIDEO

tagE: Enabling an Embodied Agent to Understand Human Instructions

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

A Zero-Shot Language Agent for Computer Control with Structured Reflection

Natural Language Learning via Interaction

1-Pager: One Pass Answer Generation and Evidence Retrieval

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Underline digital video library

Test of Time Awards

Presidential Address

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

A sanity check on emergent properties

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting | VIDEO

tagE: Enabling an Embodied Agent to Understand Human Instructions​

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

A Zero-Shot Language Agent for Computer Control with Structured Reflection

Natural Language Learning via Interaction

1-Pager: One Pass Answer Generation and Evidence Retrieval

CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

tagE: Enabling an Embodied Agent to Understand Human Instructions