Canada

Large Language Models (LLMs) have successfully been applied to code generation tasks, raising the question of how well these models understand programming.
Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. We show that LLMs not only fail to properly generate correct Python code when default function names are swapped, but some of them even become more confident in their incorrect predictions as the model size increases, an instance of the recently discovered phenomenon of Inverse Scaling, which runs contrary to the commonly observed trend of increasing prediction quality with increasing model size. Our findings indicate that, despite their astonishing typical-case performance, LLMs still lack a deep, abstract understanding of the content they manipulate, making them unsuitable for tasks that statistically deviate from their training data, and that mere scaling is not enough to achieve such capability.

ACL 2023

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

inverse scaling

python

code generation

semantic parsing

language model

workshop paper

### Welcome to ACL 2023, the 61st Annual Meeting of the Association for Computational Linguistics! 
 The conference will be held in Toronto, Canada, July 9-14, 2023. 
Following the succession of the recent conferences in our field, ACL 2023 will adopt a hybrid format.
While the impact of Covid has considerably diminished in terms of traveling, obtaining visas to Canada
entails a very long process. Moreover, the global economic conditions pose challenges for many individuals to travel to conferences. Recognizing these circumstances, we know many participants may not be
able to attend the conference in person. Therefore, we are committed to providing a great virtual platform
so everyone has the opportunity to interact with other participants and enjoy the conference. Based on the
current registered participants, approxiately 30% have chosen to attend the conference virtually. Whether
you join us in person or virtually, we sincerely hope everyone has a remarkable conference experience. 
This General Chair’s message is where I express my gratitude to the many individuals who have made
enormous contributions to the conference over the past year.

Read [**ACL 2023 General Chair's message**](https://docs.google.com/document/d/1WobYM7norbG4dI48s75HfJoD89qgX5a_F-6U8AteLSA/edit?usp=sharing/) in full.

##### **[Conference Handbook](https://2023.aclweb.org/downloads/acl2023-handbook.pdf)**

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing.

Taxonomy of Problems in Lexical Semantics

The task of noun compound interpretation, or NCI, aims to uncover the implicit relation between constituent nouns. One way of performing this task is via paraphrasing. For example, given the noun compound “chocolate bunny”, acceptable paraphrases would be “a bunny made of chocolate” or “a bunny shaped chocolate. We also introduce a task similar to NCI called noun compound conceptualization (NCC). The difference is that NCC involves infrequent or novel noun compounds, for which it may be more difficult to come up with a plausible meaning. Our work involves evaluating how well modern large language models perform on both NCI and NCC. We also investigate how much of the responses of large language models are copied from the web.

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives

Data-Efficient Finetuning Using Cross-Task Nearest Neighbors

In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale. The majority of recent scaling laws studies focused on high-compute high-parameter count settings, leaving the question of when these abilities begin to emerge largely unanswered. In this paper, we investigate whether the effects of pre-training can be observed when the problem size is reduced, modeling a smaller, reduced-vocabulary language. We show the benefits of pre-training with masked language modeling (MLM) objective in models as small as 1.25M parameters, and establish a strong correlation between pre-training perplexity and downstream performance (GLUE benchmark). We examine downscaling effects, extending scaling laws to models as small as 1M parameters. At this scale, we observe a break of the power law for compute-optimal models and show that the MLM loss does not scale smoothly with compute-cost (FLOPs) below 2.2e15 FLOPs. We also find that adding layers does not always benefit downstream performance.

Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale.

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Recurrent Attention Networks for Long-text Modeling

RHO ($\rho$): Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding

Accounts of human language processing have long appealed to implicit “situation models” that enrich comprehension with relevant but unstated world knowledge. Here, we apply causal intervention techniques to recent trans- former models to analyze performance on the Winograd Schema Challenge (WSC), where a single context cue shifts interpretation of an ambiguous pronoun. We identify a rela- tively small circuit of attention heads that are responsible for propagating information from the context word that guides which of the can- didate noun phrases the pronoun ultimately at- tends to. We then compare how this circuit be- haves in a closely matched “syntactic” control where the situation model is not strictly nec- essary. These analyses suggest distinct path- ways through which implicit situation models are constructed to guide pronoun resolution.

Causal interventions expose implicit situation models for commonsense language understanding

Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities. An important aspect of this process is the ability to capture historical states visually, which provides detailed information that is not covered by text and will guide subsequent steps. Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 5,652 tasks and 79,089 multimedia steps. This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps. We propose to encode visual state changes through a selective multimedia encoder to address the multimedia challenge, transfer knowledge from previously observed tasks using a retrieval-augmented decoder to overcome the induction challenge, and further present distinct information at each step by optimizing a diversity-oriented contrastive learning objective. We define metrics to evaluate both generation and inductive quality. Experiment results demonstrate that our approach significantly outperforms strong baselines.

Premium content

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Downloads

Next from ACL 2023

Taxonomy of Problems in Lexical Semantics

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES