Canada

Due to privacy or commercial constraints, large pre-trained language models (PLMs) are often offered as black-box APIs. Fine-tuning such models to downstream tasks is challenging because one can neither access the model’s internal representations nor propagate gradients through it. This paper addresses these challenges by developing techniques for adapting PLMs with only API access. Building on recent work on soft prompt tuning, we develop methods to tune the soft prompts without requiring gradient computation. Further, we develop extensions that in addition to not requiring gradients also do not need to access any internal representation of the PLM beyond the input embeddings. Moreover, instead of learning a single prompt, our methods learn a distribution over prompts allowing us to quantify predictive uncertainty. Ours is the first work to consider uncertainty in prompts when only having API access to the PLM. Finally, through extensive experiments, we carefully vet the proposed methods and find them competitive with (and sometimes even improving on) gradient-based approaches with full access to the PLM.

ACL 2023

Reliable Gradient-free and Likelihood-free Prompt Tuning

black-box model

gradient-free optimization

simulation-based inference

prompt tuning

ood detection

uncertainty quantification

workshop paper

### Welcome to ACL 2023, the 61st Annual Meeting of the Association for Computational Linguistics! 
 The conference will be held in Toronto, Canada, July 9-14, 2023. 
Following the succession of the recent conferences in our field, ACL 2023 will adopt a hybrid format.
While the impact of Covid has considerably diminished in terms of traveling, obtaining visas to Canada
entails a very long process. Moreover, the global economic conditions pose challenges for many individuals to travel to conferences. Recognizing these circumstances, we know many participants may not be
able to attend the conference in person. Therefore, we are committed to providing a great virtual platform
so everyone has the opportunity to interact with other participants and enjoy the conference. Based on the
current registered participants, approxiately 30% have chosen to attend the conference virtually. Whether
you join us in person or virtually, we sincerely hope everyone has a remarkable conference experience. 
This General Chair’s message is where I express my gratitude to the many individuals who have made
enormous contributions to the conference over the past year.

Read [**ACL 2023 General Chair's message**](https://docs.google.com/document/d/1WobYM7norbG4dI48s75HfJoD89qgX5a_F-6U8AteLSA/edit?usp=sharing/) in full.

##### **[Conference Handbook](https://2023.aclweb.org/downloads/acl2023-handbook.pdf)**

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing.

Large Language Models (LLMs) have successfully been applied to code generation tasks, raising the question of how well these models understand programming.
Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. We show that LLMs not only fail to properly generate correct Python code when default function names are swapped, but some of them even become more confident in their incorrect predictions as the model size increases, an instance of the recently discovered phenomenon of Inverse Scaling, which runs contrary to the commonly observed trend of increasing prediction quality with increasing model size. Our findings indicate that, despite their astonishing typical-case performance, LLMs still lack a deep, abstract understanding of the content they manipulate, making them unsuitable for tasks that statistically deviate from their training data, and that mere scaling is not enough to achieve such capability.

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Taxonomy of Problems in Lexical Semantics

The task of noun compound interpretation, or NCI, aims to uncover the implicit relation between constituent nouns. One way of performing this task is via paraphrasing. For example, given the noun compound “chocolate bunny”, acceptable paraphrases would be “a bunny made of chocolate” or “a bunny shaped chocolate. We also introduce a task similar to NCI called noun compound conceptualization (NCC). The difference is that NCC involves infrequent or novel noun compounds, for which it may be more difficult to come up with a plausible meaning. Our work involves evaluating how well modern large language models perform on both NCI and NCC. We also investigate how much of the responses of large language models are copied from the web.

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives

Data-Efficient Finetuning Using Cross-Task Nearest Neighbors

In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale. The majority of recent scaling laws studies focused on high-compute high-parameter count settings, leaving the question of when these abilities begin to emerge largely unanswered. In this paper, we investigate whether the effects of pre-training can be observed when the problem size is reduced, modeling a smaller, reduced-vocabulary language. We show the benefits of pre-training with masked language modeling (MLM) objective in models as small as 1.25M parameters, and establish a strong correlation between pre-training perplexity and downstream performance (GLUE benchmark). We examine downscaling effects, extending scaling laws to models as small as 1M parameters. At this scale, we observe a break of the power law for compute-optimal models and show that the MLM loss does not scale smoothly with compute-cost (FLOPs) below 2.2e15 FLOPs. We also find that adding layers does not always benefit downstream performance.

Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale.

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Recurrent Attention Networks for Long-text Modeling

RHO ($\rho$): Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding

Accounts of human language processing have long appealed to implicit “situation models” that enrich comprehension with relevant but unstated world knowledge. Here, we apply causal intervention techniques to recent trans- former models to analyze performance on the Winograd Schema Challenge (WSC), where a single context cue shifts interpretation of an ambiguous pronoun. We identify a rela- tively small circuit of attention heads that are responsible for propagating information from the context word that guides which of the can- didate noun phrases the pronoun ultimately at- tends to. We then compare how this circuit be- haves in a closely matched “syntactic” control where the situation model is not strictly nec- essary. These analyses suggest distinct path- ways through which implicit situation models are constructed to guide pronoun resolution.

Premium content

Reliable Gradient-free and Likelihood-free Prompt Tuning

Next from ACL 2023

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES