Canada

We introduce the range of oBERTa language models, an easy-to-use set
of language models which allows Natural Language Processing (NLP) practitioners to obtain between 3.8 and 24.3 times faster models without expertise in model compression. Specifically, oBERTa extends existing work on pruning, knowledge distillation, and quantization and leverages frozen embeddings, improves
distillation, and model initialization to deliver higher accuracy on a broad range of transfer tasks.

ACL 2023

oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes
Daniel Campos, Alexandre Marques, Mark Kurtz and Cheng Xiang Zhai (dcampos3@illinois.edu)
Status: Accept

language-model

efficient

pruning

workshop paper

### Welcome to ACL 2023, the 61st Annual Meeting of the Association for Computational Linguistics! 
 The conference will be held in Toronto, Canada, July 9-14, 2023. 
Following the succession of the recent conferences in our field, ACL 2023 will adopt a hybrid format.
While the impact of Covid has considerably diminished in terms of traveling, obtaining visas to Canada
entails a very long process. Moreover, the global economic conditions pose challenges for many individuals to travel to conferences. Recognizing these circumstances, we know many participants may not be
able to attend the conference in person. Therefore, we are committed to providing a great virtual platform
so everyone has the opportunity to interact with other participants and enjoy the conference. Based on the
current registered participants, approxiately 30% have chosen to attend the conference virtually. Whether
you join us in person or virtually, we sincerely hope everyone has a remarkable conference experience. 
This General Chair’s message is where I express my gratitude to the many individuals who have made
enormous contributions to the conference over the past year.

Read [**ACL 2023 General Chair's message**](https://docs.google.com/document/d/1WobYM7norbG4dI48s75HfJoD89qgX5a_F-6U8AteLSA/edit?usp=sharing/) in full.

##### **[Conference Handbook](https://2023.aclweb.org/downloads/acl2023-handbook.pdf)**

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing.

We consider the problem of improving the inference latency of language model-based dense retrieval systems by introducing structural compression and model size asymmetry between the context and query encoders. First, we investigate the impact of pre and post-training compression on the MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that asymmetry in the dual-encoders in dense retrieval can lead to improved inference efficiency. Knowing this, we introduce Kullback–Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods by pruning and aligning the query encoder after training. Specifically, KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.

Quick Dense Retrievers Consume KALE: Post Training Kullback–Leibler Alignment of Embeddings for Asymmetrical dual encoders
Daniel Campos, Alessandro Magnani and Chengxiang Zhai (dcampos3@illinois.edu)
Status: Accept

We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The proposed approach relaxes a widely used technique, which shares parameters for one layer with all layers such as Universal Transformers (Dehghani et al., 2019), to increase the efficiency in the computational time. We propose three strategies: Sequence, Cycle, and Cycle (rev) to assign parameters to each layer. Experimental results show that the proposed strategies are efficient in the parameter size and computational time. Moreover, we indicate that the proposed strategies are also effective in the configuration where we use many training data such as the recent WMT competition.

Lessons on Parameter Sharing across Layers in Transformers

Sequence-to-sequence language models can be used to produce abstractive summaries which are coherent, relevant, and concise. Still, model sizes can make deployment in latency-sensitive or web-scale implementations difficult. This paper studies the relationship between model size, structured pruning, inference efficiency, and summarization accuracy on widely used summarization datasets. We show that model accuracy is tied to the encoder size while inference efficiency is connected to the decoder. Using asymmetric pruning can lead to nearly 3x improvement in inference latency with ~1 point loss in Rouge-2. Moreover, we find both the average degradation and the role of asymmetry to be consistent across model sizes and variations in datasets.

To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency
Daniel Campos and ChengXiang Zhai (dcampos3@illinois.edu)
Status: Accept

We examine the effects of model size and pre- finetuning in an active learning setting where classifiers are trained from scratch on 14 binary and 3 multi-class text classification tasks. We make an important observation that, in realistic active learning settings, where the human anno- tator and the active learning system operate in asynchronous mode, a compact pre-finetuned 1-layer transformer model with 4.2 million pa- rameters is 30% more label efficient when com- pared to the larger 24-layer 84 million param- eter transformer model. Further, in line with previous studies, we note that pre-finetuning transformer models on related tasks improves label efficiency of downstream tasks by 12%- 50%. The compact pre-finetuned model does not require GPUs, making it a viable solution for large-scale real-time inference with cheaper CPU options.

Small is the New Big: Pre-finetuned compact models are better for Asynchronous Active Learning
Dantong Liu, Kaushik Pavani and Sunny Dasgupta (lidanton@amazon.com)
Status: Accept

Fine-tuning large pre-trained models for downstream tasks can be really expensive. In the past, researchers have proposed various alternatives like adapter and prompt-based methods for tuning these large language models using minimal parameters. However, applying prompt-tuning for smaller language models has not been effective so far and not much work is done in pushing forward soft prompting for
these smaller models. To improve the training efficiency of the language models and reduce the size of tuned parameters, we propose a novel Adapter-based Efficient Prompt Tuning approach (ADEPT). In this paper, we show that tuning the parameters of soft prompts with adapter modules while keeping the rest of the model frozen can be a promising method to optimize smaller language models for downstream tasks. Our method achieves up to 98% performance of full fine-tuning while using only 0.02% of total model parameters.

ADEPT: Adapter-based Efficient Prompt Tuning Approach for Language Models
Aditya Shah, Surendrabikram Thapa, Aneesh Jain and Lifu Huang (surendrabikram@vt.edu)
Status: Accept

Finetuning large language models inflates the costs of NLU applications and remains the bottleneck of development cycles. Recent works in computer vision use data pruning to reduce training time. Pruned data selection with static methods is based on a score calculated for each training example prior to finetuning, which involves important computational overhead. Moreover, the score may not necessarily be representative of sample importance throughout the entire training duration. We propose to address these issues with a refined version of dynamic data pruning, a curriculum which periodically scores and discards unimportant examples during finetuning. Our method leverages an EL2N metric that we extend to the joint intent and slot classification task, and an initial finetuning phase on the full train set. Our results on the GLUE benchmark and four joint NLU datasets show a better time-accuracy trade-off compared to static methods. Our method preserves full accuracy while training on 50% of the data points and reduces computational times by up to 41%. If we tolerate instead a minor drop of accuracy of 1%, we can prune 80% of the training examples for a reduction in finetuning time reaching 66%.

NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Tasks
Jean-Michel Attendu and Jean-Philippe Corbeil (jeanphilippecorbeil@gmail.com)
Status: Accept

In this work, we provide an analysis on the interactions of the effectiveness of decoding with structural constraints and the amount of available training data for structured prediction tasks in NLP. Our exploration adopts a simple protocol that enforces constraints upon constraint-agnostic local models at testing time. With evaluations on three typical structured prediction tasks (named entity recognition, dependency parsing, and event argument extraction), we find that models trained with less data predict outputs with more structural violations in greedy decoding mode. Incorporating constraints provides consistent performance improvements and such benefits are larger in lower resource scenarios. Moreover, there are similar patterns with regard to the model sizes and more efficient models tend to enjoy more benefits. Finally, we also investigate settings with genre transfer and discover patterns that are related to domain discrepancies.

On the Interactions of Structural Constraints and Data Resources for Structured Prediction
Zhisong Zhang, Emma Strubell and Eduard Hovy (zerozones17@gmail.com)
Status: Accept

Even though many efficient transformers have been proposed, only few such models are available for specialized domains. Additionally, since the pretraining process is extremely costly in general -- but even more so as the sequence length increases -- it is often only in reach of large research labs. One way of making pretraining cheaper is the Replaced Token Detection (RTD) task, by providing more signal during training compared to MLM, since the loss can be computed over all tokens. In this work, we train Longformer models with the efficient RTD task on long-context legal data to showcase that pretraining efficient LMs is possible using less than 12 GPU days. We evaluate the trained models on challenging summarization tasks requiring the model to summarize complex long texts. We find that both the small and base models outperform their baselines on the in-domain BillSum and out-of-domain PubMed tasks in their respective parameter range. We publish our models as a resource for researchers and practitioners.

Can we Pretrain a SotA Legal Language Model on a Budget From Scratch?
Joel Niklaus and Daniele Giofre (joel.niklaus@unibe.ch)
Status: Accept

Conventional Transformer-based Video Ques-tion Answering (VideoQA) approaches gener-ally encode frames independently through oneor more image encoders followed by interac-tion between frames and question. However,such schema would incur significant memoryuse and inevitably slow down the training andinference speed. In this work, we present ahighly efficient approach for VideoQA basedon existing vision-language pre-trained mod-els where we concatenate video frames to an × n matrix and then convert it to one image.By doing so, we reduce the use of the imageencoder from n2 to 1 while maintaining thetemporal structure of the original video. Ex-perimental results on MSRVTT and TrafficQAshow that our proposed approach achievesstate-of-the-art performance with nearly 4×faster speed and only 30% memory use. Weshow that by integrating our approach intoVideoQA systems we can achieve compara-ble, even superior, performance with a sig-nificant speed up for training and inference.We believe the proposed approach can facil-itate VideoQA-related research by reducingthe computational requirements for those whohave limited access to budgets and resources. Our code will be made publicly available forresearch use.

Is a Video worth n × n Images? A Highly Efficient Approach to Transformer-based Video Question Answering
Chenyang Lyu, Tianbo Ji, Yvette Graham and Jennifer Foster (chenyang.lyu2@mail.dcu.ie)
Status: Accept

Scaling language models have revolutionized widespread NLP tasks, yet little comprehensively explored few-shot relation extraction with large language models. In this paper, we investigate in-context learning and data generation for few-shot relation extraction via GPT-3.5 through exhaustive experiments. To enhance few-shot performance, we further propose task-related instructions and schema-constrained data generation. We observe that in-context learning can achieve performance on par with previous prompt learning approaches, and data generation with the large language model can boost previous solutions to obtain new state-of-the-art few-shot results on four widely-studied relation extraction datasets.

How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?
Xin Xu, Yuqi Zhu, Xiaohan Wang and Ningyu Zhang (xxucs@zju.edu.cn)
Status: Accept

Premium content

oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes Daniel Campos, Alexandre Marques, Mark Kurtz and Cheng Xiang Zhai (dcampos3@illinois.edu) Status: Accept

Downloads

Next from ACL 2023

Quick Dense Retrievers Consume KALE: Post Training Kullback–Leibler Alignment of Embeddings for Asymmetrical dual encoders Daniel Campos, Alessandro Magnani and Chengxiang Zhai (dcampos3@illinois.edu) Status: Accept

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES