Fine-tuning large language models (LLMs) has achieved remarkable performance across various natural language processing tasks, yet it demands more and more memory as model sizes keep growing. To address this issue, the recently proposed Memory-efficient Zeroth-order (MeZO) methods attempt to fine-tune LLMs using only forward passes, thereby avoiding the need for a backpropagation graph. However, significant performance drops and a high risk of divergence have limited their widespread adoption. In this paper, we propose the Adaptive Zeroth-order Tensor-Train Adaption (AdaZeta) framework, specifically designed to improve the performance and convergence of the ZO methods. To enhance dimension-dependent ZO estimation accuracy, we introduce a fast-forward, low-parameter tensorized adapter. To tackle the frequently observed divergence issue in large-scale ZO fine-tuning tasks, we propose an adaptive query number schedule that guarantees convergence. Detailed theoretical analysis and extensive experimental results on Roberta-Large and Llama-2-7B models substantiate the efficacy of our AdaZeta framework in terms of accuracy, memory efficiency, and convergence speed.

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning

Various parameter-efficient fine-tuning (PEFT) techniques have been proposed to enable computationally efficient fine-tuning while maintaining model performance. However, existing PEFT methods are still limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). To address this challenge, we present LoRETTA, an ultra-parameter-efficient framework that significantly reduces trainable parameters through tensor-train decomposition. Specifically, we propose two methods, named LoRETTA_adp and LoRETTA_rep. The former employs tensorized adapters, offering a high-performance yet lightweight approach for the fine-tuning of LLMs. The latter emphasizes fine-tuning via weight reparameterization with a set of small tensor factors. LoRETTA achieves comparable or better performance than most widely used PEFT methods with up to $100\times$ fewer parameters on the LLaMA-2-7B models. Furthermore, empirical results demonstrate that the proposed methods exhibit remarkable anti-overfitting capability, effectively improve training efficiency, and enjoy better multi-task learning performance. Plug-and-play loretta library built upon the Huggingface framework and PEFT library are provided.

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

This talk firstly explains why we need to rethink design automation from a hybrid model-based and data-driven perspective, and why we need to extend EDA to multiple engineering domains.  Then I will introduce an uncertainty-aware data-driven design automation framework that can be applied to multiple engineering domains, and explain the shared theoretical challenges. Finally, I will highlight some of our recent theoretical results and demonstrate the applications in electronic IC, photonic IC, autonomous systems and AI systems. 

https://underline.io/speakers/59515-zheng-zhang

Uncertainty-Aware and Data-Driven Design Automation Techniques with Multi-Domain Applications

This poster session includes Main Conference posters and Findings from the following areas:

Special Theme: Efficiency in Model Algorithms, Training, and Inference  • Summarization • Sentiment Analysis, Stylistic Analysis, and Argument Mining

In-Person Poster Session C (Jasmine)

poster

## Welcome to EMNLP 2024! 
We are excited to welcome you to one of the most prominent conferences in the field of Natural Language Processing. This year, EMNLP 2024 is being held in a hybrid format,
offering both virtual and in-person participation in beautiful Miami. Due to a record-breaking number of submissions, we've expanded the total number of accepted papers to accommodate more cutting-edge research from around the globe.
### [Conference Handbook](https://drive.google.com/file/d/1WPROgxjLAC96AJL7Ugy0tEnYm7dkrbHt/view?usp=sharing)

You are required to register for this event. **Please register [here](https://2024.emnlp.org/registration/).** The EMNLP 2024 event page on Underline will be open to public one week prior to the event.

Please register!

EMNLP 2024

EMNLP 2024 will take place in Miami, Florida from Nov 12th to Nov 16th, 2024, at the Hyatt Regency Miami Hote and on Underline for remote participants.

<head>
	<style>
		.b1 {
			color: #ffffff;
			background-color: #5b1cf4;
			font-size: 19px;
			font-family: system-ui, sans-serif;
			border-radius: 15px;
			padding: 8px 40px;
			cursor: pointer
		}
		.b1:hover {
			color: #5b1cf4;
			background-color: #ffffff;
		}
	</style>
</head>
<button name="button" class="b1" onclick="window.open('https://us06web.zoom.us/rec/share/3RfXhz-SqtlXJwO5z5qU47f_FAhFqSNitV_Ausdw3M9da8eVlS_zvd9zDm12arrF.XmCx-uAGl_KI0b50')">Live Session Recording</button>

Efficient/Low-Resources and Evaluation Methods for NLP

technical paper

## Welcome to 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics!
This year, the conference is in Mexico City. NAACL was actually already planned for Mexico City in 2021, but due to the pandemic the entire conference was moved online. This year, finally, we get to go! So it is my sincere pleasure to welcome you to Mexico City, whether in person or virtually. Having the conference in Mexico City is a good opportunity to emphasize that NAACL is our flagship conference for ACL members not only in North America but also in Central and South America, even though NAACL has been bearing “North” in its name. At this year’s conference, we have a theme to match, with a theme track on the Languages of Latin America to showcase the linguistic diversity of the region.
 
The opportunity to present at NAACL should not depend on a researcher’s travel budget, or their family status. This is why it is so important to make virtual participation at NAACL as good an experience as possible – but we want to also provide a good experience for in-person participants. As a community, we are still working out the best way to do that. This year at NAACL, we are trying out a big virtual poster session ahead of the conference, with the hope that this will make make for a lively and interactive experience. At the same time, we are reducing virtual oral presentations, which seem to be particularly tricky to make to work well. A big thanks to the NAACL program chairs and to Luciana Benotti for all their ideas and work to improve the virtual experience. And participants, virtual as well as in-person: Please let us know what worked for you and what didn’t, so we can continue to improve hybrid conferences. 
 
I have been lucky to work with many amazing people. Without their insight, dedication and patience, and without the many hours of work they put in, NAACL would not have been possible. A huge thank you to the program chairs Helena Gomez, Kevin Duh, and Steve Bethard – you are the best! 
 
Finally, I would like to thank all authors, invited speakers and panelists, area chairs and reviewers, the volunteers organizing and chairing sessions, and all attendees, in-person and virtual. Thank you for helping us make NAACL 2024 come to life. 
 
Welcome and hope you all enjoy the conference! 
Katrin Erk 
The University of Texas at Austin 
NAACL-2024 General Chair 
*You can read the full Welcome message in the [Conference Handbook (downloadable)](https://drive.google.com/file/d/1H1NvW0VASQjkSCYgw3mr-3l4yYDSxz6B/view?usp=sharing)*

To access the event page you need to register [**here.**](http://acl.swoogo.com/naacl2024) 
Your access to the event page is limited based on your registration type. If you registered for workshops only, you will gain access to full workshops content on the day of the workshops program.

NAACL 2024

2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics

You are welcome and encouraged to view the posters and start asking questions in the Q&A boxes next to each of them. 

Break/Poster Presentations 1

 Electronic design and design automation (EDA) is a key enabling technology in the advancement of information processing that we are enjoying today. EDA has provided fundamental design principles and tools empowering the exponential growth of the number of transistors on an integrated circuit chip over the past 50 years. Without EDA, it would not have been possible for billion-transistor chips to be a reality! 

After five decades of growth, the field of EDA now faces some unique challenges. On one hand, we are approaching the physical limit of transistor scaling, and the golden days of exponential growth in transistor density are over. The exorbitant cost of integrated circuit fabrication and the excessive complexity of EDA tools make it difficult to bring design automation innovations to practice. On the other hand, the recent rise of machine learning and artificial intelligence applications and emerging architectures and technologies calls for more customized solutions to designing electronic systems. 

## **Objectives of the Workshop:**

This workshop, funded by the National Science Foundation (NSF), will assemble a diverse group of leading researchers from both academia and industry to identify future research directions in design and design automation of electronic systems and architectures in response to the challenges and opportunities. It will feature plenary talks and panels as well as roundtable discussions. Collectively, the workshop attendees will explore answers to overarching questions such as 

•	what are the high-risk and high-return research topics, 

•	what other fields adjacent to EDA research should be aggressively pursued for highly interdisciplinary collaboration, and

•	where the research funding should come from and how it should be distributed to encourage more transformative research. 

The workshop will also include a satellite event with a roundtable panel discussion on strategies to improve foundry access to US-based EDA researchers. A final report, based on the deliberations of the workshop, will be produced to the government recommending future research directions.


Zheng Zhang

3

SHORT BIO

Presentations

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

Uncertainty-Aware and Data-Driven Design Automation Techniques with Multi-Domain Applications

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES