NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We examine existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input. SCROLLS contains summarization, question answering, and natural language inference tasks, covering multiple domains, including literature, science, business, and entertainment. Initial baselines, including Longformer Encoder-Decoder, indicate that there is ample room for improvement on SCROLLS. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods.

SCROLLS: Standardized CompaRison Over Long Language Sequences

Building dense retrievers requires a series of standard procedures, including training and validating neural models and creating indexes for efficient search. However, these procedures are often misaligned in that training objectives do not exactly reflect the retrieval scenario at inference time. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced, focusing on dense phrase retrieval (Lee et al., 2021) where billions of representations are indexed at inference. Since validating every dense retriever with a large-scale index is practically infeasible, we propose an efficient way of validating dense retrievers using a small subset of the entire corpus. This allows us to validate various training strategies including unifying contrastive loss terms and using hard negatives for phrase retrieval, which largely reduces the training-inference discrepancy. As a result, we improve top-1 phrase retrieval accuracy by 2~3 points and top-20 passage retrieval accuracy by 2~4 points for open-domain question answering. Our work urges modeling dense retrievers with careful consideration of training and inference via efficient validation while advancing phrase retrieval as a general solution for dense retrieval.

Bridging the Training-Inference Gap for Dense Phrase Retrieval

We propose DrBoost, a dense retrieval ensemble inspired by boosting. DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current ensemble. The final representation is the concatenation of the output vectors of all the component models, making it a drop-in replacement for standard dense retrievers at test time. DrBoost enjoys several advantages compared to standard dense retrieval models. It produces representations which are 4x more compact, while delivering comparable retrieval results. It also performs surprisingly well under approximate search with coarse quantization, reducing latency and bandwidth needs by another 4x. In practice, this can make the difference between serving indices from disk versus from memory, paving the way for much cheaper deployments.

Boosted Dense Retriever

Many NLP tasks require processing long contexts beyond the length limit of pretrained models. In order to scale these models to longer text sequences, many efficient long-range attention variants have been proposed. Despite the abundance of research along this direction, it is still difficult to gauge the relative effectiveness of these models in practical use cases, e.g., if we apply these models following the pretrain-and-finetune paradigm. In this work, we aim to conduct a thorough analysis of these emerging models with large-scale and controlled experiments. For each attention variant, we pretrain large-size models using the same long-doc corpus and then finetune these models for real-world long-context tasks. Our findings reveal pitfalls of an existing widely-used long-range benchmark and show none of the tested efficient attentions can beat a simple local window attention under standard pretraining paradigms. Further analysis on local attention variants suggests that even the commonly used attention-window overlap is not necessary to achieve good downstream results --- using disjoint local attentions, we are able to build a simpler and more efficient long-doc QA model that matches the performance of Longformer with half of its pretraining compute.

Simple Local Attentions Remain Competitive for Long-Context Tasks

**Topics:** Machine Learning for NLP 
NLP Applications 
Natural Language Generation 
Resources and Evaluation 
Semantics: Lexical, Sentence level, Textual Inference and Other areas 
Syntax, Parsing and their Applications 
Syntax, Parsing and their Applications (CL) 
Theme Track 
Unsupervised and Weakly-Supervised Methods in NLP

Poster Session 18

poster

## Welcome to EMNLP 2022!
I am delighted to welcome you to EMNLP 2022! I believe this conference has been complicated beyond any precedent. Over the past year, it’s been thrilling to see the organization team approach each new puzzle with creativity and enthusiasm. We hope that those participating in Abu Dhabi as well as those joining remotely will leave the conference feeling newly inspired by the program and newly connected to our ever-growing community. Following EMNLP 2021 and major NLP conferences since, EMNLP 2022 is “hybrid,” serving both virtual and in-person participants.

Our key innovations for EMNLP 2022 include:

* EMNLP 2022 is “hybrid” in a second sense, as well: we allowed both direct and rolling review paper submissions, building on the pilot experiment of EMNLP 2021, which considered a small number of ARR submissions. 
* Familiar from NAACL but new to EMNLP, we’ve added an industry track.
* During the conference, “portals” will link virtual poster sessions to in-person conference participants during poster sessions each day.
* The first ACL-family conference in the United Arab Emirates.

 *Message from Noah A. Smith, University of Washington and Allen Institute for AI, Seattle, Washington, USA* 
***EMNLP 2022 General Chair***
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/9eec7d4a287ee18c278b08229290aa83.png)](https://drive.google.com/file/d/1OlPv6QBeo62VVTughj2jkiLeyHd1WnUt/view)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/a3db7a768409f05192210d98601edb25.png)](https://emnlp2022.rocket.chat/)

To access this site you need to register. Please register [here](https://2022.emnlp.org/registration/).

Register here

EMNLP 2022

Welcome!
EMNLP 2022 will take place in Abu Dhabi from December 7th to December 11th, 2022. And it will be held in hybrid mode, both online and offline.

[![](https://assets.underline.io/uploads/markdown_image/1/image/26c36e70b5f083b72ae632cbdcc2b5b1.png)](https://us06web.zoom.us/rec/play/A2VNLbC1xoObGgGAXdJADePjdbrI4GfpM-olGlTMO5xp0BM1iN8lgRdm-37i8MQXEvDQHfEvE0hBkjRf.pIMa7Ex7beVmMK-E?autoplay=true&startTime=1670387560000) 

[![](https://assets.underline.io/uploads/markdown_image/1/image/7e41b6370eefd82b0bf5eb67e35a574b.png)](https://us06web.zoom.us/rec/play/CIr3QA3uTAXC4k83Z3LrY1PShqTc_tZI0yVgmzY4ofVG4e2lfSZAOLDzlKcs3qbDceTe8xcfriX6f91P.AdSAW4gIQjDk-2zh?autoplay=true&startTime=1670428311000) 

**Chairs:** Gyuwan KIm, Roy Schwartz, Iryna Gurevych, Roy Schwartz, Andreas Ruckle, Zornitsa Kozareva, Sujith Ravi, Sasha Luccioni and Angela Fan **Organizers:** Nafise Sadat Moosavi, Iryna Gurevych, Angela Fan, Yufang Hou, Zornitsa Kozareva, Sujith Ravi, Sasha Luccioni, Gyuwan Kim, Roy Schwartz, and Andreas Rücklé 
**Invited speakers**: [Hinrich Schutze](https://underline.io/speakers/40293-hinrich-schutze), [Percy Liang](https://underline.io/speakers/99010-percy-liang), [Kurt Keutzer](https://underline.io/speakers/214081-kurt-keutzer) and [Song Han](https://underline.io/speakers/61119-song-han) 
 Please visit our [**website**](https://sites.google.com/view/sustainlp2022/home) for more information and detailed schedule.

**RocketChat channel for this workshop is available here:** https://emnlp2022.rocket.chat/channel/workshop-11-sustainlp

W11 (SustaiNLP) The Third Workshop on Simple and Efficient Natural Language Processing

workshop paper

4C: Information Retrieval

technical paper

## Welcome to NAACL 2022!
The Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) is taking place July 10-15, 2022 as a hybrid event, in Seattle, WA and online. We are happy to welcome all of you to conference! 
 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, keynote presentations, a plenary panel on the place of linguistics and symbolic structures in NLP, and an Industrial Track panel on Careers in NLP.
 
Posters (including Findings of NAACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop and will have an oral session and a poster session.
 
The program also features 6 Tutorials and 26 Workshops.
 
 We wish you a wonderful conference!
 
[The NAACL 2022 Organizing Committee](https://2022.naacl.org/committees/organization/)

NAACL 2022

2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics

Wenhan Xiong

5

11

SHORT BIO

Presentations

SCROLLS: Standardized CompaRison Over Long Language Sequences

Bridging the Training-Inference Gap for Dense Phrase Retrieval

Boosted Dense Retriever

Simple Local Attentions Remain Competitive for Long-Context Tasks

Bridging the Training-Inference Gap for Dense Phrase Retrieval

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES