We combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. Sparse Mixer slightly outperforms BERT on GLUE and SuperGLUE, but more importantly trains 65% faster and runs inference 61% faster. We also present a faster variant, prosaically named Fast Sparse Mixer, that marginally underperforms BERT on SuperGLUE, but trains and runs nearly twice as fast. We justify the design of these two models by carefully ablating through various mixing mechanisms, MoE configurations, and hyperparameters. Sparse Mixer overcomes many of the latency and stability concerns of MoE models and offers the prospect of serving sparse student models, without resorting to distilling them to dense variants.

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT

We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that "mix" input tokens. These linear mixers, along with standard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text classification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths, our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs).

FNet: Mixing Tokens with Fourier Transforms

[![](https://assets.underline.io/uploads/markdown_image/1/image/26c36e70b5f083b72ae632cbdcc2b5b1.png)](https://us06web.zoom.us/rec/play/A2VNLbC1xoObGgGAXdJADePjdbrI4GfpM-olGlTMO5xp0BM1iN8lgRdm-37i8MQXEvDQHfEvE0hBkjRf.pIMa7Ex7beVmMK-E?autoplay=true&startTime=1670387560000) 

[![](https://assets.underline.io/uploads/markdown_image/1/image/7e41b6370eefd82b0bf5eb67e35a574b.png)](https://us06web.zoom.us/rec/play/CIr3QA3uTAXC4k83Z3LrY1PShqTc_tZI0yVgmzY4ofVG4e2lfSZAOLDzlKcs3qbDceTe8xcfriX6f91P.AdSAW4gIQjDk-2zh?autoplay=true&startTime=1670428311000) 

**Chairs:** Gyuwan KIm, Roy Schwartz, Iryna Gurevych, Roy Schwartz, Andreas Ruckle, Zornitsa Kozareva, Sujith Ravi, Sasha Luccioni and Angela Fan **Organizers:** Nafise Sadat Moosavi, Iryna Gurevych, Angela Fan, Yufang Hou, Zornitsa Kozareva, Sujith Ravi, Sasha Luccioni, Gyuwan Kim, Roy Schwartz, and Andreas Rücklé 
**Invited speakers**: [Hinrich Schutze](https://underline.io/speakers/40293-hinrich-schutze), [Percy Liang](https://underline.io/speakers/99010-percy-liang), [Kurt Keutzer](https://underline.io/speakers/214081-kurt-keutzer) and [Song Han](https://underline.io/speakers/61119-song-han) 
 Please visit our [**website**](https://sites.google.com/view/sustainlp2022/home) for more information and detailed schedule.

**RocketChat channel for this workshop is available here:** https://emnlp2022.rocket.chat/channel/workshop-11-sustainlp

W11 (SustaiNLP) The Third Workshop on Simple and Efficient Natural Language Processing

workshop paper

## Welcome to EMNLP 2022!
I am delighted to welcome you to EMNLP 2022! I believe this conference has been complicated beyond any precedent. Over the past year, it’s been thrilling to see the organization team approach each new puzzle with creativity and enthusiasm. We hope that those participating in Abu Dhabi as well as those joining remotely will leave the conference feeling newly inspired by the program and newly connected to our ever-growing community. Following EMNLP 2021 and major NLP conferences since, EMNLP 2022 is “hybrid,” serving both virtual and in-person participants.

Our key innovations for EMNLP 2022 include:

* EMNLP 2022 is “hybrid” in a second sense, as well: we allowed both direct and rolling review paper submissions, building on the pilot experiment of EMNLP 2021, which considered a small number of ARR submissions. 
* Familiar from NAACL but new to EMNLP, we’ve added an industry track.
* During the conference, “portals” will link virtual poster sessions to in-person conference participants during poster sessions each day.
* The first ACL-family conference in the United Arab Emirates.

 *Message from Noah A. Smith, University of Washington and Allen Institute for AI, Seattle, Washington, USA* 
***EMNLP 2022 General Chair***
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/9eec7d4a287ee18c278b08229290aa83.png)](https://drive.google.com/file/d/1OlPv6QBeo62VVTughj2jkiLeyHd1WnUt/view)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/a3db7a768409f05192210d98601edb25.png)](https://emnlp2022.rocket.chat/)

To access this site you need to register. Please register [here](https://2022.emnlp.org/registration/).

Register here

EMNLP 2022

Welcome!
EMNLP 2022 will take place in Abu Dhabi from December 7th to December 11th, 2022. And it will be held in hybrid mode, both online and offline.

1D: Efficient Methods in NLP 1

technical paper

## Welcome to NAACL 2022!
The Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) is taking place July 10-15, 2022 as a hybrid event, in Seattle, WA and online. We are happy to welcome all of you to conference! 
 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, keynote presentations, a plenary panel on the place of linguistics and symbolic structures in NLP, and an Industrial Track panel on Careers in NLP.
 
Posters (including Findings of NAACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop and will have an oral session and a poster session.
 
The program also features 6 Tutorials and 26 Workshops.
 
 We wish you a wonderful conference!
 
[The NAACL 2022 Organizing Committee](https://2022.naacl.org/committees/organization/)

NAACL 2022

2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics

James Lee-Thorp

3

18

SHORT BIO

Presentations

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT

FNet: Mixing Tokens with Fourier Transforms

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES