Standard pretrained language models operate
on sequences of subword tokens without direct access to the characters that compose each
token’s string representation. We probe the
embedding layer of pretrained language models and show that models learn the internal
character composition of whole word and subword tokens to a surprising extent, without
ever seeing the characters coupled with the tokens. Our results show that the embedding layers of RoBERTa and GPT2 each hold enough
information to accurately spell up to a third
of the vocabulary and reach high character
ngram overlap across all token types. We further test whether enriching subword models
with character information can improve language modeling, and observe that this method
has a near-identical learning curve as training without spelling-based enrichment. Overall, our results suggest that language modeling objectives incentivize the model to implicitly learn some notion of spelling, and that explicitly teaching the model how to spell does
not appear to enhance its performance on such
tasks.

Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens

* What cryptic crosswords are, and why they are an interesting problem. * A few examples of cryptic clues and how to solve them. * An overview of Cryptonite, which contains over half a million clues. * How T5-Large fairs against Cryptonite (spoiler: you have more fingers in your hands than this number)

Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

**Tracks:** 
* Interpretability and Analysis of Models for NLP
* Linguistic Theories, Cognitive Modeling and Psycholinguistics
* Machine Translation

Virtual PS1 - 3G: Group 8

poster

## Welcome to NAACL 2022!
The Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) is taking place July 10-15, 2022 as a hybrid event, in Seattle, WA and online. We are happy to welcome all of you to conference! 
 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, keynote presentations, a plenary panel on the place of linguistics and symbolic structures in NLP, and an Industrial Track panel on Careers in NLP.
 
Posters (including Findings of NAACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop and will have an oral session and a poster session.
 
The program also features 6 Tutorials and 26 Workshops.
 
 We wish you a wonderful conference!
 
[The NAACL 2022 Organizing Committee](https://2022.naacl.org/committees/organization/)

NAACL 2022

2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics

**Session Chair: **Eunsol Choi 
**In Person Volunteer:** Arman Zharmagambetov 
** Remote Volunteer:** Abhilash Nandy

5A: Question Answering 1

technical paper

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/" seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

Omer Levy

3

SHORT BIO

Presentations

Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens

Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES