We conjecture that hidden state vectors corresponding to individual input tokens encode information sufficient to accurately predict several tokens ahead. More concretely, in this paper we ask: Given a hidden (internal) representation of a single token at position t in an input, can we reliably anticipate the tokens that will appear at positions ≥ t + 2? To test this, we measure linear approximation and causal intervention methods in GPT-J-6B to evaluate the degree to which individual hidden states in the network contain signal rich enough to predict future hidden states and, ultimately, token outputs. We find that, at some layers, we can approximate a model's output with more than 48% accuracy with respect to its prediction of subsequent tokens through a single hidden state. Finally we present a "Future Lens" visualization that uses these methods to create a new view of transformer states.

Future Lens: Anticipating Subsequent Tokens from a Single Hidden State

**[[SIGNLL 2023 Part 2 Live Session Recording]](https://us06web.zoom.us/rec/share/Fb8BmKGoDzRS1ludb312jim4hY5ue8FRhX4Jp3IobAl9LfvhEYkHTv8OHU69N_FW.WP27Xv4DQ3y46hgv)** 
**[[SIGNLL 2023 Day 2 Live Session Recording]](https://us06web.zoom.us/rec/play/zVY0g0L3khSZ4NML4EjKw8M3TvMHEKl1LRouAtEyJl8WfTT7Qu1yavsyfW3VLuVB-70OPGzmUo2bW-51.lh2VfJJW-iHATvqw?canPlayFromShare=true&from=recording_mg&continueMode=true&componentName=rec-play&originRequestUrl=https%3A%2F%2Fus06web.zoom.us%2Frec%2Fshare%2FtW1mfqioXRyu4zINRhyliXQvFD9kPUfdDApBgQw6ytIez82oJzj2ISXXEAeK19AL.6RjVwcGUPKUowuEf)** 
**Workshop Organizers:** David Reitter and Jing Jiang
 *Coffee Break 10:30 am* 
*Lunch Break 12:30 pm* 
*Coffee Break 3:30 pm* [**Workshop Website**](https://www.conll.org/2023) 
CoNLL is a yearly conference organized by SIGNLL (ACL's Special Interest Group on Natural Language Learning), focusing on theoretically, cognitively and scientifically motivated approaches to computational linguistics.

W01: The SIGNLL Conference on Computational Natural Language Learning(CoNLL)

workshop paper

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

Koyena Pal

1

SHORT BIO

Presentations

Future Lens: Anticipating Subsequent Tokens from a Single Hidden State

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES