Dominican Republic

Consistency of a model — that is, the invariance of its behavior under meaning preserving alternations in its input — is a highly desirable property in natural language processing. In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge? To this end, we create PARAREL , a high-quality resource of cloze-style query English paraphrases. It contains a total of 328 paraphrases for 38 relations. Using PARAREL , we show that the consistency of all PLMs we experiment with is poor – though with high variance between relations. Our analysis of the representational spaces of PLMs suggests that they have a poor structure and are currently not suitable for representing knowledge robustly. Finally, we propose a method for improving model consistency and experimentally demonstrate its effectiveness.

EMNLP 2021

Measuring and Improving Consistency in Pretrained Language Models

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

technical paper

Certified Robustness to Programmable Transformations in LSTMs

ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models

How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction

Detecting Local Insights from Global Labels: Supervised & Zero-Shot Sequence Labeling via a Convolutional Decomposition

All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

Do Long-Range Language Models Actually Use Long-Range Context?

FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging

Towards Syntax-Aware DialogueSummarization using Multi-task Learning

Detoxifying Language Models with Proximal Policy Optimization

Towards Personalized Descriptions of Scientific Concepts

poster

workshop paper

https://d3smihljt9218e.cloudfront.net/transcript/26652/subtitle/8a60110e02239a26937aabe2cc6bcad5.vtt

English (automatic)

Measuring and Improving Consistency in Pretrained Language Models

Downloads

Next from EMNLP 2021

Certified Robustness to Programmable Transformations in LSTMs

Similar lecture

STANKER: Stacking Network based on Level-grained Attention-masked BERT for Rumor Detection on Social Media

VIDEO LIBRARY

EVENTS

COMPANY

RESOURCES