Recent studies have suggested that neural language models learn and store a large amount of facts and commonsense knowledge from training data. The ability of language models to restore such knowledge is often evaluated via zero-shot cloze-style QA tasks. However, such evaluations rely only on prediction accuracy without punishing the systems for their mistakes, e.g., simply guessing or hallucinating likely answers. Selective prediction is a more informative evaluation framework that takes the confidence of predictions into account. Under the selective prediction setting, a model is evaluated not only by the number of correct predictions, but also by the ability to filter out dubious predictions by estimating the confidence of individual predictions. Such confidence-aware evaluation is crucial for determining whether to
trust zero-shot predictions of language models. In this paper, we apply the selective prediction setting to an existing benchmark, LAMA probe, and conduct extensive experiments with recent neural language models and different confidence functions. We empirically show that our Selective-LAMA evaluation is more robust to the effect of simple guesses than the conventional accuracy-based evaluation. Our
evaluation reveals the importance of the choice of confidence functions by showing that simply relying on token probabilities is not always the best choice. Further analysis shows that various confidence functions exhibit different preferences over predicted tokens for a given context.

Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models

Recent studies have suggested that neural language models learn and store a large amount of facts and commonsense knowledge from training data. 
The ability of language models to restore such knowledge is often evaluated via zero-shot cloze-style QA tasks. 
However, such evaluations rely only on prediction accuracy without punishing the systems for their mistakes, e.g., simply guessing or hallucinating likely answers. 
Selective prediction is a more informative evaluation framework that takes the confidence of predictions into account. 
Under the selective prediction setting, a model is evaluated not only by the number of correct predictions, but also by the ability to filter out dubious predictions by estimating the confidence of individual predictions. 
Such confidence-aware evaluation is crucial for determining whether to trust zero-shot predictions of language models. 
In this paper, we apply the selective prediction setting to an existing benchmark, LAMA probe, and conduct extensive experiments with recent neural language models and different confidence functions. 
We empirically show that our Selective-LAMA evaluation is more robust to the effect of simple guesses than the conventional accuracy-based evaluation.
Our evaluation reveals the importance of the choice of confidence functions by showing that simply relying on token probabilities is not always the best choice.
Further analysis shows that various confidence functions exhibit different preferences over predicted tokens for a given context.

**Organizers:** Mubashara Akhtar -King's College London; Rami Aly - University of Cambridge; Christos Christodoulopoulos -Amazon; Oana Cocarascu - King's College London; Zhijiang Guo - University of Cambridge; Arpit Mittal - Meta; Michael Schlichtkrull - University of Cambridge; James Thorne - KAIST AI; Andreas Vlachos - University of Cambridge For more details, please visit our [workshop website](https://fever.ai/workshop.html))

Sixth Workshop on Fact Extraction and VERification

workshop paper

### Welcome to the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL) 
Continuing its mission of expanding and involving the science community of all European countries, EACL had selected the Ukrainian community for the 16th EACL, which took place online due to the COVID pandemic. Unfortunately, the ongoing war made the organisation in Kyiv impossible. Considering the importance of physical interaction among researchers, especially after the restrictions imposed by the COVID pandemic, in addition to an online mode, the conference will be held in Dubrovnik, Croatia, from 2 to 6 of May, 2023. The original aim of strengthening the connection with the Ukrainian community will not change as our program will feature a dedicated session and a workshop to highlight work on Ukrainian language technologies. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

You need to register for the conference in order to access this site. Please visit https://2023.eacl.org/registration for more information.

EACL 2023

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

Please feel free to view the Findings papers at your convenience. You are welcome to leave your comment or a question in the Chat box.

Hiyori Yoshikawa

2

1

SHORT BIO

Presentations

Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models

Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES