FROMANNUAL REVIEWS

EMNLP 2026

•

October 26, 2026

•

Budapest, Hungary

While retrieval-augmented generation (RAG) relies on retrieved evidence to mitigate hallucination, its reliability is constrained by retrieval quality and diagnosing where retrieval fails remains difficult. Existing benchmarks typically report a single aggregate score over heterogeneous query sets and are often dominated by queries whose relevant passages can be identified through simple lexical overlap, concealing the failure modes that matter most in practice. We introduce GutenBench, a fine-grained, zero-shot evaluation dataset built around a four-level retrieval taxonomy: lexical, semantic, explicit aggregation, and latent conceptual retrieval. It pairs queries with relevance judgments adjudicated by subject-matter experts, enabling precise and controlled comparisons of retrievers. We evaluate several off-the-shelf retrievers in an asymmetric, English-to-target cross-lingual setting across 272 Bible translations (141 languages) and find that performance is strong on lexical and semantic queries, but degrades when models must aggregate information across passages or recognize conceptually relevant passages with no surface overlap.

Downloads

Paper

Next from EMNLP 2026

On the Value of Open Research in Language Modeling - An Adversarial Dialog
keynote

On the Value of Open Research in Language Modeling - An Adversarial Dialog

EMNLP 2026

Graham Neubig
Graham Neubig

26 October 2026

Similar lecture

Poisoning Retrieval Corpora by Injecting Adversarial Passages
poster

Poisoning Retrieval Corpora by Injecting Adversarial Passages

EMNLP 2023

Zexuan Zhong
+1
Zexuan Zhong and 3 other authors

10 December 2023

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)

PRESENTATIONS

  • All Presentations
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2026 Underline - All rights reserved

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.