Recent work has shown that large language models (LLMs) are capable of generating summaries zero-shot---i.e., without explicit supervision---that, under human assessment, are often comparable or even preferred to manually composed reference summaries. 
However, this prior work has focussed almost exclusively on evaluating news article summarization. How do zero-shot summarizers perform in other (potentially more specialized) domains?
In this work we evaluate zero-shot generated summaries across specialized domains including: biomedical articles, and legal bills (in addition to standard news benchmarks for reference). We focus especially on the factuality of outputs. 
We acquire annotations from domain experts to identify inconsistencies in summaries and systematically categorize these errors. We analyze whether the prevalence of a given domain in the pretraining corpus affects extractiveness and faithfulness of generated summaries of articles in this domain. 
We release all collected annotations to facilitate additional research toward measuring and realizing factually accurate summarization, beyond news articles (The dataset can be downloaded from 
https://anonymous.4open.science/r/zero_shot_faceval_domains-9B83)

Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains

While the NLP community has produced numerous summarization benchmarks,
none provide the rich annotations required 
to simultaneously address many important problems related 
to control and reliability.
We introduce a Wikipedia-derived benchmark,
complemented by a rich set of crowd-sourced annotations,
that supports $8$ interrelated tasks:
(i) extractive summarization;
(ii) abstractive summarization;
(iii) topic-based summarization;
(iv) compressing selected sentences into a one-line summary;
(v) surfacing evidence for a summary sentence;
(vi) predicting the factual accuracy of a summary sentence;
(vii) identifying unsubstantiated spans in a summary sentence;
(viii) correcting factual errors in summaries.
We compare various methods on this benchmark and discover
that on multiple tasks, moderately-sized fine-tuned models 
consistently outperform much larger few-shot prompted language models.
For factuality-related tasks, we also evaluate existing heuristics 
to create training data and find that training on them 
results in worse performance than training on $20\times$ less human-labeled data.
Our articles draw from 6 domains,
facilitating cross-domain analysis.
On some tasks, the amount of training data 
matters more than the domain where it comes from,
while for other tasks training specifically on data from the target domain, 
even if limited, is more beneficial.

USB: A Unified Summarization Benchmark Across Tasks and Domains

Virtual Poster Session A

poster

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

Please register!

EACL 2024

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

**Are you attending this poster session virtually?** 
 Virtual posters are presented in Gather Town. Connect by clicking on the link that you can find in the 'Abstracts' tab of the posters.
 
**Are you attending this poster session in person?** 
Virtual posters are presented in Gather Town. Connect by clicking on the link that you can find in the 'Abstracts' tab of the posters.

PS7 (GatherTown) Posters, Demo, Industry, Findings

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

**Workshop Organizers:** Dieuwke Hupkes, Verna Dankers, Koustuv Sinha, Amirhossein Kazemnejad, Christos Christodoulopoulos, Ryan Cotterell and Elia Bruni
 *Coffee Break 10:30 am* 
*Lunch Break 12:30 pm* 
*Coffee Break 3:30 pm* [**Workshop Website**](https://genbench.org/workshop/) The ability to generalise is one of the primary desiderata of natural language processing (NLP). Yet, how "good generalisation" should be defined and evaluated is not well understood, nor are there common standards to evaluate generalisation. As a consequence, newly proposed models are usually not systematically tested for their ability to generalise. GenBench's mission is to make state-of-the-art generalisation testing the new status-quo in NLP. As a first step, we present a generalisation taxonomy, describing the underlying building blocks of generalisation in NLP. We use the taxonomy to do an elaborate review of over 400 generalisation papers, and we make recommendations for promising areas for the future. 
[**Keynote 1, by Anna Rogers**](https://genbench.org/workshop_programme/#0915-1000-am--keynote-1-by-anna-rogers) [**Keynote 2, by Adina Williams**](https://genbench.org/workshop_programme/#1115-1200-pm--keynote-2-by-adina-williams) [**Keynote 3, by Tatsunori Hashimoto**](https://genbench.org/workshop_programme/#1400-1445-pm--keynote-3-by-tatsunori-hashimoto) 
[**Schedule**](https://genbench.org/workshop_programme/)

 [**Go to Gather**](https://app.gather.town/app/ROH1myZvp9ba7a3C/EMNLP2023?spawnToken=_5ieSSlxSvuKGu7FtBcT)

Sanjana Ramprasad

3

Presentations

Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains

USB: A Unified Summarization Benchmark Across Tasks and Domains

USB: A Unified Summarization Benchmark Across Tasks and Domains

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES