Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples as entailing when the hypothesis is attested in training data, and that entities are used as "indices" to access the memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hypothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which do not conform to these biases than those which do, and we offer these as valuable controls for future LLM evaluation.

Sources of Hallucination by Large Language Models on Inference Tasks

We compare MR-based and LM-based approaches in QA tasks and propose a novel methodology that leverages directional predicate entailment (inference) to address their limitations. Additionally, we present a Boolean QA task, demonstrating that
entailment graphs (EGs) exhibit comparable directional inference capabilities to large language models (LLMs). Our results highlight the importance of inference in open-domain QA and the improvements brought by leveraging EGs.

Complementary Roles of Inference and Language Models in QA

**Are you attending this poster session virtually?** 
 Virtual posters are presented in Gather Town. Connect by clicking on the link that you can find in the 'Abstracts' tab of the posters.
 
**Are you attending this poster session in person?** 
Virtual posters are presented in Gather Town. Connect by clicking on the link that you can find in the 'Abstracts' tab of the posters.

PS5 (GatherTown) Posters, Demo, Industry, Findings

poster

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

**Workshop Organizers:** Mihai Surdeanu, Laura Chiticariu, Dayne Freitag, Gus Hahn-Powell, Clayton Morrison, Enrique Noriega, Ellen Riloff, Becky Sharp and Marco A. Valenzuela Escárcega
 *Coffee Break 10:30 am* 
*Lunch Break 12:30 pm* 
*Coffee Break 3:30 pm* [**Workshop Website**](https://pan-dl.github.io/2023/about) This workshop will focus on all aspects of pattern-based approaches, including their application, representation, and interpretability, as well as their strengths and weaknesses relative to state-of-the-art machine learning approaches. It will also explore ways of combining the strengths of pattern-based, deep learning and other statistical methods. We wish the workshop to be the first step in building a community of researchers from different areas of NLP, both applied and theoretical, who are interested in pattern-based approaches and who use them in their work (e.g., industry practitioners and domain experts). As such, beyond the standard keynote talks and user-contributed presentations, we plan for the workshop to include a panel discussion, breakout sessions on different sub-topics, and a meeting to discuss a potential shared task for future events.
 [**Go to Gather**](https://app.gather.town/app/ROH1myZvp9ba7a3C/EMNLP2023?spawnToken=IXHhoRVZTdGROtO5t26n)

Liang Cheng

2

SHORT BIO

Presentations

Sources of Hallucination by Large Language Models on Inference Tasks

Complementary Roles of Inference and Language Models in QA

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES