The rapid advancement of foundation models (FMs) across language, image, audio, and video domains has shown remarkable capabilities in diverse tasks. However, the proliferation of FMs brings forth a critical challenge: the potential to generate hallucinated outputs, particularly in high-stakes applications. The tendency of foundation models to produce hallucinated content arguably represents the biggest hindrance to their widespread adoption in real-world scenarios, especially in domains where reliability and accuracy are paramount. This survey paper presents a comprehensive overview of recent developments that aim to identify and mitigate the problem of hallucination in FMs, spanning text, image, video, and audio modalities. By synthesizing recent advancements in detecting and mitigating hallucination across various modalities, the paper aims to provide valuable insights for researchers, developers, and practitioners. Essentially, it establishes a clear framework encompassing definition, taxonomy, and detection strategies for addressing hallucination in multimodal foundation models, laying the foundation for future research and development in this pivotal area.

A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models

As generative AI progresses, collaboration be-
tween doctors and AI scientists is leading to the
development of personalized models to stream-
line healthcare tasks and improve productivity.
Summarizing doctor-patient dialogues has be-
come important, helping doctors understand
conversations faster and improving patient care.
While previous research has mostly focused on
text data, incorporating visual cues from pa-
tient interactions allows doctors to gain deeper
insights into medical conditions. Most of this
research has centered on English datasets, but
real-world conversations often mix languages
for better communication. To address the lack
of resources for multimodal summarization of
code-mixed dialogues in healthcare, we devel-
oped the MCDH dataset. Additionally, we cre-
ated HealthAlignSumm, a new model that in-
tegrates visual components with the BART ar-
chitecture. This represents a key advancement
in multimodal fusion, applied within both the
encoder and decoder of the BART model. Our
work is the first to use alignment techniques,
including state-of-the-art algorithms like Direct
Preference Optimization, on encoder-decoder
models with synthetic datasets for multimodal
summarization. Through extensive experi-
ments, we demonstrated the superior perfor-
mance of HealthAlignSumm across several
metrics validated by both automated assess-
ments and human evaluations. The dataset
MCDH and our proposed model HealthAlign-
Summ will be available in this GitHub account
https://github.com/AkashGhosh/Health
AlignSumm-Utilizing-Alignment-for-M
ultimodal-Summarization-of-Code-Mix
ed-Healthcare-Dialogues
Disclaimer: This work involves medical im-
agery based on the subject matter of the topic.

HealthAlignSumm : Utilizing Alignment for Multimodal Summarization of Code-Mixed Healthcare Dialogues

The advancement of Artificial Intelligence is pivotal in reshaping healthcare, enhancing diagnostic precision, and facilitating personalized treatment strategies. One major challenge for healthcare professionals is quickly navigating through long clinical documents to provide timely and effective solutions. Doctors often struggle to draw quick conclusions from these extensive documents. To address this issue and save time for healthcare professionals, an effective summarization model is essential. Most current models assume the data is only text-based. However, patients often include images of their medical conditions in clinical documents. To effectively summarize these multimodal documents, we introduce \textbf{\textit{EDI-Summ}}, an innovative Image-Guided Encoder-Decoder Model. This model uses modality-aware contextual attention on the encoder and an image cross-attention mechanism on the decoder, enhancing the BART base model to create detailed visual-guided summaries. We have tested our model extensively on three multimodal clinical benchmarks involving multimodal question and dialogue summarization tasks. Our analysis demonstrates that \textbf{\textit{EDI-Summ}} outperforms state-of-the-art large language and vision-aware models in these summarization tasks. 



\textbf{Disclaimer}: The work includes vivid medical illustrations, depicting the essential aspects of the subject matter.

From Sights to Insights: Towards Summarization of Multimodal Clinical Documents

In the era of modern healthcare, swiftly generating medical question summaries is crucial for informed and timely patient care. Despite the increasing complexity and volume of medical data, existing studies have focused solely on text-based summarization, neglecting the integration of visual information. Recognizing the untapped potential of combining textual queries with visual representations of medical conditions, we introduce the Multimodal Medical Question Summarization (MMQS) Dataset. This dataset, a major contribution of our work, pairs medical queries with visual aids, facilitating a richer and more nuanced understanding of patient needs. We also propose a framework, utilizing the power of Contrastive Language Image Pretraining(CLIP) and Large Language Models(LLMs), consisting of four modules that identify medical disorders, generate relevant context, filter medical concepts, and craft visually aware summaries. Our comprehensive framework harnesses the power of CLIP, a multimodal foundation model, and various general-purpose LLMs, comprising four main modules: the medical disorder identification module, the relevant context generation module, the context filtration module for distilling relevant medical concepts and knowledge, and finally, a general-purpose LLM to generate visually aware medical question summaries. Leveraging our MMQS dataset, we showcase how visual cues from images enhance the generation of medically nuanced summaries. This multimodal approach not only enhances the decision-making process in healthcare but also fosters a more nuanced understanding of patient queries, laying the groundwork for future research in personalized and responsive medical care.

CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare | VIDEO

Findings Papers for Asynchronous Viewing

poster

## Welcome to EMNLP 2024! 
We are excited to welcome you to one of the most prominent conferences in the field of Natural Language Processing. This year, EMNLP 2024 is being held in a hybrid format,
offering both virtual and in-person participation in beautiful Miami. Due to a record-breaking number of submissions, we've expanded the total number of accepted papers to accommodate more cutting-edge research from around the globe.
### [Conference Handbook](https://drive.google.com/file/d/1WPROgxjLAC96AJL7Ugy0tEnYm7dkrbHt/view?usp=sharing)

You are required to register for this event. **Please register [here](https://2024.emnlp.org/registration/).** The EMNLP 2024 event page on Underline will be open to public one week prior to the event.

Please register!

EMNLP 2024

EMNLP 2024 will take place in Miami, Florida from Nov 12th to Nov 16th, 2024, at the Hyatt Regency Miami Hote and on Underline for remote participants.

This poster session includes Main Conference posters and Findings from the following areas:

Special Theme: Efficiency in Model Algorithms, Training, and Inference  • Summarization • Sentiment Analysis, Stylistic Analysis, and Argument Mining

In-Person Poster Session C (Jasmine)

This poster session includes Main Conference posters, TACL and Demos.

In-Person Poster Session 4

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

ACL 2024

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

AI for Social Impact

technical paper

We are pleased to announce the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24), which will be held in Vancouver, British Columbia at the Vancouver Convention Centre – West Building from 20-27 February, 2024.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-24 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

We expect for AAAI-24 to be an in-person conference – one author of all accepted papers will be expected to present work in person unless there are exceptional circumstances that prevent this.

In order to access the AAAI-24 event page you need to register [here](https://aaai.org/aaai-conference/registration/)

AAAI 2024

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. 

Akash Ghosh

4

27

SHORT BIO

Presentations

A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models

HealthAlignSumm : Utilizing Alignment for Multimodal Summarization of Code-Mixed Healthcare Dialogues

From Sights to Insights: Towards Summarization of Multimodal Clinical Documents

CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare | VIDEO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES