Lecture image placeholder

Premium content

Access to this content requires a subscription. You must be a premium user to view this content.

Need help?
Contact us
Lecture placeholder background
VIDEO DOI: https://doi.org/10.48448/bc6h-bk91

poster

ACL 2024

August 13, 2024

Bangkok, Thailand

From Sights to Insights: Towards Summarization of Multimodal Clinical Documents

keywords:

healthcare

multimodality

summarization

The advancement of Artificial Intelligence is pivotal in reshaping healthcare, enhancing diagnostic precision, and facilitating personalized treatment strategies. One major challenge for healthcare professionals is quickly navigating through long clinical documents to provide timely and effective solutions. Doctors often struggle to draw quick conclusions from these extensive documents. To address this issue and save time for healthcare professionals, an effective summarization model is essential. Most current models assume the data is only text-based. However, patients often include images of their medical conditions in clinical documents. To effectively summarize these multimodal documents, we introduce \textbf{\textit{EDI-Summ}}, an innovative Image-Guided Encoder-Decoder Model. This model uses modality-aware contextual attention on the encoder and an image cross-attention mechanism on the decoder, enhancing the BART base model to create detailed visual-guided summaries. We have tested our model extensively on three multimodal clinical benchmarks involving multimodal question and dialogue summarization tasks. Our analysis demonstrates that \textbf{\textit{EDI-Summ}} outperforms state-of-the-art large language and vision-aware models in these summarization tasks.

\textbf{Disclaimer}: The work includes vivid medical illustrations, depicting the essential aspects of the subject matter.

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved