
Premium content
Access to this content requires a subscription. You must be a premium user to view this content.

Would you like to see your presentation here, made available to a global audience of researchers?
Add your own presentation or have us affordably record your next conference.
Large vision-language models (LVLMs) have shown remarkable capabilities across a wide range of multimodal tasks. However, they remain prone to visual hallucination (VH), often producing confident but incorrect descriptions of visual content. In this work, we present VisFlow, an efficient and training-free framework designed to mitigate VH by directly manipulating attention patterns during inference. To address two main hallucination causes: insufficient visual attention and language prior dominance, through systematic analysis, we identify three key undesirable attention behaviors in LVLMs: (1) misallocated attention to visual tokens, often focused on uninformative or trailing regions; (2) over-reliance on the previous token, with several heads mainly focus on the previously generated token; (3) over-attention to system prompt, where many heads assign excessive weight to system prompt tokens, hindering multimodal integration. To address these issues, we propose two effective interventions: Token-level Attention Intervention (TAI) to enhance attention to salient visual content, and Head-level Attention Intervention (HAI) to suppress over-attention to system prompt tokens and adjacent text tokens, thereby enhancing visual alignment and mitigating the model’s over-reliance on linguistic priors encoded in textual inputs. Comprehensive evaluations on multiple models and benchmarks show that VisFlow effectively mitigates hallucinations, while introducing negligible computational cost.
