Would you like to see your presentation here, made available to a global audience of researchers?
Add your own presentation or have us affordably record your next conference.
Vision-Language Models (VLMs) are widely used in tasks like Open-Vocabulary Object Detection and zero-shot Classification, owing to their powerful generalization. However, recent research reveals that VLMs exhibit significant performance instability when tasked with recognizing concepts at varying granularities (\textit{e.g.}, animal vs. dog). Prevailing methods inject external knowledge from Large Language Models, but this unconstrained approach distorts the VLM's inherent hierarchical orthogonal geometry, leading to performance collapse on general concepts. To address this, we introduce \textbf{\textit{GeCoin}}, an innovative \textbf{\textit{Ge}}ometrically \textbf{\textit{Co}}nstra\textbf{\textit{in}}ed framework that safely enhances existing VLMs with external knowledge for improved hierarchical understanding, without additional training. By projecting knowledge into the null-space of a query concept's feature space, \textit{GeCoin} mathematically guarantees the preservation of general knowledge while integrating specialized information. Extensive experiments across large-scale benchmarks, diverse VLMs (\textit{e.g.}, CLIP, SigLip 2), and knowledge from various LLMs (\textit{e.g.}, GPT-3.5, Claude-3, Gemini-Pro) show that \textbf{GeCoin} boosts performance by an average of 3.9\% over the strongest baseline—crucially eradicating performance collapse on general concepts. The code link is in the supplementary material.
