The proliferation of multimodal fake news across various domains presents a significant challenge to information ecosystems. While existing multi-domain fake news detection methods attempt to leverage data from multiple domains to improve generalization, they often suffer from a critical drawback: negative transfer. This problem arises from their common practice of indiscriminately aggregating information, failing to account for the complex, often asymmetric, relationships between domains. Consequently, knowledge from irrelevant or conflicting domains can severely degrade detection performance.

To address this challenge, we propose a novel framework named \textbf{PANDA: Prototype-driven Asymmetric Neighbor-Domain Adaptation}. PANDA is the first framework, to our knowledge, that explicitly models the directional transferability between news domains to mitigate negative transfer. At its core, PANDA learns a set of compact and representative \textit{prototypes} for each domain to encapsulate its core characteristics. Based on these prototypes, we devise a novel \textbf{P}rototype-based \textbf{A}symmetric \textbf{D}istance (PAD) metric to quantify the potential benefit of transferring knowledge from a source domain to a target one. Guided by this metric, a \textbf{G}umbel-based \textbf{N}eighbor \textbf{S}elector (GNS) dynamically identifies the most beneficial neighbor domains for each instance. Finally, a \textbf{D}omain-\textbf{C}ollaborative \textbf{A}ttention (DCA) module adaptively fuses knowledge from the selected domains. Extensive experiments on benchmark datasets demonstrate that PANDA significantly outperforms state-of-the-art methods and effectively mitigates negative transfer, showcasing its superior adaptability and robustness in real-world scenarios.