Multi-instance learning (MIL) has become a powerful paradigm for weakly supervised learning tasks, where instance-level annotations are unavailable or costly. While graph-based MIL methods enhance bag topological structure modeling, they often suffer from high computational costs and limited representation due to rigid graph construction and insufficient integration of bag-level semantics. To address these challenges, we propose GDF-MIL, a novel graph-driven MIL framework, which introduces a dual-path feature fusion mechanism to adaptively balance topological structure modeling and semantic feature preservation. First, the adaptive bag mapping module (ABMM) performs soft clustering to extract compact and informative representations. Subsequently, a dynamic graph structure learning (DGSL) component efficiently learns sparse topological structures via weighted connectivity, aggregating them into a comprehensive graph-level representation. Finally, to balance fast graph construction and bag-level knowledge, dual-path feature fusion (DPFF) employs a dual-path gating mechanism to integrate both types of features, which are then passed to the classification layer for bag label prediction. Extensive experiments on 24 datasets across 4 domains shown that GDF-MIL significantly outperforms 18 state-of-the-art methods on the majority of datasets.