Protein-Ligand Affinity (PLA) prediction quantifies interaction strength to guide rational drug design. Existing approaches typically analyze protein-ligand interaction at a single granularity and overlook their tightly coupled relationships in both structure and functionality, consequently yielding suboptimal representations, leading to significant performance drops in real-world scenarios. To address this problem, we propose PLA-MGRA, a minimalist and effective PLA prediction framework. Specifically, PLA-MGRA captures both fine-grained atomic details and coarse-grained functional semantics within the 3D structure of protein–ligand complexes, through multi-granularity learning. To further parse the coupled protein–ligand relationships, we design relation-aware learning for enhancing binding nature of representations. Extensive experiments demonstrate our method achieves state-of-the-art performance on multiple protein–ligand affinity prediction benchmarks, while also offering generalizability and interpretability.