Deep learning has advanced medical imaging, but limited interpretability hinders clinical adoption. Class activation maps (CAMs) provide visual explanations, yet methods such as Score-CAM are computationally expensive, requiring a forward pass for each activation map and limiting real-time applicability despite their high fidelity. To overcome this limitation, LowRank-CAM is proposed, which aggregates activation maps into a global matrix and applies singular value decomposition (SVD) to extract dominant spatial modes. The resulting top-r attention masks, with r much smaller than K, replace per-channel perturbations and require only r forward passes through the classifier head. This low-rank formulation substantially reduces complexity while preserving class-discriminatory importance. Experiments on musculoskeletal radiographs with Inception-v3 demonstrate that LowRank-CAM achieves a 4.73× speedup over Score-CAM while maintaining comparable visual clarity and diagnostic relevance.