Images are typically sampled on a uniform grid, despite the fact that visual information is distributed non-uniformly,leading inefficiency in processing.To reduce computation while retaining important content,recent studies have proposed predictive downsampling methods that adaptively downsample images based on predicted pixel importance.However, these methods require high-resolution processing to accurately estimate importance,which undermines their efficiency:the prediction stage itself must process the full-resolution image, consuming most of the computational budget.This high-resolution inference is necessary because each inputvaries significantly in structure and content.In this paper,we take a different approach and introduce a learn-to-downsample paradigm tailored for aligned vision recognition tasks,such as face and palmprint recognition,where input alignment ensures consistent spatial structure across images.Leveraging this property,we learn an input-agnostic downsampling template shared across all inputs.This enables significant acceleration while preserving task-specific performance,offering a more efficient and principled alternative to dynamic,prediction-based methods.Furthermore, instead of relying on implicit importance maps,we introduce a flow-based representation that directly models the spatial warping from the original image to thedownsampled version.Unlike importance map–based approaches that rely on gradients and default to uniform sampling in flat regions, our flow-based representation learns the sampling pattern directly, enabling more efficient downsampling even in textureless areas.Extensive experiments on face and palmprint recognition demonstrate that our method substantially reduces computational cost with minimal accuracy degradation, achieving a significantly better performance-efficiency trade-off than existing predictive downsampling methods.