Irregular multivariate time series (IMTS), characterized by uneven sampling and inter‑variate asynchrony, fuel many forecasting applications yet remain challenging to model efficiently. Canonical Pre‑Alignment (CPA) has been widely adopted in IMTS modeling by padding zeros at every global timestamp, thereby alleviating inter-variate asynchrony and unifying the series length, but its dense zero‑padding inflates the pre‑aligned series length, especially when numerous variates are present, causing prohibitive compute overhead. Recent graph‑based models with patching strategies sidestep CPA, but their local message passing struggles to capture global inter‑variate correlations. Therefore, we posit that CPA should be retained, with the pre‑aligned series properly handled by the model, enabling it to outperform state‑of‑the‑art graph‑based baselines that sidestep CPA. Technically, we propose KAFNet, a compact architecture grounded in CPA for IMTS forecasting that couples (1) Pre‑Convolution module for sequence smoothing and sparsity mitigation, (2) Temporal Kernel Aggregation module for learnable compression and modeling of intra-series irregularity, and (3) Frequency Linear Attention blocks for the low‑cost inter-series correlations modeling in the frequency domain. Experiments on multiple IMTS datasets show that KAFNet achieves state-of-the-art forecasting performance, with a 7.2× parameter reduction and a 8.4× training-inference acceleration. The source code can be accessed at https://github.com/zhouziyu02/KAFNet.