Attention-based language models typically rely on the softmax function to convert attention logits into probability distributions. However, this mechanism can lead to attention entropy collapse, where attention is focused on a single token, causing training instability. In this work, we identify the high variance sensitivity of softmax as a primary cause of this collapse. We show that entropy-stable attention mechanisms, which either control or are insensitive to the variance of attention logits, can prevent entropy collapse and enable more stable training. We provide empirical evidence of this effect in both large language models (LLMs) and a small Transformer model composed solely of self-attention and support our findings with theoretical analysis. Moreover, we identify that the concentration of attention probabilities increases the probability matrix norm, leading to a gradient exploding.