Dynamic graph learning focuses on representing time-varying graphs, enabling the modeling of evolving relationships between nodes. This approach is essential for applications such as traffic systems, social networks, and recommendation engines, where interactions shift dynamically. While existing methods often utilize temporal modules and transformer networks to capture these changes, a major challenge lies in the high computational demands of self-attention mechanisms, which scale quadratically with the number of nodes. To address this, we propose a novel transformer-based framework for dynamic graph learning that incorporates a more efficient token mixer. Our key insight is that the Transformer's performance primarily stems from its architecture rather than the self-attention mechanism itself. Thus, we introduce an adaptive token mixer, which aggregates tokens based on their order and timing within a sliding window. Furthermore, we design a hierarchical learning module to capture long-term dependencies by leveraging long-range neighbor information across layers. Our approach significantly reduces computational complexity while preserving the ability to model both short-term and long-term dependencies in dynamic graphs effectively. Experimental results demonstrate that our framework achieves robust performance, showing that the simplified architectures can deliver competitive results without the resource-intensive requirements of traditional Transformers.