Heterogeneous agent reinforcement learning (HARL) enables intelligent agents to execute complex cooperative tasks by adopting agent-specific policies. Existing HARL methods often use distinct networks per agent to ensure monotonic improvement, leading to substantial computational overhead and limited scalability in resource-constrained environments. To overcome this limitation, we propose SDE-HARL to scale HARL to a large number of agents while maintaining effective inter-agent coordination. Specifically, SDE-HARL decomposes the policy network of each agent into a lightweight local network and a global network. As such, our proposed method enables efficient local computing while allowing agent-specific properties. Moreover, to achieve efficient adaptation, agents with similar roles are grouped via a role-aware mechanism and share partial parameters in their global networks, while an identity-aware mechanism is introduced to promote behavioral diversity among agents within the same group. In certain scenarios across Google Research Football and StarCraft II, SDE-HARL reaches about 90% win rate while halving inference time compared to standard architectures.