For the past two decades, sustainability and carbon reduction have emerged as critical factors for data center (DC) design and operation. A set of advances has been encapsulated in green DCs with the onsite generation of renewable energy and efficient cooling systems. In this paper, we study how to apply Deep Reinforcement Learning (DRL) to optimize green DC operation. Green DC management is typically an infinite-horizon problem with exogenous stochastic input processes. We propose EA, a framework that applies DRL to the typical infinite-horizon problem without discounting. EA approximates the infinite-horizon problem with a finite-horizon one. In this approach, it is important to avoid actions optimized for the end of the finite-horizon problem but inappropriate for the true infinite-horizon one. EA addresses this challenge by combining a stationary policy with the fact that green DC management has repeating patterns (e.g., daily temperature, solar energy generation, and workload). We apply EA to the management of a green DC with onsite solar energy generation and a hybrid cooling system that includes free'' cooling. Evaluation results show that EA successfully learns important principles such as delaying deferrable jobs to solar-rich times and gracefully maintaining inside temperature. Further, EA outperforms three state-of-the-art DRL algorithms, realizing the greatest benefits on days with high outside temperature and high solar generation. While we evaluate EA in the specific context of a green DC, we believe that EA is a promising approach for more general system management settings.