Detecting Out-of-Distribution (OOD) graphs is a critical task for ensuring the safety and reliability of Graph Neural Networks. The main challenge for unsupervised graph-level Out-of-Distribution detection is its common reliance on purely in-distribution (ID) data. This ID-only training paradigm yields an incomplete characterization of the feature space, leading to decision boundaries that lack the robustness required to effectively separate ID from OOD samples. While incorporating synthesized outliers into the training process is a promising approach, existing generation methods are constrained by their reliance on pre-defined, non-adaptive sampling heuristics (e.g., distance or density). Such fixed strategies lack the adaptability to systematically explore the most informative OOD regions for refining the decision boundaries. To overcome this limitation, we propose a novel Policy-Guided Outlier Synthesis (PGOS) framework that replaces static heuristics with a learned and adaptive exploration policy. PGOS trains a reinforcement learning agent to autonomously navigate the low-density voids within a structured latent space, sampling representations that are maximally effective for regularizing the OOD decision boundary. These sampled points are then decoded into high-quality pseudo-OOD graphs to improve the detector's robustness. Extensive experiments demonstrate our method's strong performance, including state-of-the-art results on several graph OOD and anomaly detection benchmarks.