The deployment of decision-making AI agents presents a critical challenge in maintaining alignment with values or guidelines while operating in complex environments. Agents trained solely to achieve their task objectives may adopt harmful behaviors, exposing a key trade-off between maximizing reward and alignment. Avoiding misalignment is particularly difficult for pre-trained agents, where retraining is costly. This is further complicated by the diverse and potentially conflicting attributes representing ethical values. To address these challenges, we propose a test-time alignment technique based on model-guided policy shaping. Our method allows precise control over individual behavioral attributes, generalizes across diverse reinforcement learning (RL) environments, and facilitates a principled trade-off between ethical alignment and reward maximization without requiring agent retraining. We evaluate our approach using the MACHIAVELLI benchmark, which comprises 134 text-based game environments and thousands of annotated scenarios involving ethical decisions. The RL agents are first trained to maximize reward in their respective games. At test time, we apply policy shaping via scenario-action attribute classifiers to ensure decision alignment with ethical attributes. We compare our approach against prior training-time methods and general-purpose agents, and study several types of ethical violations and power-seeking behavior. Our results demonstrate that test-time policy shaping provides an effective and scalable solution for mitigating unethical behavior across diverse environments and alignment attributes.