Many methods have demonstrated promising results in zero-shot anomaly detection (ZSAD) by incorporating prompt learning (PL) to fine-tune Vision-Language Models. However, the prompt learners proposed in recent studies remain relatively simple, such as static learnable textual and/or visual prompts. Relying solely on the current PL paradigm restricts the ability to generate more precise prompts, thereby hindering improved ZSAD performance, particularly in detecting nuanced anomalies. To address this limitation, this paper proposes a high-order-aware prompt learning framework, termed HPL, which facilitates the detection of unseen anomalies through prompts fortified by hypergraph. Specifically, HPL models high-order correlations among patches through a dynamically constructed hypergraph structure. Then we propose to hypergraph semantic convolution to capture potential collaborative information (generic semantic information). Meanwhile, HPL introduces a Mixture-of-Experts prompt learner (MoEPLer), where the specialized experts within MoEPLer can generate multiple distinct prompts based on the modeled high-order correlations. Then, the final elaborate and dynamic prompts can be generated by synthetically considering each expert's prompt. This enables a comprehensive understanding of potential anomalous patterns, thereby facilitating ZSAD performance. Large-scale experiments conducted on 12 datasets, spanning natural, industrial, and medical domains, demonstrate that the validity of proposed HPL. The code will be made available upon acceptance.