In customer-oriented industries, compliance-guaranteed chatbots are essential for delivering accurate, verifiable responses. Retrieval-based systems leverage a Match-and-Respond mechanism with predefined knowledge bases of human-verified question-answer (QA) pairs, providing hallucination-free responses. To effectively handle diverse customer inquiries, augmenting the knowledge base with similar questions that maintain semantic consistency and increase schematic diversity is crucial for improved query matching. In this paper, we define the Similar Question Generation (SQG) task for LLM training and inference, presenting context-aware methods for generating similar questions through a cost-efficient one-to-many paradigm. This approach facilitates broader semantic exploration and better alignment with the source question-answer pairs. We also propose combinatorial optimization techniques to construct in-context prompt demonstrations and to select an optimal subset of similar questions for knowledge base expansion. Experiments on a conversational QA-based SQG dataset demonstrate the effectiveness of these methods in both quantitative and human evaluations. A 92\% user satisfaction rate in deployed chatbot systems, reflecting an 18\% relative improvement, further highlights the practical advantages of the proposed augmentation approach in ensuring compliance and enhancing customer service chatbot applications.