Predicting the user’s shopping intent is a crucial task in e-commerce. In particular determining the product category, which the user wants to shop, is essential for delivering relevant search results and website navigation options. Existing query classification models are reported to have excellent predictive performance on the single-intent queries (e.g. ‘running shoes’), but there is only little research on predicting multiple-intents for a broad query (e.g. ‘running gear’). Although the training data for broad query classification can be easily obtained, the evaluation of multi-label categorization remains challenging, as the set of true labels for multi-intent queries is subjective and ambiguous.

In this work we propose an automatic method of creating the evaluation data for multi-label e- commerce query classification. We reduce the ambiguity of the annotations by blending the label assessment from three different sources: user click data, query-item relevance and LLM judgments.