Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s). In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.
翻译:用户往往无法在单一查询中制定复杂的信息需求。 因此,他们可能需要扫描多个结果页面或重新排列其查询,这可能是一个令人沮丧的经历。 或者,系统可以通过主动询问用户的问题以澄清其信息需求来提高用户的满意度。 询问澄清问题在对话系统中特别重要,因为他们只能返回数量有限的结果( 通常只有一个) 。 在本文件中,我们制定了在开放域信息搜索对话系统中提出澄清问题的任务。 为此,我们提议了一个任务离线评价方法,并通过众包收集数据集,称为Qulac。 我们的数据集建在TREC网络轨道2009-2012的数据顶端上,包含198 TREC专题的10K问答配对,有762个方面。 我们对一个孔模型的实验显示,只要提出一个好的问题,就能在P@1上超过170 %的检索业绩改进,这清楚地表明了任务的潜在影响。 我们还提议了一个由三个组成部分组成的检索框架:问题检索、问题选择和文件检索。 Qulac。 我们的数据集是在TREWWWWWWWWE C track Translal relishing the relifer relifer relifer sual sual subal sub subal sub sub sub sub