Overview - LiveQA 2016

Proceedings | Data | Runs | Participants

The LiveQA track, now in its second year, is focused on real-time question answering for real-user questions. During the test period, real user questions are drawn from those newly submitted on a popular community question answering site, Yahoo Answers (YA), that have not yet been answered. These questions are sent to the participating systems, who provide an answer in real time. Returned answers are judged by the NIST assessors on a 4-level Likert scale. The most challenging aspects of this task are that the questions can be on any one of many popular topics, are informally stated, and are often complex and at least partly subjective. Furthermore, the participant systems must return an answer in under 60 seconds, which places additional, and realistic, constraints on the kind of processing that a system can do. In addition to the main real-time question answering task, this year we introduced a pilot task aimed at identifying the question intent. As human questions submitted on forums and CQA sites are verbose in nature and contain many redundant or unnecessary terms, participants were challenged to identify the significant parts of the question. The main theme of the question is marked by the systems by specifying a list of spans that capture its main intent. This automatic “summary” of the question was evaluated by measuring its ROUGEand METEOR-based similarity to a succinct rephrase of the question, manually provided by NIST assessors.

Track coordinator(s):

• Eugene Agichtein, Emory University
• David Carmel, Yahoo Research
• Dan Pelleg, Yahoo Research
• Yuval Pinter, Yahoo Research
• Donna Harman, National Institute of Standards and Technology (NIST)

Track Web Page: https://web.archive.org/web/20160220151945/https://sites.google.com/site/trecliveqa2016/

---