Text REtrieval Conference (TREC) 2014¶

Clinical Decision Support¶

To make biomedical information more accessible and to meet the requirements for the meaningful use of electronic health records, a goal of modern clinical decision support systems is to anticipate the needs of physicians by linking electronic health records with information relevant for patient care. The Clinical Decision Support Track aims to simulate the requirements of such systems and to encourage the creation of tools and resources necessary for their implementation. The focus of the 2014 track was the retrieval of biomedical articles relevant for answering generic clinical questions about medical records. In the absence of a reusable, de-identified collection of medical records, we used short case reports, such as those published in biomedical articles, as idealized representations of actual medical records. A case report typically describes a challenging medical case, and it is often organized as a well-formed narrative summarizing the portions of a patient’s medical record that are pertinent to the case.

Track coordinator(s):

Matthew S. Simpson, U.S. National Library of Medicine
Ellen M. Voorhees, National Institute of Standards and Technology (NIST)
William Hersh, Oregon Health and Science University

Track Web Page: http://www.trec-cds.org/

Contextual Suggestion¶

Overview | Proceedings | Data | Runs | Participants

The contextual suggestion track investigates search techniques for complex information needs that are highly dependent on context and user interests. For example, imagine an information retrieval researcher with a November evening to spend in Gaithersburg, Maryland. A contextual suggestion system might recommend a beer at the Dogfish Head Alehouse, dinner at the Flaming Pit, or even a trip into Washington on the metro to see the National Mall. The primary goal of this track is to develop evaluation methodologies for such systems.

Track coordinator(s):

Adriel Dean-Hall, University of Waterloo
Charles L. A. Clarke, University of Waterloo
Jaap Kamps, University of Amsterdam
Paul Thomas, CSIRO
Ellen M. Voorhees, National Institute of Standards and Technology (NIST)

Track Web Page: https://sites.google.com/site/treccontext/

Microblog¶

This year represents the fourth iteration of the TREC Microblog track, which has been running since 2011. The track continued using the “evaluation as a service” model, in which participants had access to the document collection only through an API. In addition to the temporallyanchored ad hoc retrieval task, which has been running since the inception of the track, we introduced a new task called tweet timeline generation (TTG), where the goal is to produce concise “summaries” about a particular topic for human consumption.

Track coordinator(s):

Jimmy Lin, University of Maryland
Yulu Wang, University of Maryland
Miles Efron- University of Illinois
Garrick Sherman, University of Illinois

Track Web Page: https://github.com/lintool/twitter-tools/wiki

Web¶

The goal of the TREC Web track over the past few years has been to explore and evaluate innovative retrieval approaches over large-scale subsets of the Web – currently using ClueWeb12, on the order of one billion pages. For TREC 2014, the sixth year of the Web track, we implemented the following significant updates compared to 2013. First, the risk-sensitive retrieval task was modified to assess the ability of systems to adaptively perform risk-sensitive retrieval against multiple baselines, including an optional selfprovided baseline. In general, the risk-sensitive task explores the tradeoffs that systems can achieve between effectiveness (overall gains across queries) and robustness (minimizing the probability of significant failure, relative to a particular provided baseline). Second, we added query performance prediction as an optional aspect of the risk-sensitive task. The Adhoc task continued as for TREC 2013, evaluated using both adhoc and diversity relevance criteria.

Track coordinator(s):

Kevyn Collins-Thompson, University of Michigan
Craig Macdonald, University of Glasgow
Paul Bennett, Microsoft Research
Fernando Diaz, Microsoft Research
Ellen M. Voorhees, National Institute of Standards and Technology (NIST)

Track Web Page: https://www-personal.umich.edu/~kevynct/trec-web-2014/

Federated Web Search¶

Overview | Proceedings | Data | Runs | Participants

The TREC Federated Web Search track facilitates research on federated web search, by providing a large realistic data collection sampled from a multitude of online search engines. The FedWeb 2013 Resource Selection and Results Merging tasks are again included in FedWeb 2014, and we additionally introduced the task of vertical selection. Other new aspects are the required link between the Resource Selection and Results Merging tasks, and the importance of diversity in the merged results. After an overview of the new data collection and relevance judgments, the individual participants’ results for the tasks are introduced, analyzed, and compared.

Track coordinator(s):

Thomas Demeester, Ghent University
Dolf Trieschnigg, University of Twente
Dong Nguyen, University of Twente
Djoerd Hiemstra, University of Twente
Ke Zhou, Yahoo Labs London

Track Web Page: http://sites.google.com/site/trecfedweb

Knowledge Base Acceleration¶

Overview | Proceedings | Data | Runs | Participants

The Knowledge Base Acceleration (KBA) track ran in TREC 2012, 2013, and 2014 as an entitycentric filtering evaluation. This track evaluates systems that filter a time-ordered corpus for documents and slot fills that would change an entity profile in a predefined list of entities. Compared with the 2012 and 2013 evaluations, the 2014 evaluation introduced several refinements, including high-quality community metadata from running Raytheon/BBN’s Serif named entity recognizer, sentence parser, and relation extractor on 579,838,246 English documents in the corpus.

Track coordinator(s):

John R. Frank, Diffeo
Max Kleiman-Weiner, Diffeo
Daniel A. Roberts, Diffeo
Ellen M. Voorhees, National Institute of Standards and Technology (NIST)
Ian Soboroff, National Institute of Standards and Technology (NIST)

Track Web Page: https://trec-kba.org/

Temporal Summarization¶

Overview | Proceedings | Data | Runs | Participants

News events such as protests, accidents or natural disasters represent a unique information access problem where traditional approaches fail. For example, immediately after an event, the corpus may be sparsely populated with relevant content. Even when, after a few hours, relevant content becomes available, it is often inaccurate or highly redundant. At the same time, crisis events demonstrate a scenario where users urgently need information, especially if they are directly affected by the event. The goal of this track is to develop systems for efficiently monitoring the information associated with an event over time. Specifically, we are interested in developing systems which can broadcast short, relevant, and reliable sentencelength updates about a developing event.

Track coordinator(s):

Javed Aslam, Northeastern University
Matthew Ekstrand-Abueg, Northeastern University
Virgil Pavlu, Northeastern University
Fernando Diaz, Microsoft Research
Richard McCreadie, University of Glasgow
Tetsuya Sakai, Waseda University

Track Web Page: https://web.archive.org/web/20170618023232/http://www.trec-ts.org/

Session¶

The TREC Session track ran for the fourth time in 2014. The track has the primary goal of providing test collections and evaluation measures for studying information retrieval over user sessions rather than one-time queries. These test collections are meant to be portable, reusable, statistically powerful, and open to anyone that wishes to work on the problem of retrieval over sessions.

Track coordinator(s):

Ben Carterette, University of Delaware
Evangelos Kanoulas, Google
Mark Hall, Edge Hill University
Paul Clough, University of Sheffield

Track Web Page: http://ir.cis.udel.edu/sessions