Text REtrieval Conference (TREC) 2001¶

Web¶

TREC-2001 saw the falling into abeyance of the Large Web Task but a strengthening and broadening of activities based on the 1.69 million page WT10g corpus. There were two tasks. The topic relevance task was like traditional TREC ad hoc but used queries taken from real web search logs from which description and narrative fields of a topic description were inferred by the topic developers. There were 50 topics. In the homepage finding task queries corresponded to the name of an entity whose home page (site entry page) was included in WT10g. The challenge in this task was to return all of the homepages at the very top of the ranking. Cursory analysis suggests that once again, exploitation of link information did not help on the topic relevance task. By contrast, in the homepage finding task, the best performing run which did not make use of either link information or properties of the document's URL achieved only half of the mean reciprocal rank of the best run.

Track coordinator(s):

D. Hawking, CSIRO
N. Craswell, CSIRO

Track Web Page: https://trec.nist.gov/data/t10.web.html

Question Answering¶

Overview | Proceedings | Data | Runs | Participants

The TREC question answering track is an effort to bring the benefits of large-scale evaluation to bear on the question answering problem. In its third year, the track continued to focus on retrieving small snippets of text that contain an answer to a question. However, several new conditions were added to increase the realism, and the difficulty, of the task. In the main task, questions were no longer guaranteed to have an answer in the collection; systems returned a response of 'NIL' to indicate their belief that no answer was present. In the new list task, systems assembled a set of instances as the response for a question, requiring the ability to distinguish among instances found in multiple documents. Another new task, the context task, required systems to track discourse objects through a series of questions.

Track coordinator(s):

E. Voorhees, National Institute of Standards and Technology (NIST)

Track Web Page: https://trec.nist.gov/data/qamain.html

Cross-Language¶

Overview | Proceedings | Results | Runs | Participants

Ten groups participated in the TREC-2001 cross-language information retrieval track, which focussed on retrieving Arabic language documents based on 25 queries that were originally prepared in English. French and Arabic translations of the queries were also available. This was the first year in which a large Arabic test collection was available, so a variety of approaches were tried and a rich set of experiments performed using resources such as machine translation, parallel corpora, several approaches to stemming and/or morphology, and both pre-translation and post-translation blind relevance feedback. On average, forty percent of the relevant documents discovered by a participating team were found by no other team, a higher rate than normally observed at TREC. This raises some concern that the relevance judgment pools may be less complete than has historically been the case.

Track coordinator(s):

F.C. Gey, University of California, Berkeley
D.W. Oard, University of Maryland, College Park

Filtering¶

Overview | Proceedings | Data | Runs | Participants

Given a topic description, build a filtering profile which will select the most relevant examples from an incoming stream of documents. As the document stream is processed, the system may be provided with a binary judgement of relevance for some of the retrieved documents. This information can be used to adaptively update the filtering profile.

Track coordinator(s):

S. Robertson, Microsoft Research
I. Soboroff, National Institute of Standards and Technology (NIST)

Track Web Page: https://trec.nist.gov/data/filtering/T10filter_guide.htm

Video¶

Overview | Proceedings | Runs | Participants

New in TREC-2001 was the Video Track, the goal of which was to promote progress in content-based retrieval from digital video via open, metric-based evaluation. The trak built on publily available video provided by the Open Video Project of the University of North Carolina at Chapel Hill under Gary Marchionini, the NIST Digital Video Library, and stock shot video provided for TREC-2001 by the British Broadcasting Corporation. The track used very nice work on shot boundary evaluation done as part of the ISIS Coordinated Researh Project (AIM, 2001).

Track coordinator(s):

A.F. Smeaton, Dublin City Univ.
P. Over, National Institute of Standards and Technology (NIST)
R. Taban, National Institute of Standards and Technology (NIST)

Interactive¶

Overview | Proceedings | Data | Runs | Participants

In the TREC 2001 Interactive Track six research teams carried out observational studies which increased the realism of the searching by allowing the use of data and search systems/tools publicly accessible via the Internet. To the extent possible, searchers were allowed to choose tasks and systems/tools for accomplishing those tasks.

Track coordinator(s):

W. Hersh, Oregon Health Sciences Univ.
P. Over, National Institute of Standards and Technology (NIST)

Track Web Page: https://trec.nist.gov/data/t10i/t10i.html