Text REtrieval Conference (TREC) 2024¶

Adhoc Video Search¶

Overview | Proceedings | Data | Runs | Participants

The Ad-hoc search task goal is to model the end user search use-case, who is searching (using textual sentence queries) for segments of video containing persons, objects, activities, locations, etc. and combinations of the former. While the Internet Archive (IACC.3) dataset was adopted between 2016 to 2018, from 2019 to 2021 a new data collection (V3C1) based on Vimeo Creative Commons (V3C) datset was adopted. Starting in 2022 the task started to utilize a new sub-collection V3C2 to test systems on a new set of queries in addition to common (fixed) progress query set to measure system progress from 2022 to 2024.

Track coordinator(s):

Georges Quenot, University of Grenoble
George Awad, NIST

Track Web Page: https://www-nlpir.nist.gov/projects/tv2024/avs.html

AToMiC¶

Overview | Data | Results | Runs | Participants

The Authoring Tools for Multimedia Content (AToMiC) Track aims to build reliable benchmarks for multimedia search systems. The focus of this track is to develop and evaluate IR techniques for text-to-image and image-to-text search problems.

Track coordinator(s):

Jheng-Hong (Matt) Yang, University of Waterloo
Jimmy Lin, University of Waterloo
Carlos Lassance, Naver Labs Europe
Rafael S. Rezende, Naver Labs Europe
Stéphane Clinchant, Naver Labs Europe
Krishna Srinivasan, Google Research
Miriam Redi, Wikimedia Foundation

Track Web Page: https://trec-atomic.github.io/

Biomedical Generative Retrieval (BioGen) Track¶

Overview | Proceedings | Data | Runs | Participants

Large language models (LLMs) adapted for the biomedical domain show exceptional performance on many tasks, but are also known to provide false information, i.e., hallucinations or confabulations. Inaccuracies may be particularly harmful in high-risk situations, such as making clinical decisions or appraising biomedical research. The TREC 2024 BioGen task will focus on reference attribution as a means to mitigate generation of false statements by LLMs. The goal of the TREC 2024 BioGen task will be to cite references to support the text of the sentences and the overall answer from LLM output for each topic. Each run will be scored by the proportion of sentences and overall answer that have correctly supporting attributions.

Track coordinator(s):

Bill Hersh, Oregon Health & Science University
Dina Demner-Fushman, National Library of Medicine
Deepak Gupta, National Library of Medicine
Steven Bedrick, Oregon Health & Science University
Kirk Roberts, University of Texas Houston

Track Web Page: https://dmice.ohsu.edu/trec-biogen/task.html

Interactive Knowledge Assistance¶

iKAT is the successor to the Conversational Assistance Track (CAsT). The fourth year of CAST aimed to add more conversational elements to the interaction streams, by introducing mixed initiatives (clarifications, and suggestions) to create multi-path, multi-turn conversations for each topic. TREC iKAT evolves CAsT into a new track to signal this new trajectory. iKAT aims to focus on supporting multi-path, multi-turn, multi-perspective conversations. That is for a given topic, the direction and the conversation that evolves depends not only on the prior responses but also on the user.

Track coordinator(s):

Mohammed Aliannejadi, University of Amsterdam
Zahra Abbasiantaeb, University of Amsterdam
Simon Lupart, University of Amsterdam
Shubham Chatterjee, University of Glasgow
Jeff Dalton, University of Glasgow
Leif Azzopardi, University of Strathclyde

Track Web Page: https://trecikat.com/

Lateral Reading¶

The TREC Lateral Reading Track is for researchers interested in addressing the problems of misinformation and trust in search and online content. The current web landscape requires the ability to make judgments about the trustworthiness of information, which is a difficult task for most people. Meanwhile, automated detection of misinformation is likely to remain limited to well-defined domains or be limited to simple fact-checking.

Track coordinator(s):

Dake Zhang, University of Waterloo
Mark Smucker, University of Waterloo
Charles L. A. Clarke, University of Waterloo

Medical Video Question Answering¶

Overview | Proceedings | Data | Runs | Participants

The recent surge in the availability of online videos has changed the way of acquiring information and knowledge. Many people prefer instructional videos to teach or learn how to accomplish a particular task in an effective and efficient manner with a series of step-by-step procedures. Similarly, medical instructional videos are more suitable and beneficial for delivering key information through visual and verbal communication to consumers' healthcare questions that demand instruction. We aim to extract the visual information from the video corpus for consumers' first aid, medical emergency, and medical educational questions. Extracting the relevant information from the video corpus requires relevant video retrieval, moment localization, video summarization, and captioning skills. Toward this, the TREC task, Medical Video Question Answering, focuses on developing systems capable of understanding medical videos and providing visual answers (from single and multiple videos) and instructional step captions to answer natural language questions. Emphasizing the importance of multimodal capabilities, the task requires systems to generate instructional questions and captions based on medical video content. Following the MedVidQA 2023, TREC 2024 expanded the tasks considering language-video understanding and generation. This track is comprised of two main tasks: Video Corpus Visual Answer Localization (VCVAL) and Query-Focused Instructional Step Captioning (QFISC).

Track coordinator(s):

Deepak Gupta, National Library of Medicine, NIH
Dina Demner-Fushman, National Library of Medicine, NIH

NeuCLIR¶

Overview | Data | Results | Runs | Participants

Cross-language Information Retrieval (CLIR) has been studied at TREC and subsequent evaluation forums for more than twenty years, but recent advances in the application of deep learning to information retrieval (IR) warrant a new, large-scale effort that will enable exploration of classical and modern IR techniques for this task.

Track coordinator(s):

Dawn Lawrie, Johns Hopkins University
Sean MacAvaney, University of Glasgow
James Mayfield, Johns Hopkins University
Paul McNamee, Johns Hopkins University
Douglas W. Oard, University of Maryland
Luca Soldaini, Allen Institute for AI
Eugene Yang, Johns Hopkins University

Track Web Page: https://neuclir.github.io/

Plain-Language Adaptation of Biomedical Abstracts¶

Overview | Proceedings | Data | Runs | Participants

The goal of the PLABA track is to improve health literacy by adapting biomedical abstracts for the general public using plain language. When adapting, source sentences may be split, in which case the output for one source sentence will be multiple target sentences. However, source sentences may not be merged, and the output for a given source sentence should not contain information from other source sentences. Both source and output will be in English. An example of adaptation is below.

Track coordinator(s):

Brian Ondov, Yale School of Medicine
Bill Xia, U.S. National Library of Medicine
Ishita Unde, U.S. National Library of Medicine
Dina Demner-Fushman, U.S. National Library of Medicine
Hoa T. Dang, NIST

Product Search¶

The product search track focuses on IR tasks in the world of product search and discovery. This track seeks to understand what methods work best for product search, improve evaluation methodology, and provide a reusable dataset which allows easy benchmarking in a public forum.

Track coordinator(s):

Daniel Campos, University of Illinois at Urbana-Champaign
Corby Rosset, Microsoft
Surya Kallumadi, Lowes
ChengXiang Zhai, University of Illinois at Urbana-Champaign
Alessandro Magnani, Walmart

Track Web Page: https://trec-product-search.github.io/

Retrieval-Augmented Generation¶

The (TREC) Retrieval-Augmented Generation Track is intended to foster innovation and research within the field of retrieval-augmented generation systems. This area of research focuses on combining retrieval methods - techniques for finding relevant information within large corpora with Large Language Models (LLMs) to enhance the ability of systems to produce relevant, accurate, updated and contextually appropriate content.

Track coordinator(s):

Ronak Pradeep, University of Waterloo
Nandan Thakur, University of Waterloo
Jimmy Lin, University of Waterloo
Nick Craswell, Microsoft

Track Web Page: https://trec-rag.github.io

Tip-of-the-Tongue¶

The Tip-of-the-Tongue (ToT) Track focuses on the known-item identification task where the searcher has previously experienced or consumed the item (e.g., a movie) but cannot recall a reliable identifier (i.e., It's on the tip of my tongue...). Unlike traditional ad-hoc keyword-based search, these information requests tend to be natural-language, verbose, and complex containing a wide variety of search strategies such as multi-hop reasoning, and frequently express uncertainty and suffer from false memories.

Track coordinator(s):

Jaime Arguello, University of North Carolina
Samarth Bhargav, University of Amsterdam
Bhaskar Mitra, Microsoft Research
Fernando Diaz, Google
Evangelos Kanoulas, University of Amsterdam

Track Web Page: https://trec-tot.github.io/

Video-To-Text¶

Automatic annotation of videos using natural language text descriptions has been a long-standing goal of computer vision. The task involves understanding of many concepts such as objects, actions, scenes, person-object relations, temporal order of events and many others. In recent years there have been major advances in computer vision techniques that enabled researchers to try to solve this problem. A lot of use case application scenarios can greatly benefit from such technology such as video summarization in the form of natural language, facilitating the search and browsing of video archives using such descriptions, describing videos to the blind, etc. In addition, learning video interpretation and temporal relations of events in the video will likely contribute to other computer vision tasks, such as prediction of future events from videos.

Track coordinator(s):

George Awad, NIST
Yvette Graham, Trinity College Dublin
Afzal Godil, NIST

Track Web Page: https://www-nlpir.nist.gov/projects/tv2024/vtt.html