Skip to content

Proceedings - Tip-of-the-Tongue 2024

OVERVIEW OF THE TREC 2024 TIP-OF-THE-TONGUE TRACK

Jaime Arguello, Samarth Bhargav, Fernando Diaz, To Eun Kim, Yifan He, Evangelos Kanoulas, Bhaskar Mitra

Abstract

Tip-of-the-tongue (ToT) known-item retrieval involves re-finding an item for which the searcher does not reliably recall an identifier. ToT information requests (or queries) are verbose and tend to include several complex phenomena, making them especially difficult for existing information retrieval systems. The TREC 2024 ToT track focused on a single ad-hoc retrieval task. Participants were provided with training and development data in the movie domain. Conversely, systems were tested on data that combined three domains: movies, celebrities, and landmarks. This year, 6 groups (including the track coordinators) submitted 18 runs.

Bibtex 
@inproceedings{coordinators-trec2024-papers-proc-3,
    title = {OVERVIEW OF THE TREC 2024 TIP-OF-THE-TONGUE TRACK},
    author = {Jaime Arguello and Samarth Bhargav and Fernando Diaz and To Eun Kim and Yifan He and Evangelos Kanoulas and Bhaskar Mitra},
    booktitle = {Proceedings of the 33th Text {REtrieval} Conference (TREC 2024)},
    year = {2024},
    address = {Gaithersburg, Maryland},
    series = {NIST SP 1329}
}

Yale NLP at TREC 2024: Tip-of-the-Tongue Track

Rohan Phanse, Gabrielle Kaili-May Liu, Arman Cohan

Abstract

After preparing training sets for each domain, we trained a single “general” DPR model to handle queries from all domains and used it in our first two runs. In addition, we developed an approach to route queries to multiple single-domain “expert” DPR models for our third run. We used GPT-4o mini to rerank the results retrieved by our DPR models. We developed an initial pointwise reranking stage that we used along with Borges et al.’s [3] list-wise round-robin approach in our first run. We only performed listwise reranking in our other two runs to measure the specific contribution of our proposed pointwise reranking stage to overall performance.

Bibtex 
@inproceedings{yalenlp-trec2024-papers-proc-1,
    title = {Yale NLP at TREC 2024: Tip-of-the-Tongue Track},
    author = {Rohan Phanse and Gabrielle Kaili-May Liu and Arman Cohan},
    booktitle = {Proceedings of the 33th Text {REtrieval} Conference (TREC 2024)},
    year = {2024},
    address = {Gaithersburg, Maryland},
    series = {NIST SP 1329}
}

Webis at TREC 2024: Biomedical Generative Retrieval, Retrieval-Augmented Generation, and Tip-of-the-Tongue Tracks

Lukas Gienapp, Maik Fröbe, Jan Heinrich Merker, Harrisen Scells, Eric Oliver Schmidt, Matti Wiegmann, Martin Potthast, Matthias Hagen

Abstract

In this paper, we describe the Webis Group’s participation in the 2024 edition of TREC. We participated in the Biomedical Generative Retrieval track, the Retrieval-Augmented Generation track, and the Tip-of-the-Tongue track. For the biomedical track, we applied different paradigms of retrieval-augmented generation with open- and closed-source LLMs. For the Retrieval-Augmented Generation track, we aimed to contrast manual response submissions with fully-automated responses. For the Tip-of-the-Tongue track, we employed query relaxation as in our last year’s submission (i.e., leaving out terms that likely reduce the retrieval effectiveness) that we combine with a new cross-encoder that we trained on an enriched version of the TOMT-KIS dataset.

Bibtex 
@inproceedings{webis-trec2024-papers-proc-1,
    title = {Webis at TREC 2024: Biomedical Generative Retrieval, Retrieval-Augmented Generation, and Tip-of-the-Tongue Tracks},
    author = {Lukas Gienapp and Maik Fröbe and Jan Heinrich Merker and Harrisen Scells and Eric Oliver Schmidt and Matti Wiegmann and Martin Potthast and Matthias Hagen},
    booktitle = {Proceedings of the 33th Text {REtrieval} Conference (TREC 2024)},
    year = {2024},
    address = {Gaithersburg, Maryland},
    series = {NIST SP 1329}
}

IISERK@ToT_2024: Query Reformulation and Layered Retrieval for Tip-of-Tongue Items

Subinay Adhikary, Shuvam Banerji Seal, Soumyadeep Sar, Dwaipayan Roy

Abstract

In this study, we explore various approaches for known-item retrieval, referred to as “Tip-of-the-Tongue” (ToT). The TREC 2024 ToT track involves retrieving previously encountered items, such as movie names or landmarks when the searcher struggles to recall their exact identifiers. In this paper, we (ThinkIR) focus on four different approaches to retrieve the correct item for each query, including BM25 with optimized parameters and leveraging Large Language Models (LLMs) to reformulate the queries. Subsequently, we utilize these reformulated queries during retrieval using the BM25 model for each method. The four-step query reformulation technique, combined with two-layer retrieval, has enhanced retrieval performance in terms of NDCG and Recall. Eventually, two-layer retrieval achieves the best performance among all the runs, with a Recall@1000 of 0.8067.

Bibtex 
@inproceedings{IISER-K-trec2024-papers-proc-1,
    title = {IISERK@ToT\_2024: Query Reformulation and Layered Retrieval for Tip-of-Tongue Items},
    author = {Subinay Adhikary and Shuvam Banerji Seal and Soumyadeep Sar and Dwaipayan Roy},
    booktitle = {Proceedings of the 33th Text {REtrieval} Conference (TREC 2024)},
    year = {2024},
    address = {Gaithersburg, Maryland},
    series = {NIST SP 1329}
}