Skip to content

Proceedings 2022

NeuCLIR

Overview of the TREC 2022 NeuCLIR Track

Dawn J. Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, Eugene Yang

Abstract

This is the first year of the TREC Neural CLIR (NeuCLIR) track, which aims to study the impact of neural approaches to cross-language information retrieval. The main task in this year's track was ad hoc ranked retrieval of Chinese, Persian, or Russian newswire documents using queries expressed in English. Topics were developed using standard TREC processes, except that topics developed by an annotator for one language were assessed by a different annotator when evaluating that topic on a different language. There were 172 total runs submitted by twelve teams.

Bibtex 
@inproceedings{DBLP:conf/trec/LawrieMMMOSY22,
    author = {Dawn J. Lawrie and Sean MacAvaney and James Mayfield and Paul McNamee and Douglas W. Oard and Luca Soldaini and Eugene Yang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Overview of the {TREC} 2022 NeuCLIR Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Overview\_neuclir.pdf},
    timestamp = {Wed, 30 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/LawrieMMMOSY22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

CFDA & CLIP at TREC 2022 NeuCLIR Track

Jia-Huei Ju, Wei-Chih Chen, Heng-Ta Chang, Cheng-Wei Lin, Ming-Feng Tsai, Chuan-Ju Wang

Abstract

In this notebook paper, we report our methods and submitted results for the NeuCLIR track in TREC 2022. We adopt the common multi-stage pipeline for the cross-language information retrieval task (CLIR). The pipeline includes machine translation, sparse passage retrieval, and cross-language passage re-ranking. Particularly, we fine-tune cross-language passage re-rankers with different settings of query formulation. In the empirical evaluation on the HC4 dataset, our passage re-rankers achieved better passage re-ranking effectiveness compared to the baseline multilingual re-rankers. The evaluation results of our submitted runs in NeuCLIR are also reported.

Bibtex 
@inproceedings{DBLP:conf/trec/JuCCLTW22,
    author = {Jia{-}Huei Ju and Wei{-}Chih Chen and Heng{-}Ta Chang and Cheng{-}Wei Lin and Ming{-}Feng Tsai and Chuan{-}Ju Wang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{CFDA} {\&} {CLIP} at {TREC} 2022 NeuCLIR Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/CFDA\_CLIP.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/JuCCLTW22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

HNUST @ TREC 2022 NeuCLIR Track

Ge Zhang, Qiwen Ye, Mengmeng Wang, Dong Zhou

Abstract

With the rapid development of deep learning, neural-based cross-language information retrieval (CLIR) has attracted extensive attention from researchers. To explore the effectiveness of neural-based CLIR, large-scale efforts, and new platforms are in need. To that end, the TREC 2022 NeuCLIR track presents a cross-language information retrieval challenge. This paper describes our first participation in the TREC 2022 NeuCLIR track. We explored two approaches for CLIR: (1) the lexical-based CLIR method and (2) the neural-based CLIR method, where the lexical-based method consists of two steps of translation and retrieval, and the neural-based method introduces the DISTILDistilmBERT model, an end-to-end neural network. In our preliminary results, the lexical-based CLIR method performs better than the neural-based method.

Bibtex 
@inproceedings{DBLP:conf/trec/ZhangYWZ22,
    author = {Ge Zhang and Qiwen Ye and Mengmeng Wang and Dong Zhou},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{HNUST} @ {TREC} 2022 NeuCLIR Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/F4.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/ZhangYWZ22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Extremely Fast Fine-Tuning for Cross Language Information Retrieval via Generalized Canonical Correlation

John M. Conroy, Neil P. Molino, Julia S. Yang

Abstract

Recently, work using language agnostic transformer neural sentence embeddings show promise for a robust multilingual sentence representation. Our submission to TREC was to test how well these embeddings could be fine-tuned cheaply to perform the task of cross-lingual information retrieval. We explore the use of the MS MARCO dataset with machine translations as a model problem. We demonstrate that a single generalized canonical correlation analysis (GCCA) model trained on previous queries significantly improves the ability of sentence embeddings to find relevant passages. The dominant computational cost for training is computing dense singular value decompositions (SVDs) of matrices derived from the fine-tuning training data. (The number of SVDs used is the number of languages retrieval views and query views plus 1). This approach illustrates that GCCA methods can be used as a rapid training alternative to fine-tuning a neural net, allowing models to be fine-tuned frequently based on a user's previous queries. This model was then used to prepare submissions for the re-ranking NeuCLIR task.

Bibtex 
@inproceedings{DBLP:conf/trec/ConroyMY22,
    author = {John M. Conroy and Neil P. Molino and Julia S. Yang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Extremely Fast Fine-Tuning for Cross Language Information Retrieval via Generalized Canonical Correlation},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/IDACCS.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/ConroyMY22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

KASYS at the TREC 2022 NeuCLIR Track

Kenya Abe

Abstract

This paper describes the KASYS team's participation in the TREC 2022 NeuCLIR track. Our approach is One-for-All, which employs a single multilingual pre-trained language model to retrieve documents of any languages in response to an English query. The basic architecture is the same as ColBERT and its application to CLIR, ColBERT-X, but only a single model was trained with the mixture of MS MARCO and its translated version, neuMARCO, in our approach. Through the run submission, we evaluated two variants of the One-for-All approach, namely, the end-to-end and reranking approaches. As the first-stage retriever, the former uses approximated nearest neighbor search proposed in ColBERT, while the latter uses the track organizers' (top 1,000 documents in the baseline run were used as the results of the first-stage retrieval). To evaluate our runs, we used the results provided by the track organizers as a baseline (document translation). The official evaluation results showed that the reranking approaches outperforms the baseline in all the languages. On the other hand, the end-to-end approach achieved higher scores than the baseline only in Russian. In addition to the submissions to the TREC 2022 NeuCLIR track, we also conducted experiments with the development data called HC4. The results in HC4 also showed a similar trend: the reranking approach was superior to the end-to-end approach in Persian and Russian. We also found the discrepancy that even in the same language, the performance of our approaches varies depending on the datasets.

Bibtex 
@inproceedings{DBLP:conf/trec/Abe22,
    author = {Kenya Abe},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{KASYS} at the {TREC} 2022 NeuCLIR Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/KASYS.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/Abe22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Carlos Lassance, Stéphane Clinchant

Abstract

This paper describes our participation in the 2022 TREC NeuCLIR challenge. We submitted runs to two out of the three languages (Farsi and Russian), with a focus on first-stage rankers and comparing mono-lingual strategies to Adhoc ones. For monolingual runs, we start from pretraining models on the target language using MLM+FLOPS and then finetuning using the MSMARCO translated to the language either with ColBERT or SPLADE as the retrieval model. While for the Adhoc task we test both query translation (to the target language) and back-translation of the documents (to english). Initial result analysis shows that the monolingual strategy is strong, but that for the moment Adhoc achieved the best results, with back-translating documents being better than translating queries.

Bibtex 
@inproceedings{DBLP:conf/trec/LassanceC22,
    author = {Carlos Lassance and St{\'{e}}phane Clinchant},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Naver Labs Europe {(SPLADE)} @ {TREC} NeuCLIR 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/NLE.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/LassanceC22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

NeuralMind-UNICAMP at 2022 TREC NeuCLIR: Large Boring Rerankers for Cross-lingual Retrieval

Vitor Jeronymo, Roberto de Alencar Lotufo, Rodrigo Frassetto Nogueira

Abstract

This paper reports on a study of cross-lingual information retrieval (CLIR) using the mT5-XXL reranker on the NeuCLIR track of TREC 2022. Perhaps the biggest contribution of this study is the finding that despite the mT5 model being fine-tuned only on query-document pairs of the same language it proved to be viable for CLIR tasks, where query-document pairs are in different languages, even in the presence of suboptimal first-stage retrieval performance. The results of the study show outstanding performance across all tasks and languages, leading to a high number of winning positions. Finally, this study provides valuable insights into the use of mT5 in CLIR tasks and highlights its potential as a viable solution.

Bibtex 
@inproceedings{DBLP:conf/trec/JeronymoLN22,
    author = {Vitor Jeronymo and Roberto de Alencar Lotufo and Rodrigo Frassetto Nogueira},
    editor = {Ian Soboroff and Angela Ellis},
    title = {NeuralMind-UNICAMP at 2022 {TREC} NeuCLIR: Large Boring Rerankers for Cross-lingual Retrieval},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/NM.unicamp.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/JeronymoLN22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval

Jimmy Lin, David Alfonso-Hermelo, Vitor Jeronymo, Ehsan Kamalloo, Carlos Lassance, Rodrigo Frassetto Nogueira, Odunayo Ogundepo, Mehdi Rezagholizadeh, Nandan Thakur, Jheng-Hong Yang, Xinyu Zhang

Abstract

The advent of multilingual language models has generated a resurgence of interest in cross-lingual information retrieval (CLIR), which is the task of searching documents in one language with queries from another. However, the rapid pace of progress has led to a confusing panoply of methods and reproducibility has lagged behind the state of the art. In this context, our work makes two important contributions: First, we provide a conceptual framework for organizing different approaches to cross-lingual retrieval using multi-stage architectures for mono-lingual retrieval as a scaffold. Second, we implement simple yet effective reproducible baselines in the Anserini and Pyserini IR toolkits for test collections from the TREC 2022 NeuCLIR Track, in Persian, Russian, and Chinese. Our efforts are built on a collaboration of the two teams that submitted the most effective runs to the TREC evaluation. These contributions provide a firm foundation for future advances.

Bibtex 
@inproceedings{DBLP:conf/trec/LinAJKLNORTYZ22,
    author = {Jimmy Lin and David Alfonso{-}Hermelo and Vitor Jeronymo and Ehsan Kamalloo and Carlos Lassance and Rodrigo Frassetto Nogueira and Odunayo Ogundepo and Mehdi Rezagholizadeh and Nandan Thakur and Jheng{-}Hong Yang and Xinyu Zhang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/h2oloo.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/LinAJKLNORTYZ22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

HLTCOE at TREC 2022 NeuCLIR Track

Eugene Yang, Dawn J. Lawrie, James Mayfield

Abstract

The HLTCOE team applied ColBERT-X to the TREC 2022 NeuCLIR track with two training techniques - translate-train (TT) and multilingual translate-train (MTT). TT trains ColBERT-X with English queries and passages automatically translated into the document language from the MS-MARCO v1 collection. This results in three cross-language models for the track, one per language. MTT creates a single model for all three document languages by combining the translations of MS-MARCO passages in all three languages into mixed language batches. Thus the model learns about matching queries to passages simultaneously in all languages. While TT is more effective than MTT in each individual language due to its specificity, MTT still outperforms a strong baseline of BM25 with document translation. On average, MTT and TT perform 34% and 48% higher than the median in MAP with title queries, respectively.

Bibtex 
@inproceedings{DBLP:conf/trec/YangLM22,
    author = {Eugene Yang and Dawn J. Lawrie and James Mayfield},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{HLTCOE} at {TREC} 2022 NeuCLIR Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/hltcoe-jhu.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/YangLM22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Huawei Noah's Ark Lab at TREC NeuCLIR 2022

Ehsan Kamalloo, David Alfonso-Hermelo, Mehdi Rezagholizadeh

Abstract

In this paper, we describe our participation in the NeuCLIR track at TREC 2022. Our focus is to build strong ensembles of full-ranking models including dense retrievers, BM25 and learned sparse models.

Bibtex 
@inproceedings{DBLP:conf/trec/KamallooAR22,
    author = {Ehsan Kamalloo and David Alfonso{-}Hermelo and Mehdi Rezagholizadeh},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Huawei Noah's Ark Lab at {TREC} NeuCLIR 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/huaweimtl.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/KamallooAR22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Non-Neural Baselines Experiments for CLIR at TREC 2022

Paul McNamee

Abstract

Cross-Language Information Retrieval (CLIR) returned to TREC with the advent of the NeuCLIR track in 2022. The track provided document collections in three languages: Chinese, Farsi, and Russian, and the principal task involved ranking documents in response to English language queries. Our goal in participating in the NeuCLIR track was to provide a statistical baseline for retrieval, for which we used the HAIRCUT retrieval engine. Experiments included use of character n-gram indexing, use of pseudo-relevance feedback, and application of collection enrichment.

Bibtex 
@inproceedings{DBLP:conf/trec/McNamee22,
    author = {Paul McNamee},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Non-Neural Baselines Experiments for {CLIR} at {TREC} 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/jhu.mcnamee.N.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/McNamee22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Probabilistic Structured Queries: The University of Maryland at the TREC 2022 NeuCLIR Track

Suraj Nair, Douglas W. Oard

Abstract

The University of Maryland submitted three baseline runs to the Ad Hoc CLIR Task of the TREC 2022 NeuCLIR track. This paper describes three baseline systems that cross the language barrier using a well-known translation-based CLIR technique, Probabilistic Structured Queries.

Bibtex 
@inproceedings{DBLP:conf/trec/NairO22,
    author = {Suraj Nair and Douglas W. Oard},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Probabilistic Structured Queries: The University of Maryland at the {TREC} 2022 NeuCLIR Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/umcp.N.pdf},
    timestamp = {Wed, 06 Sep 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/NairO22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Health Misinformation

CiTIUS at the TREC 2022 Health Misinformation Track

Marcos Fernández-Pichel, Manuel de Prada Corral, David E. Losada, Juan Carlos Pichel

Abstract

The TREC Health Misinformation Track fosters the development of retrieval methods that promote credible and correct information over misinformation for health-related decision tasks. To make the procedure more realistic, an Answer Prediction challenge was added to this year's track. In these working notes, we describe our endeavours to estimate the correct response to each topic using search engine results and Transformer models. On the other hand, our adhoc retrieval solutions are based on new addons to our pipeline and the weighted fusion of signals.

Bibtex 
@inproceedings{DBLP:conf/trec/Fernandez-Pichel22,
    author = {Marcos Fern{\'{a}}ndez{-}Pichel and Manuel de Prada Corral and David E. Losada and Juan Carlos Pichel},
    editor = {Ian Soboroff and Angela Ellis},
    title = {CiTIUS at the {TREC} 2022 Health Misinformation Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/CiTIUS.H.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/Fernandez-Pichel22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

UWaterlooMDS at the TREC 2022 Health Misinformation Track

Amir Vakili Tahami, Dake Zhang, Mark D. Smucker

Abstract

In this paper, we report our participation in the TREC 2022 Health Misinformation Track. With the aim to foster research on retrieval algorithms to promote correct information over misinformation for health-related queries, this year's track had two tasks: web retrieval and answer prediction. We reused our previous method with minor modifications to create our baselines. To overcome some limitations of our previous methods, we investigated a document-aware sentence-level passage extraction model based on the BigBird transformer. The upgraded pipeline with this model achieved our best automatic performance on the web retrieval task but failed to beat our baselines on the answer prediction task. Meanwhile, our manual runs still outperformed our automatic runs by great margins on both tasks, showing room for further improvements.

Bibtex 
@inproceedings{DBLP:conf/trec/TahamiZS22,
    author = {Amir Vakili Tahami and Dake Zhang and Mark D. Smucker},
    editor = {Ian Soboroff and Angela Ellis},
    title = {UWaterlooMDS at the {TREC} 2022 Health Misinformation Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/UWaterlooMDS.H.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/TahamiZS22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Webis at TREC 2022: Deep Learning and Health Misinformation

Alexander Bondarenko, Maik Fröbe, Lukas Gienapp, Alexander Pugachev, Jan Heinrich Reimer, Ferdinand Schlatt, Ekaterina Artemova, Martin Potthast, Benno Stein, Pavel Braslavski, Matthias Hagen

Abstract

We describe the Webis group's participation in the TREC 2022 Deep Learning and Health Misinformation tracks. Our runs submitted to the Deep Learning track focus on improving the pairwise retrieval model duoT5 by combining a greedy aggregation algorithm with document augmentation. In the Health Misinformation track, our submissions to the Answer Prediction task exploit pre-trained question answering and claim verification models, whose input is extended by evidence information from PubMed abstracts. For the Web Retrieval task, we explore re-ranking based on the closeness of the predicted answers for each web document in the initial ranking to the predicted “true” answer of a topic's question.

Bibtex 
@inproceedings{DBLP:conf/trec/0001FGPRSAP0BH22,
    author = {Alexander Bondarenko and Maik Fr{\"{o}}be and Lukas Gienapp and Alexander Pugachev and Jan Heinrich Reimer and Ferdinand Schlatt and Ekaterina Artemova and Martin Potthast and Benno Stein and Pavel Braslavski and Matthias Hagen},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Webis at {TREC} 2022: Deep Learning and Health Misinformation},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Webis.DH.pdf},
    timestamp = {Tue, 29 Aug 2023 17:20:28 +0200},
    biburl = {https://dblp.org/rec/conf/trec/0001FGPRSAP0BH22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Deep Learning

Overview of the TREC 2022 Deep Learning Track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff

Abstract

This is the fourth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we also leverage both the refreshed passage and document collections that were released last year leading to a nearly 16 times increase in the size of the passage collection and nearly four times increase in the document collection size. Unlike previous years, in 2022 we mainly focused on constructing a more complete test collection for the passage retrieval task, which has been the primary focus of the track. The document ranking task was kept as a secondary task, where document-level labels were inferred from the passage-level labels. Our analysis shows that similar to previous years, deep neural ranking models that employ large scale pretraining continued to outperform traditional retrieval methods. Due to the focusing our judging resources on passage judging, we are more confident in the quality of this year's queries and judgments, with respect to our ability to distinguish between runs and reuse the dataset in future. We also see some surprises in overall outcomes. Some top-performing runs did not do dense retrieval. Runs that did single-stage dense retrieval were not as competitive this year as they were last year.

Bibtex 
@inproceedings{DBLP:conf/trec/Craswell0YCLVS22,
    author = {Nick Craswell and Bhaskar Mitra and Emine Yilmaz and Daniel Campos and Jimmy Lin and Ellen M. Voorhees and Ian Soboroff},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Overview of the {TREC} 2022 Deep Learning Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Overview\_deep.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/Craswell0YCLVS22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Hybrid Retrieval and Multi-stage Ranking at TREC 2022 Deep Learning Track

Guangwei Xu, Yanzhao Zhang, Longhui Zhang, Dingkun Long, Pengjun Xie, Ruijie Guo

Abstract

Large-scale text retrieval technology has been widely used in various practical business scenarios. This paper presents our systems for the TREC 2022 Deep Learning Track. We explain the hybrid text retrieval and multi-stage text ranking method adopted in our solution. The retrieval stage combined the two structures of traditional sparse retrieval and neural dense retrieval. In the ranking stage, in addition to the full interaction-based ranking model built on large pre-trained language model, we also proposes a lightweight sub-ranking module to further enhance the final text ranking performance. Evaluation results demonstrate the effectiveness of our proposed approach. Our models achieve the 1st and 4th rank on the test set of passage ranking and document ranking respectively.

Bibtex 
@inproceedings{DBLP:conf/trec/XuZZLXG22,
    author = {Guangwei Xu and Yanzhao Zhang and Longhui Zhang and Dingkun Long and Pengjun Xie and Ruijie Guo},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Hybrid Retrieval and Multi-stage Ranking at {TREC} 2022 Deep Learning Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Ali.D.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/XuZZLXG22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

M4D-MKLab/ITI-CERTH Participation in TREC Deep Learning Track 2022

Alexandros-Michail Koufakis, Theodora Tsikrika, Stefanos Vrochidis, Ioannis Kompatsiaris

Abstract

Our team's (CERTH ITI M4D) participation in TREC Deep Learning Track this year focused on the use of the pretrained BERT model in Query Expansion. We submitted two runs in the document retrieval subtask; both include a final reranking step. The first run incorporates a novel pooling approach for the Contextualized Embedding Query Expansion (CEQE) methodology. The second run introduces a novel term selection mechanism that complements the RM3 query expansion method by filtering disadvantageous expansion terms. The term selection mechanism capitalizes on the BERT model by fine tuning it to predict the quality of terms as expansion terms and can be used on any query expansion technique.

Bibtex 
@inproceedings{DBLP:conf/trec/KoufakisTVK22,
    author = {Alexandros{-}Michail Koufakis and Theodora Tsikrika and Stefanos Vrochidis and Ioannis Kompatsiaris},
    editor = {Ian Soboroff and Angela Ellis},
    title = {M4D-MKLab/ITI-CERTH Participation in {TREC} Deep Learning Track 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/CERTH\_ITI\_M4D.D.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/KoufakisTVK22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

CIP at TREC 2022 Deep Learning Track

Jian Luo, Xinlin Peng, Xuanang Chen, Ben He, Le Sun, Yingfei Sun

Abstract

This paper describes the CIP participation in the TREC 2022 Deep Learning Track. We submitted runs to both full ranking and re-ranking subtasks of the passage ranking task. In the full ranking sub-task, we adopt a query noise resistant dense retrieval model RoDR. In the re-ranking subtask, we adopt localized contrastive estimation loss and hinge loss rather than pointwise cross-entropy loss for training rerankers. Besides, We utilize both the MS MARCO v1 and v2 passage datasets to generate hopefully sufficient training data, and our models are fine-tuned on these two kinds of training data one by one selectively. Additionally, we introduce docT5query to further enhance the performance.

Bibtex 
@inproceedings{DBLP:conf/trec/LuoPCH0S22,
    author = {Jian Luo and Xinlin Peng and Xuanang Chen and Ben He and Le Sun and Yingfei Sun},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{CIP} at {TREC} 2022 Deep Learning Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/CIP.D.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/LuoPCH0S22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Efficient Document Representations for Neural Text Ranking

David Rau, Jaap Kamps

Abstract

This paper documents the University of Amsterdam's participation in the TREC 2022 Deep Learning Track. We investigate novel document representation approaches capturing long documents within the input token length of neural rankers, and even in a fraction of the maximum input token length. Reducing input length of the document representation leads to dramatic gains in efficiency, as the self-attention over token length is the main culprit of the high gpu memory footprint, low query latency, and small batch sizes. Our experiments result in a number of findings. First, we observe dramatic gains in efficiency of the document representation approaches mindful of what tokens matter most for the neural rankers. Second, we observe the expected trade-off between effectiveness and efficiency, but also observe that document native approaches retrieve numerous documents missed by passage based approaches. This leads to a significant underestimation of their effectiveness, but also highlights their potential to retrieve documents not considered by traditional rankers or passage based neural rankers. There is great potential to study these in future editions of the track.

Bibtex 
@inproceedings{DBLP:conf/trec/RauK22,
    author = {David Rau and Jaap Kamps},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Efficient Document Representations for Neural Text Ranking},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/UAmsterdam.D.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/RauK22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Université Grenoble Alpes / LIG at TREC Deep Learning Track 2022

Petra Galuscáková, Lucas Alberede, Gabriela Nicole González Sáez, Aidan Mannion, Philippe Mulhem, Georges Quénot

Abstract

This paper describes the submissions of the MRIM research Group/Laboratoire d'Informatique de Grenoble from the Universite Grenoble Alpes at the TREC Deep Learning Track 2022. We participated in both fullranking and reranking sub-tasks in the passages ranking task. We proposed several ways to combine neural and non-neural runs using reciprocal-rank based fusion.

Bibtex 
@inproceedings{DBLP:conf/trec/GaluscakovaASMM22,
    author = {Petra Galusc{\'{a}}kov{\'{a}} and Lucas Alberede and Gabriela Nicole Gonz{\'{a}}lez S{\'{a}}ez and Aidan Mannion and Philippe Mulhem and Georges Qu{\'{e}}not},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Universit{\'{e}} Grenoble Alpes / {LIG} at {TREC} Deep Learning Track 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/UGA.D.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/GaluscakovaASMM22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Experiments with Adaptive ReRanking and ColBERT-PRF: University of Glasgow Terrier Team at TREC DL 2022

Xiao Wang, Sean MacAvaney, Craig Macdonald, Iadh Ounis

Abstract

This paper describes our participation in the TREC 2022 Deep Learning Track. In our participation, we applied the Adaptive ReRanking technique on the constructed corpus graph from various first-stage retrieval models, namely the BM25 and SPLADE retrieval models, before applying the reranker, namely the ELECTRA reranking model. In addition, we employed the ColBERT-PRF technique on various first stage retrieval models. Finally, we experimented with ensemble retrieval for implementing both the Adaptive ReRanking and the ColBERT-PRF techniques. We submitted 14 passage ranking runs (including six baseline runs). Among the submitted runs, the run where the Adaptive ReRanking technique is applied on the ensemble of BM25 and SPLADE retrieval, namely uogtr_e_gb, is the most effective in terms of nDCG@10.

Bibtex 
@inproceedings{DBLP:conf/trec/WangMMO22,
    author = {Xiao Wang and Sean MacAvaney and Craig Macdonald and Iadh Ounis},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Experiments with Adaptive ReRanking and ColBERT-PRF: University of Glasgow Terrier Team at {TREC} {DL} 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/UoGTr.D.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/WangMMO22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Webis at TREC 2022: Deep Learning and Health Misinformation

Alexander Bondarenko, Maik Fröbe, Lukas Gienapp, Alexander Pugachev, Jan Heinrich Reimer, Ferdinand Schlatt, Ekaterina Artemova, Martin Potthast, Benno Stein, Pavel Braslavski, Matthias Hagen

Abstract

We describe the Webis group's participation in the TREC 2022 Deep Learning and Health Misinformation tracks. Our runs submitted to the Deep Learning track focus on improving the pairwise retrieval model duoT5 by combining a greedy aggregation algorithm with document augmentation. In the Health Misinformation track, our submissions to the Answer Prediction task exploit pre-trained question answering and claim verification models, whose input is extended by evidence information from PubMed abstracts. For the Web Retrieval task, we explore re-ranking based on the closeness of the predicted answers for each web document in the initial ranking to the predicted “true” answer of a topic's question.

Bibtex 
@inproceedings{DBLP:conf/trec/0001FGPRSAP0BH22,
    author = {Alexander Bondarenko and Maik Fr{\"{o}}be and Lukas Gienapp and Alexander Pugachev and Jan Heinrich Reimer and Ferdinand Schlatt and Ekaterina Artemova and Martin Potthast and Benno Stein and Pavel Braslavski and Matthias Hagen},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Webis at {TREC} 2022: Deep Learning and Health Misinformation},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Webis.DH.pdf},
    timestamp = {Tue, 29 Aug 2023 17:20:28 +0200},
    biburl = {https://dblp.org/rec/conf/trec/0001FGPRSAP0BH22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

York University at TREC 2022: Deep Learning Track

Yizheng Huang, Jimmy X. Huang

Abstract

The present study outlines the involvement of the YorkU group in the TREC 2022 Deep Learning Track. This year, the investigation into the fusion of BM25 and the deep learning model, which was initiated in the previous year, is pursued further. The findings from last year's experiments indicate that while the deep learning model was superior for most queries, BM25 demonstrated better performance for particular queries. In our contribution, the queries were classified: BM25 was utilized directly as the final ranking result for queries suited to it, whereas the results of the deep learning model were employed for queries incompatible with BM25. The experimental results indicate that this integrated approach yields improved results.

Bibtex 
@inproceedings{DBLP:conf/trec/HuangH22,
    author = {Yizheng Huang and Jimmy X. Huang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {York University at {TREC} 2022: Deep Learning Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/yorku22.D.pdf},
    timestamp = {Tue, 17 Oct 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/HuangH22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Conversational Assistance

TREC CAsT 2022: Going Beyond User Ask and System Retrieve with Initiative and Response Generation

Paul Owoicho, Jeff Dalton, Mohammad Aliannejadi, Leif Azzopardi, Johanne R. Trippas, Svitlana Vakulenko

Abstract

The fourth year of the TREC Conversational Assistance Track (CAsT) continues to focus on evaluating Conversational Passage Ranking (ConvPR) for information seeking but with several new additions to improve the realism of the task and to improve our understanding of conversational search. [...]

Bibtex 
@inproceedings{DBLP:conf/trec/Owoicho0AATV22,
    author = {Paul Owoicho and Jeff Dalton and Mohammad Aliannejadi and Leif Azzopardi and Johanne R. Trippas and Svitlana Vakulenko},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{TREC} CAsT 2022: Going Beyond User Ask and System Retrieve with Initiative and Response Generation},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Overview\_cast.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/Owoicho0AATV22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

CFDA & CLIP at TREC 2022 Conversational Assistance Track (CAsT)

Jia-Huei Ju, Sheng-Chieh Lin, Li-Young Chang, Ming-Feng Tsai, Chuan-Ju Wang

Abstract

In this notebook, we introduce a new pipeline for TREC CAsT 2022. Comparing to the common multistage pipeline for conversational search, we experimented an alternative that does not require conversational query reformulation (CQR). Specifically, our pipeline equipped with conversational dense retriever and conversational passage re-ranker. Our empirical evaluation result on TREC CAsT dataset is also reported in this paper

Bibtex 
@inproceedings{DBLP:conf/trec/JuLCTW22,
    author = {Jia{-}Huei Ju and Sheng{-}Chieh Lin and Li{-}Young Chang and Ming{-}Feng Tsai and Chuan{-}Ju Wang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{CFDA} {\&} {CLIP} at {TREC} 2022 Conversational Assistance Track (CAsT)},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/CFDA\_CLIP.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/JuLCTW22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Context Propagation in Conversational Search Utterances Participation of the CNR Team in CAsT 2022

Ida Mele, Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto

Abstract

Every year, NIST organizes the Text REtrieval Conference (TREC) which gathers competitions for forecasting research on text retrieval. Since 2019, it has included a contest on conversational assistant systems, called Conversational Assistant Track (CAsT) with the purpose of helping research on conversational information systems. CAsT provides test collections for open-domain conversational seeking where the users can ask multiple questions to the system and get answers like in a multi-turn conversation. For our participation in CAsT 2022, we implemented an architecture consisting of two steps: utterance rewriting and passage retrieval. Each run is based on a different utterance rewriting technique for enriching the raw utterance with context extracted from the previous utterances and/or from the replies in the conversation. Three of our approaches are completely automatic, while another one uses the manually rewritten utterances provided by the organizers of TREC CAsT 2022.

Bibtex 
@inproceedings{DBLP:conf/trec/MeleMN0T22,
    author = {Ida Mele and Cristina Ioana Muntean and Franco Maria Nardini and Raffaele Perego and Nicola Tonellotto},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Context Propagation in Conversational Search Utterances Participation of the {CNR} Team in CAsT 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/CNR.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/MeleMN0T22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

University of Cambridge at TREC Cast 2022

Adian Liusie, Mengjie Qian, Xiang Li, Mark J. F. Gales

Abstract

Team heatwave (of the University of Cambridge) submitted 3 automatic runs to the TREC 2022 Conversational Assistance Track. This paper discusses our approach to the challenge of conversational informational retrieval. We first describe our four stage approach of query reformulation, BM25 retrieval, passage reranking, and response extraction. Our experiments then show that our multi-query approach, which uses the raw concatenated conversational context for BM25 and the rewritten query for reranking, shows considerable performance improvement over a single-query approach, where our best performing system achieves a NDCG@3 of 0.440 in the 2022 CAsT challenge.

Bibtex 
@inproceedings{DBLP:conf/trec/LiusieQLG22,
    author = {Adian Liusie and Mengjie Qian and Xiang Li and Mark J. F. Gales},
    editor = {Ian Soboroff and Angela Ellis},
    title = {University of Cambridge at {TREC} Cast 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/HEATWAVE.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/LiusieQLG22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

MLIA-DAC@TREC CAsT 2022: Sparse Contextualized Query Embedding

Nam Le Hai, Thomas Gerald, Thibault Formal, Jian-Yun Nie, Benjamin Piwowarski, Laure Soulier

Abstract

We extend SPLADE, a sparse information retrieval model, as our first stage ranker for the conversational task. This end-to-end approach achieves a high recall (as measure on TREC CAsT 2021). To further increase the effectiveness of our approach, we train a T5-based re-ranker. This working note fully describes our model and the four runs submitted to TREC CAsT 2022.

Bibtex 
@inproceedings{DBLP:conf/trec/HaiGFNPS22,
    author = {Nam Le Hai and Thomas Gerald and Thibault Formal and Jian{-}Yun Nie and Benjamin Piwowarski and Laure Soulier},
    editor = {Ian Soboroff and Angela Ellis},
    title = {MLIA-DAC@TREC CAsT 2022: Sparse Contextualized Query Embedding},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/MLIA-DAC.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/HaiGFNPS22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

The University of Stavanger (IAI) at the TREC 2022 Conversational Assistance Track

Weronika Lajewska, Nolwenn Bernard, Ivica Kostric, Ivan Sekulic, Krisztian Balog

Abstract

This paper describes the participation of the IAI group at the University of Stavanger in the TREC 2022 Conversational Assistance track. We employ an established two-stage passage ranking architecture, i.e., first-pass passage retrieval (with standard BM25 ranking and pseudo-relevance feedback) followed by re-ranking (with mono and duo T5) using a rewritten query (with a T5 model fine-tuned on the CANARD dataset). In our runs, we experiment with intent classification based on MSDialog-Intent and term expansion using beam search scores for query rewriting as well as with clarifying questions for the mixed-initiative subtask.

Bibtex 
@inproceedings{DBLP:conf/trec/LajewskaBKSB22,
    author = {Weronika Lajewska and Nolwenn Bernard and Ivica Kostric and Ivan Sekulic and Krisztian Balog},
    editor = {Ian Soboroff and Angela Ellis},
    title = {The University of Stavanger {(IAI)} at the {TREC} 2022 Conversational Assistance Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/UiS.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/LajewskaBKSB22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

University of Glasgow Terrier Team at the TREC 2022 Conversational Assistance Track

Sarawoot Kongyoung, Craig Macdonald, Iadh Ounis

Abstract

In this paper, we describe our methods and submitted runs for the TREC 2022 Conversational Assistance Track. In our participation, we leverage Multi-Task Learning (MTL) methods to enhance the performance of the conversational search system. For the main task, we use our recently proposed monoQA model, which applies Multi-Task Learning (MTL) on reranking and answer extraction by sharing a single text generation model, predicts both the answer and the reranking score simultaneously. For the mixed-initiative sub-task, we propose T5MI, which is trained on the ClariQ dataset, to determine whether a user utterance needs to ask clarifying questions, as well as to generate useful clarifying questions. This year, we submitted three runs based on the data used in the testing step consisting of 1) uogTr-MT: using the provided manually rewritten utterances as the queries; 2) uogTr-AT: using the raw utterances and the provided provenances as the context for rewriting the queries; 3) uogTr-MI-HB: using the raw utterances and the output from the mixed-initiative sub-task as the context for rewriting the queries.

Bibtex 
@inproceedings{DBLP:conf/trec/KongyoungMO22,
    author = {Sarawoot Kongyoung and Craig Macdonald and Iadh Ounis},
    editor = {Ian Soboroff and Angela Ellis},
    title = {University of Glasgow Terrier Team at the {TREC} 2022 Conversational Assistance Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/UoGTr.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/KongyoungMO22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

WaterlooClarke at the TREC 2022 Conversational Assistant Track

Siqing Huo, Xinyi Yan, Charles L. A. Clarke

Abstract

The WaterlooClarke group submitted three runs to TREC Conversational Assistant Track (CAsT) 2022: UWCmanual22, UWCauto22, and UWCcano22. This report describes the techniques used and the results of these three runs. The pipeline to generate each run contains three main steps: 1) query reformulation, 2) passage retrieval and reranking, 3) passage summarization and reranking. For the UWCmanual22 run, we used the manual rewritten utterances provided as reformulated queries. For the UWCauto22 run, we used the automatic rewritten utterances provided as reformulated queries. For the UWCcano22 run, we augmented the automatic rewritten utterances provided with keywords extracted from the previous responses' titles, and used them as reformulated queries. We continued last year's strategy [6] of performing passage reranking with both MonoT5 and DuoT5 [2] on the pooled retrieved passages from both sparse and dense retrievers [1]. This year we focused on experimenting with the following changes: 1) Incorporating generative summarization techniques to make the produced answers more conversational. 2) Incorporating pseudo-relevance feedback into the passage retrieval stage to improve performance. In the next section, we will present the details of our pipeline

Bibtex 
@inproceedings{DBLP:conf/trec/HuoYC22,
    author = {Siqing Huo and Xinyi Yan and Charles L. A. Clarke},
    editor = {Ian Soboroff and Angela Ellis},
    title = {WaterlooClarke at the {TREC} 2022 Conversational Assistant Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/WaterlooClarke.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/HuoYC22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Question Answering-Based Query Expansion for Conversational Search: IIIA@UNIPD at TREC CAsT 2022

Guglielmo Faggioli, Nicola Ferro, Mattia Romanello

Abstract

Conversational Search task is becoming ubiquitous in our daily interaction with information systems. Nevertheless, it remains an extremely complex task due to its profoundly different nature from ad-hoc retrieval, and the challenges presented by the way a user interacts with the system. CAsT TREC Track explicitly aims at fostering the development of conversational systems and the discussion of the linked challenges within the research community. In this work, we describe the approach that we propose to CAsT 2022 to tackle the conversational task. In particular, our approach is based on expanding and rewriting the utterance at hand with the response to the previous utterance, using a QA model to extract it from the first passage retrieved in response to the previous query.

Bibtex 
@inproceedings{DBLP:conf/trec/Faggioli0R22,
    author = {Guglielmo Faggioli and Nicola Ferro and Mattia Romanello},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Question Answering-Based Query Expansion for Conversational Search: IIIA@UNIPD at {TREC} CAsT 2022},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/iiia-unipd.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/Faggioli0R22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

An Exploration Study of Mixed-initiative Query Reformulation in Conversational Passage Retrieval

Dayu Yang, Yue Zhang, Hui Fang

Abstract

In this paper, we report our methods and experiments for the TREC Conversational Assistance Track (CAsT) 2022. In this work, we aim to reproduce multi-stage retrieval pipelines and explore one of the potential benefits of involving mixed-initiative interaction in conversational passage retrieval scenarios: reformulating raw queries. Before the first ranking stage of a multi-stage retrieval pipeline, we propose a mixed-initiative query reformulation module, which achieves query reformulation based on the mixed-initiative interaction between the users and the system, as the replacement for the neural reformulation method. Specifically, we design an algorithm to generate appropriate questions related to the ambiguities in raw queries, and another algorithm to reformulate raw queries by parsing users' feedback and incorporating it into the raw query. For the first ranking stage of our multi-stage pipelines, we adopt a sparse ranking function: BM25, and a dense retrieval method: TCT-ColBERT. For the second-ranking step, we adopt a pointwise reranker: MonoT5, and a pairwise reranker: DuoT5. Experiments on both TREC CAsT 2021 and TREC CAsT 2022 datasets show the effectiveness of our mixed-initiative-based query reformulation method on improving retrieval performance compared with two popular reformulators: a neural reformulator: CANARD-T5 and a rule-based reformulator: historical query reformulator(HQE)

Bibtex 
@inproceedings{DBLP:conf/trec/YangZF22,
    author = {Dayu Yang and Yue Zhang and Hui Fang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {An Exploration Study of Mixed-initiative Query Reformulation in Conversational Passage Retrieval},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/udel\_fang.C.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/YangZF22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Clinical Trials

Overview of the TREC 2022 Clinical Trials Track

Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, Steven Bedrick, William R. Hersh

Abstract

Clinical trials are the primary means for new medical treatments-such as drugs, surgical procedures, and behavior interventions-to demonstrate their effectiveness in an evidence-based manner. However, clinical trials have high costs, can take years to complete, and oftentimes fail to identify a sufficient number of patients to establish clinical significance. Automated methods for improving the patient recruitment process can aid in all three areas: reducing manual and expensive chart review, more quickly identifying eligible patients, and expanding the pool of candidate patients that may be eligible. The primary means of automating clinical trial recruitment is through the use of electronic health record (EHR) data. EHRs are responsible for documenting routine medical care, as well as being the legal and billing record. However, the re-use of EHR data for research is well-established (Hersh, 2007), commonly for observational studies but also as the source data for informatics-driven models, including machine learning (ML) and information retrieval (IR). This was the inspiration behind the TREC Medical Records track (2011-2012) (Voorhees and Tong, 2011; Voorhees and Hersh, 2012), which used short cohort descriptions as queries (e.g., “Patients treated for vascular claudication surgically”) and used EHR visit records as the document collection. Unfortunately, this track was discontinued due to the the lack of an EHR dataset of sufficient size to merit a proper IR evaluation. The TREC Clinical Trials track, instead, flips the trial-to-patients paradigm to a patient-to-trials paradigm. This has enabled the building of a large test collection for clinical trial search. In this paradigm, the topic is a (synthetic) patient description and the document collection is a large set of clinical trial descriptions (which are, notably, publicly and freely available). There are several challenges involved with task, however. The first set of challenges revolve around using clinical trial descriptions as the document collection. Clinical trial descriptions are often very long (see link to trial in Table 2). The core part of the description with regards to trial matching is the eligibility criteria, a (often long) list of inclusion criteria (the patient must meet all these requirements) and exclusion criteria (if the patient meets any of these criteria, they are ineligible and would be excluded from the trial). These criteria not only use complex medical terminology, but they are often written in a way that does not correspond directly to how patient cases are described in the EHR, making direct term mapping problematic. The second set of challenges revolves around the patient cases. In addition to the linguistic issues of how identical clinical concepts in EHR text versus trial descriptions, patient cases contain significant amounts of extraneous information with respect to the clinical trial. That is, not all of the information in a patient case need be covered in the trial. Rather, a sufficient amount of information must be present to suggest the patient may be eligible, while also not containing information showing the patient to be excluded. This means that many of the conditions in the patient description are irrelevant for a single clinical trial, whereas matching to a different clinical trial may involve a different subset of conditions in the patient case. As in 2021 Roberts et al. (2021b), to ensure this task focuses on information retrieval and not supervised information extraction, we present a lengthy (5-10 sentence) patient case description as the topic that simulates an admission statement in an EHR. The evaluation is further be broken down into Eligible, Excludes, and Not Relevant to allow retrieval methods to distinguish between patients that do not have sufficient information to qualify for the trial (Not Relevant) and those that are explicitly Excluded. This latter category can be difficult for retrieval systems without strong semantic understanding.

Bibtex 
@inproceedings{DBLP:conf/trec/RobertsDVBH22,
    author = {Kirk Roberts and Dina Demner{-}Fushman and Ellen M. Voorhees and Steven Bedrick and William R. Hersh},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Overview of the {TREC} 2022 Clinical Trials Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Overview\_trials.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/RobertsDVBH22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Matching a Patient from An Admission Note to Clinical Trials: Experiments with Query Generation and Neural-Ranking

Vincent Nguyen, Maciej Rybinski, Sarvnaz Karimi

Abstract

Many clinical trials fail to attract enough eligible participants. The TREC 2022 Clinical Trials track set a task where patient data, in the form of clinical notes, can be used to match eligible patients to a relevant clinical trial. We explore a number of dense retrieval methods using Bidirectional Encoder Representations from Transformers (BERT). Our best method used BERT reranking using models based on monoBERT architecture. Our self-supervised monoBERT run achieved effectiveness competitive to that of a fully-tuned monoBERT run.

Bibtex 
@inproceedings{DBLP:conf/trec/NguyenRK22,
    author = {Vincent Nguyen and Maciej Rybinski and Sarvnaz Karimi},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Matching a Patient from An Admission Note to Clinical Trials: Experiments with Query Generation and Neural-Ranking},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/CSIROmed.T.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/NguyenRK22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

UNIMIB at TREC 2022 Clinical Trials Track

Georgios Peikos, Gabriella Pasi

Abstract

This notebook summarizes our participation as the UNIMIB team in the TREC 2022 Clinical Trials Track. In this work, we extend our last year's participation by further investigating the retrieval performance achieved by our decision-theoretic model for relevance estimation. Specifically, our objective was to investigate the effectiveness of our decision-theoretic model by heavily penalizing those clinical trials for which the patient has high topical similarity to the exclusion criteria. The model has been employed to estimate relevance in two retrieval settings, ranking and re-ranking. The obtained results showed that the proposed model performs equally well in both of them, while the best results in terms of precision were achieved in the re-ranking setting.

Bibtex 
@inproceedings{DBLP:conf/trec/PeikosP22,
    author = {Georgios Peikos and Gabriella Pasi},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{UNIMIB} at {TREC} 2022 Clinical Trials Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/UNIMIB.T.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/PeikosP22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Elsevier Data Science Health Sciences at TREC 2022 Clinical Trials: Exploring Transformer Embeddings for Clinical Trial Retrieval

Drahomira Herrmannova, Sharvari Jadhav, Harsh Sindhwa, Hina Nazir, Elia Lima-Walton

Abstract

In this paper, we describe the submissions of Elsevier Data Science Health Sciences to the TREC 2022 Clinical Trials Track. Our submissions explored the applicability of transformer embeddings to the task and demonstrated a straightforward retriever using the MiniLM model can achieve competitive performance. Additionally, we share observations from a manual evaluation we performed to better understand the performance of our embedding-based retrievers

Bibtex 
@inproceedings{DBLP:conf/trec/HerrmannovaJSNL22,
    author = {Drahomira Herrmannova and Sharvari Jadhav and Harsh Sindhwa and Hina Nazir and Elia Lima{-}Walton},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Elsevier Data Science Health Sciences at {TREC} 2022 Clinical Trials: Exploring Transformer Embeddings for Clinical Trial Retrieval},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/els\_dshs.T.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/HerrmannovaJSNL22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Summarize and Expand Queries in Clinical Trials Retrieval. The IIIA Unipd at TREC 2022 Clinical Trials

Giorgio Maria Di Nunzio, Guglielmo Faggioli, Stefano Marchesin

Abstract

We present the methodology and the experimental setting we, the Intelligent Interactive Information Access (IIIA)1 UNIPD team, used in the TREC Clinical Trials 2022. This work continues the long streak of studies we carried out at TREC Precision Medicine to evaluate the effectiveness of query reformulation, pseudo-relevance feedback, and document filtering. Compared to the procedure proposed in 2021, we introduced the use of manual summarization and removed rank fusion. The obtained results provide interesting insights on the different per-topic effectiveness and will be used for further analyses.

Bibtex 
@inproceedings{DBLP:conf/trec/NunzioF022,
    author = {Giorgio Maria Di Nunzio and Guglielmo Faggioli and Stefano Marchesin},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Summarize and Expand Queries in Clinical Trials Retrieval. The {IIIA} Unipd at {TREC} 2022 Clinical Trials},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/iiia-unipd.T.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/NunzioF022.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

JBNU at TREC 2022 Clinical Trials Track

Dalya Sin, Woo-Kyoung Lee, Seung-Hyeon Jo, Kyung-Soon Lee

Abstract

This paper describes the participation of the JBNU team at the TREC 2022 Clinical Trials Track. Our approach is to focus on clinical terms detected from the ClinicalBERT and BioBERT. In order to expand clinical terms, the synonym terms are extracted by BERT embedding model. Our experimental results showed 0.4527, 0.3220 and 0.5543 by NDCG@10, P@10 and MRR, respectively.

Bibtex 
@inproceedings{DBLP:conf/trec/SinLJL22,
    author = {Dalya Sin and Woo{-}Kyoung Lee and Seung{-}Hyeon Jo and Kyung{-}Soon Lee},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{JBNU} at {TREC} 2022 Clinical Trials Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/jbnu.T.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/SinLJL22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

CogStack Cohort at TREC 2022 Clinical Trials Track

Jack Wu, Zeljko Kraljevic, Thomas Searle, Daniel Bean, Richard J. B. Dobson

Abstract

This notebook paper describes the methodology used to produce the retrieval results we submitted for TREC 2022 Clinical Trials Track. The method is based on named entity recognition and linking (NER+L). Medical Concept Annotation Tool (MedCAT) is used to perform NER+L on the topics and documents to produce entities in the SNOMED Clinical Terms ontology. The clinical terms extracted by the annotation process are used for indexing and retrieval purposes. The retrieval model used is a passage-based retrieval model that gives different weights to different document portions.

Bibtex 
@inproceedings{DBLP:conf/trec/WuKSBD22,
    author = {Jack Wu and Zeljko Kraljevic and Thomas Searle and Daniel Bean and Richard J. B. Dobson},
    editor = {Ian Soboroff and Angela Ellis},
    title = {CogStack Cohort at {TREC} 2022 Clinical Trials Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/phi\_lab.T.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/WuKSBD22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Hybrid Re-ranking for Biomedical Information Retrieval at the TREC 2021 Clinical Trials Track

Ming-Xuan Shi, Tsung-Hsuan Pan, Hsin-Hsi Chen, Hen-Hsen Huang

Abstract

This paper presents our methodology for the task of TREC 2021 Clinical Trials Track, which requires a system to retrieve and return the most relevant biomedical articles after giving queries. We propose a novel approach to biomedical information retrieval by leveraging the term-based and the embedding-based retrieval models with a re-ranking strategy. In our hybrid framework, all the documents will be indexed by using a term-based, efficient search engine. For the given query, a smaller set of candidate results are retrieved from the search engine. The ranking is determined not only by the term-based ranking score but also by the term relationships labeled by the Amazon Comprehend service1 for refinement. Then, the candidate results are further scored by using the embedding-based model. We represent the document and the query with bioBERT and compute the cosine similarity between a pair of the document embedding and the query embedding as their relevance score. The final score is a linear combination of the term-based and the embedding-based scores. Experimental results show that our hybrid re-ranking method improves both Precision@k and R-precision scores on the 2016 Clinical Decision Support Track and 2021 Clinical Trials Track dataset.

Bibtex 
@inproceedings{DBLP:conf/trec/ShiPCH21,
    author = {Ming{-}Xuan Shi and Tsung{-}Hsuan Pan and Hsin{-}Hsi Chen and Hen{-}Hsen Huang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Hybrid Re-ranking for Biomedical Information Retrieval at the {TREC} 2021 Clinical Trials Track},
    booktitle = {Proceedings of the Thirtieth Text REtrieval Conference, {TREC} 2021, online, November 15-19, 2021},
    series = {{NIST} Special Publication},
    volume = {500-335},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2021},
    url = {https://trec.nist.gov/pubs/trec30/papers/NTU\_NLP-CT.pdf},
    timestamp = {Mon, 28 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/ShiPCH21.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Fair Ranking

Overview of the TREC 2022 Fair Ranking Track

Michael D. Ekstrand, Graham McDonald, Amifa Raj, Isaac Johnson

Abstract

The TREC Fair Ranking Track aims to provide a platform for participants to develop and evaluate novel retrieval algorithms that can provide a fair exposure to a mixture of demographics or attributes, such as ethnicity, that are represented by relevant documents in response to a search query. For example, particular demographics or attributes can be represented by the documents' topical content or authors. The 2022 Fair Ranking Track adopted a resource allocation task. The task focused on supporting Wikipedia editors who are looking to improve the encyclopedia's coverage of topics under the purview of a WikiProject.1 WikiProject coordinators and/or Wikipedia editors search for Wikipedia documents that are in need of editing to improve the quality of the article. The 2022 Fair Ranking track aimed to ensure that documents that are about, or somehow represent, certain protected characteristics receive a fair exposure to the Wikipedia editors, so that the documents have an fair opportunity of being improved and, therefore, be well-represented in Wikipedia. The under-representation of particular protected characteristics in Wikipedia can result in systematic biases that can have a negative human, social, and economic impact, particularly for disadvantaged or protected societal groups [4, 7].

Bibtex 
@inproceedings{DBLP:conf/trec/EkstrandMR022,
    author = {Michael D. Ekstrand and Graham McDonald and Amifa Raj and Isaac Johnson},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Overview of the {TREC} 2022 Fair Ranking Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/Overview\_fair.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/EkstrandMR022.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

RMIT CIDDA IR at the TREC 2022 Fair Ranking Track

Sachin Pathiyan Cherumanal, Marwah Alaofi, Reham Abdullah Altalhi, Elham Naghizade, Falk Scholer, Damiano Spina

Abstract

This report describes the participation of the RMIT CIDDA IR1 group at the TREC 2022 Fair Ranking Track (Task 1). We submitted 8 runs with the aim to explore the role of explicit search result diversification, ranking fusion, and the use of a multi-criteria decision-making method to generate fair rankings considering multiple protected attributes.

Bibtex 
@inproceedings{DBLP:conf/trec/CherumanalAANSS22,
    author = {Sachin Pathiyan Cherumanal and Marwah Alaofi and Reham Abdullah Altalhi and Elham Naghizade and Falk Scholer and Damiano Spina},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{RMIT} {CIDDA} {IR} at the {TREC} 2022 Fair Ranking Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/rmit\_cidda\_ir.F.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/CherumanalAANSS22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

An Exploration of Learning-to-re-rank Using a Two-step Framework for Fair Ranking

Fumian Chen, Hui Fang

Abstract

As fairness has been attracting increasing attention in search engines, producing both fair and relevant rankings has become a major challenge for many information retrieval systems. In 2022, TREC fair ranking track launched an ad-hoc retrieval task for Wikipedia editors, which returns not only relevant documents of each query but also provides fair exposure to each document. The main goal of our participation in this track is to explore a fairness-aware two-step fair ranking framework using supervised learning-based re-rankers. Given a query, we first retrieve relevant documents and then use the re-ranker to re-rank the documents so that the final ranking is fair evaluated by the attention-weighted rank fairness (AWRF). We utilize the simple yet well-performed BM25 retrieval model to obtain relevant documents of each query. We tested a gradient-boosted tree-based (GBDT) LambdaMART model and a neural-based multilayer perceptron model. To better train the supervised learning models, we also extracted contextual-based features to augment our training data. We encoded fairness through a pre-processing method by constructing fairness scores. Our results also show the possibility of leveraging supervised learning-based re-rankers and contextual features.

Bibtex 
@inproceedings{DBLP:conf/trec/ChenF22,
    author = {Fumian Chen and Hui Fang},
    editor = {Ian Soboroff and Angela Ellis},
    title = {An Exploration of Learning-to-re-rank Using a Two-step Framework for Fair Ranking},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/udel\_fang.F.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/ChenF22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

CrisisFACTs

IRIT-IRIS at TREC 2022: CrisisFACTS Track

Alexis Dusart, Gilles Hubert, Karen Pinel-Sauvagnat

Abstract

This paper presents the approaches proposed by the IRIS team of the IRIT laboratory for the TREC CrisisFACTS track. The CrisisFACTS track aims to summarize online data sources during crisis events. In our participation, we used neural language models according to three different strategies.

Bibtex 
@inproceedings{DBLP:conf/trec/DusartHP22,
    author = {Alexis Dusart and Gilles Hubert and Karen Pinel{-}Sauvagnat},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{IRIT-IRIS} at {TREC} 2022: CrisisFACTS Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/IRIT\_IRIS.R.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/DusartHP22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Using Neural Reranking and GPT-3 for Social Media Disaster Content Summarization

Jayr Alencar Pereira, Robson do Nascimento Fidalgo, Roberto de Alencar Lotufo, Rodrigo Frassetto Nogueira

Abstract

Managing emergency events, such as natural disasters, necessitates real-time situational awareness for management teams. This paper presents a novel approach to obtaining accurate and comprehensive summaries of these events by utilizing a state-of-the-art open-source search engine and a large language model to retrieve and summarize information from social media and online news sources. The efficacy of this approach was evaluated on the TREC CrisisFACTS challenge dataset, utilizing automatic summarization metrics (e.g. Rouge-2 and BERTScore) and manual evaluation by the challenge organizers. Results indicate that while our approach achieves high comprehensiveness, it also exhibits a high degree of summary redundancy. Importantly, the pipeline components are few-shot, avoiding the need for training data collection and enabling rapid deployment.

Bibtex 
@inproceedings{DBLP:conf/trec/PereiraFLN22,
    author = {Jayr Alencar Pereira and Robson do Nascimento Fidalgo and Roberto de Alencar Lotufo and Rodrigo Frassetto Nogueira},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Using Neural Reranking and {GPT-3} for Social Media Disaster Content Summarization},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/NM.unicamp.R.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/PereiraFLN22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

L3S at the TREC 2022 CrisisFACTS Track

Thi Huyen Nguyen, Koustav Rudra

Abstract

This paper describes our proposed approach for the multi-stream summarization of the crisis-related events in the TREC 2022 CrisisFACTS track [2]. We apply a retrieval and ranking-based two-step summarization approach. First, we employ a sparse retrieval framework where content texts from multiple online streams are treated as a document corpus, and a term matching-based retrieval strategy is used to retrieve relevant contents, so-called facts, to the set of queries in a given event day. Next, we use several pre-trained models to measure semantic similarity between query-fact or fact-fact pairs, score and rank the facts for the extraction of daily event summaries.

Bibtex 
@inproceedings{DBLP:conf/trec/NguyenR22,
    author = {Thi Huyen Nguyen and Koustav Rudra},
    editor = {Ian Soboroff and Angela Ellis},
    title = {{L3S} at the {TREC} 2022 CrisisFACTS Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/eXSum22.R.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/NguyenR22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Combining Deep Neural Reranking and Unsupervised Extraction for Multi-Query Focused Summarization

Philipp Seeberger, Korbinian Riedhammer

Abstract

The CrisisFACTS Track aims to tackle challenges such as multi-stream fact-finding in the domain of event tracking; participants' systems extract important facts from several disasterrelated events while incorporating the temporal order. We propose a combination of retrieval, reranking, and the well-known Integer Linear Programming (ILP) and Maximal Marginal Relevance (MMR) frameworks. In the former two modules, we explore various methods including an entity-based baseline, pre-trained and fine-tuned Question Answering systems, and ColBERT. We then use the latter module as an extractive summarization component by taking diversity and novelty criteria into account. The automatic scoring runs show strong results across the evaluation setups but also reveal shortcomings and challenges.

Bibtex 
@inproceedings{DBLP:conf/trec/SeebergerR22,
    author = {Philipp Seeberger and Korbinian Riedhammer},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Combining Deep Neural Reranking and Unsupervised Extraction for Multi-Query Focused Summarization},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/ohm\_kiz.R.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/SeebergerR22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Multi-Faceted Question Fusion in the TREC 2022 CrisisFACTS Track

Nathaniel W. Rollings, Peter A. Rankel, Douglas W. Oard

Abstract

To address the challenges of multi-faceted questions in rapidly evolving environments, this paper introduces a system with an architecture based on recency-weighted and score-weighted reciprocal rank fusion of per-facet ranked lists. In the absence of existing data for parameter tuning, a small test collection built to support formative evaluation was developed and employed in system refinement. Issues of duplication were addressed through pruning near duplicates and, in one variation, synthesizing rather than simply selecting responses.

Bibtex 
@inproceedings{DBLP:conf/trec/RollingsRO22,
    author = {Nathaniel W. Rollings and Peter A. Rankel and Douglas W. Oard},
    editor = {Ian Soboroff and Angela Ellis},
    title = {Multi-Faceted Question Fusion in the {TREC} 2022 CrisisFACTS Track},
    booktitle = {Proceedings of the Thirty-First Text REtrieval Conference, {TREC} 2022, online, November 15-19, 2022},
    series = {{NIST} Special Publication},
    volume = {500-338},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2022},
    url = {https://trec.nist.gov/pubs/trec31/papers/umcp.R.pdf},
    timestamp = {Tue, 29 Aug 2023 01:00:00 +0200},
    biburl = {https://dblp.org/rec/conf/trec/RollingsRO22.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}