Runs - Tip-of-the-Tongue 2023¶

baseline_bm25¶

Results | Participants | Input | Summary | Appendix

Run ID: baseline_bm25
Participant: UAmsterdam
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/18/2023
MD5: 6e3a4d4dff7cdd9f7e7e268aa3f7ef7a

baseline_distilbert¶

Results | Participants | Input | Summary | Appendix

Run ID: baseline_distilbert
Participant: UAmsterdam
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/18/2023
MD5: 63d36419b3fd2f13b3b1887101bd7636

baseline_gpt4_db¶

Results | Participants | Input | Summary | Appendix

Run ID: baseline_gpt4_db
Participant: UAmsterdam
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/18/2023
MD5: 1f33733c4c4ef8c6b757e0dd675730c3

dpr-100-rerank¶

Run ID: dpr-100-rerank
Participant: CMU-LTI
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: 074ca0084e61515586027954eb38e098
Run description: I used the movie tip-of-the-tongue queries and answers from the Reddit ToT dataset in https://github.com/webis-de/QPP-23.

dpr-1000-rerank-robin¶

Run ID: dpr-1000-rerank-robin
Participant: CMU-LTI
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: 8607f9892803f1bc034353b7977b2e3c
Run description: I used the movie tip-of-the-tongue queries and answers from the Reddit ToT dataset in https://github.com/webis-de/QPP-23.

dpr-abstract-100-rerank¶

Run ID: dpr-abstract-100-rerank
Participant: CMU-LTI
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: ea7ef469a2032bd3cfed08b73296a71f
Run description: I split the query sentence into "Abstract sentences", i.e., sentences which I would expect to match with the Abstract of a Wikipedia document. For example, a query sentence regarding the movie release date is an Abstract sentence, since it's usually in the first paragraph of a movie wikipedia document. I used the movie tip-of-the-tongue queries and answers from the Reddit ToT dataset in https://github.com/webis-de/QPP-23

dpr-abstract-1000-robin¶

Run ID: dpr-abstract-1000-robin
Participant: CMU-LTI
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: ba1ff79c98fa997182aa5345a753e8ce
Run description: I split the query sentence into "Abstract sentences", i.e., sentences which I would expect to match with the Abstract of a Wikipedia document. For example, a query sentence regarding the movie release date is an Abstract sentence, since it's usually in the first paragraph of a movie wikipedia document. I search only the abstract sentences, only on the abstracts of the documents. I used the movie tip-of-the-tongue queries and answers from the Reddit ToT dataset in https://github.com/webis-de/QPP-23

dpr_multidoc_roberta¶

Results | Participants | Input | Summary | Appendix

Run ID: dpr_multidoc_roberta
Participant: CIIR
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 001207dc9209539bbd9a29a8203f872c

endicott_unc_baseline¶

Run ID: endicott_unc_baseline
Participant: endicott-unc
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/28/2023
MD5: 445e42a37b92bcf38a6162ca0e5539f5

endicott_unc_boost_conf¶

Run ID: endicott_unc_boost_conf
Participant: endicott-unc
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/29/2023
MD5: 4260fbfa11a97901bb378067c77871e4
Run description: We used predicted sentence annotations to boost certain sentences in the query. We used the train and dev set to predict sentence annotations using a KNN classifier. The degree of boosting was proportional to the classifier's predicted confidence value.

endicott_unc_boost_oracle¶

Run ID: endicott_unc_boost_oracle
Participant: endicott-unc
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/29/2023
MD5: 407cacb5d65f6ecc17a0303553ac63ac
Run description: We used the gold standard sentence annotations to boost certain sentences in the query.

endicott_unc_boost_pred¶

Run ID: endicott_unc_boost_pred
Participant: endicott-unc
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/29/2023
MD5: 097ad177d21dd0e521e7d5bf1c1392c6
Run description: We used predicted sentence annotations to boost certain sentences in the query. We used the train and dev set to predict sentence annotations using a KNN classifier.

pre_aug_vat¶

Run ID: pre_aug_vat
Participant: snuldilab
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: ea5675a62ceeeb76ed7de1ff474c0cb6
Run description: During training, we used them to augment data by cropping unnecessary information. We followed the paper [1] to select what is 'unnecessary'- annotations that lower the model performance are cropped. During test, we removed unnecessary sentences in the query. [1] Jaime Arguello, Adam Ferguson, Emery Fine, Bhaskar Mitra, Hamed Zamani, and Fernando Diaz. Tip of the tongue known-item retrieval: A case study in movie identification. In Proc. ACM CHIIR21, pp. 5-14. 2021. english wikipedia and bookcorpus (via BERT-base)

pre_aug_vat_max4¶

Run ID: pre_aug_vat_max4
Participant: snuldilab
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 8c65c7a17badadea0ee24925282773fc
Run description: During training, we used them to augment data by cropping unnecessary information. We followed the paper [1] to select what is 'unnecessary'- annotations that lower the model performance are cropped. During test, we removed unnecessary sentences in the query. [1] Jaime Arguello, Adam Ferguson, Emery Fine, Bhaskar Mitra, Hamed Zamani, and Fernando Diaz. Tip of the tongue known-item retrieval: A case study in movie identification. In Proc. ACM CHIIR21, pp. 5-14. 2021. I used the BERT-base model as my backbone model, so English Wikipedia and Bookcorpus are used as pretraining data.

pre_aug_vat_max4_origin¶

Run ID: pre_aug_vat_max4_origin
Participant: snuldilab
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 2ddbafaf92431af7aa34767bc06975f1
Run description: During training, we used them to augment data by cropping unnecessary information. We followed the paper [1] to select what is 'unnecessary'- annotations that lower the model performance are cropped. [1] Jaime Arguello, Adam Ferguson, Emery Fine, Bhaskar Mitra, Hamed Zamani, and Fernando Diaz. Tip of the tongue known-item retrieval: A case study in movie identification. In Proc. ACM CHIIR21, pp. 5-14. 2021. I used the BERT-base model as my backbone model, so English Wikipedia and Bookcorpus are used as pretraining data.

RSLTOTY¶

Run ID: RSLTOTY
Participant: RSLTOT
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/31/2023
MD5: 7207c66f914d009749fe4f913802f702
Run description: bert-base-uncased, IMDb website

runid1¶

Results | Participants | Input | Summary | Appendix

Run ID: runid1
Participant: WaterlooClarke
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: c18feb336ab8b3c2ff33c27d997ce576

ufmgDBmBdTQD¶

Run ID: ufmgDBmBdTQD
Participant: ufmg
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 8afe65defc21368b34d030f60b93c30c
Run description: I removed sentences marked as social from the queries

ufmgDBmBQ¶

Run ID: ufmgDBmBQ
Participant: ufmg
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: d5268c2ac9c67746e9ee6aa38ce4d003
Run description: I removed sentences marked as social from the queries

ufmgDBmBQD¶

Run ID: ufmgDBmBQD
Participant: ufmg
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 3a7108f3b5561b570f15dc64866594e0
Run description: I removed sentences markes as social from the queries

ufmgG4dTQD¶

Run ID: ufmgG4dTQD
Participant: ufmg
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: f1ec59ca5cae9bca607fe1072eed97b3
Run description: I removed sentences marked as social from the queries

ufmgG4mBQD¶

Run ID: ufmgG4mBQD
Participant: ufmg
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: a1ac8563ffe879b8d57558f8204c1f4d
Run description: I removed sentences marked as social from the queries

WatS-DR¶

Run ID: WatS-DR
Participant: UWaterlooMDS
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 9e5de5c0c549c6050a5249c946a0fcd6

WatS-TDR¶

Run ID: WatS-TDR
Participant: UWaterlooMDS
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 143d5b8f054445adedfeca14b11204d3

WatS-TDR-RR¶

Run ID: WatS-TDR-RR
Participant: UWaterlooMDS
Track: Tip-of-the-Tongue
Year: 2023
Submission: 9/1/2023
MD5: 48c0a3ca53709bd8ef07547a983bb291

webis-bm25r-1¶

Run ID: webis-bm25r-1
Participant: Webis
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: 0a33b07a4897fcb69e8ca2cf01576b90
Run description: We used the TOMT-KIS dataset (tip-of-my-tongue known-item search: https://webis.de/downloads/publications/papers/froebe_2023c.pdf) to train deepct for long query reduction. We removed questions from the MS-TOT dataset, but we did not checked if other questions that are not in the MS-TOT dataset but in our dataset would link to the same known-item.

webis-fus-01¶

Run ID: webis-fus-01
Participant: Webis
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: 023f1bd6b69db03d6a66ab0c190db053

webis-t5-01¶

Run ID: webis-t5-01
Participant: Webis
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: 72f5e92fed7462a7b554d3e061319edd

webis-t5-f¶

Run ID: webis-t5-f
Participant: Webis
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/30/2023
MD5: 33036bbf144b9d0a6c9a7ced52cf201a

webis-t53b-01¶

Run ID: webis-t53b-01
Participant: Webis
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/31/2023
MD5: a93495f7f0ef05ee93cae3e5bce5a04d

WIS_DB_FT¶

Results | Participants | Input | Summary | Appendix

Run ID: WIS_DB_FT
Participant: WIS_TUD
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/31/2023
MD5: 33fa8de742fb0f3d45005388afb888c8
Run description: https://github.com/samarthbhargav/tomt-data

WIS_LSR_SPLADE_ASM_QMLP¶

Results | Participants | Input | Summary | Appendix

Run ID: WIS_LSR_SPLADE_ASM_QMLP
Participant: WIS_TUD
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/31/2023
MD5: 218b03486fea34896bd5c144587a5c76
Run description: https://github.com/samarthbhargav/tomt-data

WIS_LSR_UNICOIL¶

Results | Participants | Input | Summary | Appendix

Run ID: WIS_LSR_UNICOIL
Participant: WIS_TUD
Track: Tip-of-the-Tongue
Year: 2023
Submission: 8/31/2023
MD5: f25e9fb2174ff11d1a352171261c65c8
Run description: https://github.com/samarthbhargav/tomt-data