Results - Conversational Assistance 2021¶

umd2021_run1¶

summary (trec_eval)

map             all 0.2622
bpref           all 0.3781
recip_rank      all 0.6563
P_10            all 0.4411
P_100           all 0.1477
P_1000          all 0.0203
recall_10       all 0.1639
recall_100      all 0.4624
recall_1000     all 0.6071
ndcg            all 0.4651
ndcg_cut_10     all 0.3749
ndcg_cut_100    all 0.4126

HBKU_CQR-HC¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.2507
bpref           all 0.3598
recip_rank      all 0.6351
P_10            all 0.4551
P_100           all 0.1423
P_1000          all 0.0185
recall_10       all 0.1509
recall_100      all 0.4109
recall_1000     all 0.5178
ndcg            all 0.4235
ndcg_cut_10     all 0.3785
ndcg_cut_100    all 0.3849

HBKU_CQRHC_BM25¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.2958
bpref           all 0.4023
recip_rank      all 0.7435
P_10            all 0.5127
P_100           all 0.1572
P_1000          all 0.0202
recall_10       all 0.1836
recall_100      all 0.4718
recall_1000     all 0.5782
ndcg            all 0.4912
ndcg_cut_10     all 0.4479
ndcg_cut_100    all 0.4523

HBKU_CQR_TC¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.3232
bpref           all 0.4438
recip_rank      all 0.7447
P_10            all 0.5291
P_100           all 0.1720
P_1000          all 0.0242
recall_10       all 0.1870
recall_100      all 0.5239
recall_1000     all 0.6971
ndcg            all 0.5448
ndcg_cut_10     all 0.4601
ndcg_cut_100    all 0.4833

HBKU_CQR_POS¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.2959
bpref           all 0.4081
recip_rank      all 0.7166
P_10            all 0.5203
P_100           all 0.1568
P_1000          all 0.0200
recall_10       all 0.1848
recall_100      all 0.4743
recall_1000     all 0.5789
ndcg            all 0.4877
ndcg_cut_10     all 0.4430
ndcg_cut_100    all 0.4496

clarke-manual¶

summary (trec_eval)

map             all 0.4734
bpref           all 0.6188
recip_rank      all 0.8839
P_10            all 0.6816
P_100           all 0.2278
P_1000          all 0.0307
recall_10       all 0.2525
recall_100      all 0.6956
recall_1000     all 0.8952
ndcg            all 0.7290
ndcg_cut_10     all 0.6239
ndcg_cut_100    all 0.6602

clarke-cc¶

summary (trec_eval)

map             all 0.4002
bpref           all 0.5461
recip_rank      all 0.8027
P_10            all 0.5778
P_100           all 0.2029
P_1000          all 0.0290
recall_10       all 0.2231
recall_100      all 0.6217
recall_1000     all 0.8410
ndcg            all 0.6435
ndcg_cut_10     all 0.5040
ndcg_cut_100    all 0.5641

clarke-auto¶

summary (trec_eval)

map             all 0.2796
bpref           all 0.4143
recip_rank      all 0.6204
P_10            all 0.4177
P_100           all 0.1561
P_1000          all 0.0239
recall_10       all 0.1399
recall_100      all 0.4705
recall_1000     all 0.6920
ndcg            all 0.4920
ndcg_cut_10     all 0.3578
ndcg_cut_100    all 0.4139

IITD-RAW_U_T5_1¶

summary (trec_eval)

map             all 0.1759
bpref           all 0.2346
recip_rank      all 0.6450
P_10            all 0.4323
P_100           all 0.1009
P_1000          all 0.0101
recall_10       all 0.1333
recall_100      all 0.2902
recall_1000     all 0.2902
ndcg            all 0.3029
ndcg_cut_10     all 0.3429
ndcg_cut_100    all 0.3031

IITD-RAW_U_T5_2¶

summary (trec_eval)

map             all 0.1874
bpref           all 0.2462
recip_rank      all 0.6708
P_10            all 0.4506
P_100           all 0.1066
P_1000          all 0.0107
recall_10       all 0.1400
recall_100      all 0.3014
recall_1000     all 0.3014
ndcg            all 0.3161
ndcg_cut_10     all 0.3562
ndcg_cut_100    all 0.3163

astypalaia256¶

summary (trec_eval)

map             all 0.1444
bpref           all 0.2940
recip_rank      all 0.5220
P_10            all 0.3114
P_100           all 0.0946
P_1000          all 0.0162
recall_10       all 0.1008
recall_100      all 0.2743
recall_1000     all 0.4559
ndcg            all 0.3109
ndcg_cut_10     all 0.2316
ndcg_cut_100    all 0.2428

LTI-rewriter-tc¶

summary (trec_eval)

map             all 0.2275
bpref           all 0.3219
recip_rank      all 0.6582
P_10            all 0.4291
P_100           all 0.1258
P_1000          all 0.0159
recall_10       all 0.1587
recall_100      all 0.3827
recall_1000     all 0.4591
ndcg            all 0.3875
ndcg_cut_10     all 0.3585
ndcg_cut_100    all 0.3598

umd2021_run2doc¶

summary (trec_eval)

map             all 0.2811
bpref           all 0.3885
recip_rank      all 0.6868
P_10            all 0.4506
P_100           all 0.1571
P_1000          all 0.0203
recall_10       all 0.1678
recall_100      all 0.4837
recall_1000     all 0.6009
ndcg            all 0.4786
ndcg_cut_10     all 0.3880
ndcg_cut_100    all 0.4363

umd2021_run3rrf¶

summary (trec_eval)

map             all 0.3223
bpref           all 0.4253
recip_rank      all 0.7328
P_10            all 0.4930
P_100           all 0.1796
P_1000          all 0.0246
recall_10       all 0.1824
recall_100      all 0.5483
recall_1000     all 0.7148
ndcg            all 0.5405
ndcg_cut_10     all 0.4221
ndcg_cut_100    all 0.4797

LTI-rewriter-5q¶

summary (trec_eval)

map             all 0.1788
bpref           all 0.2712
recip_rank      all 0.5489
P_10            all 0.3665
P_100           all 0.1102
P_1000          all 0.0143
recall_10       all 0.1180
recall_100      all 0.3160
recall_1000     all 0.3932
ndcg            all 0.3207
ndcg_cut_10     all 0.2919
ndcg_cut_100    all 0.2935

cqe-t5¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.3798
bpref           all 0.5091
recip_rank      all 0.7921
P_10            all 0.5835
P_100           all 0.1961
P_1000          all 0.0283
recall_10       all 0.2189
recall_100      all 0.6024
recall_1000     all 0.8290
ndcg            all 0.6230
ndcg_cut_10     all 0.4935
ndcg_cut_100    all 0.5416

cqe¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.3193
bpref           all 0.4722
recip_rank      all 0.7597
P_10            all 0.5095
P_100           all 0.1691
P_1000          all 0.0261
recall_10       all 0.1912
recall_100      all 0.5320
recall_1000     all 0.7744
ndcg            all 0.5633
ndcg_cut_10     all 0.4302
ndcg_cut_100    all 0.4742

t5¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.3181
bpref           all 0.4699
recip_rank      all 0.7165
P_10            all 0.5019
P_100           all 0.1754
P_1000          all 0.0259
recall_10       all 0.1814
recall_100      all 0.5249
recall_1000     all 0.7376
ndcg            all 0.5532
ndcg_cut_10     all 0.4297
ndcg_cut_100    all 0.4746

LTI-entity-g¶

summary (trec_eval)

map             all 0.2771
bpref           all 0.3692
recip_rank      all 0.7561
P_10            all 0.5013
P_100           all 0.1353
P_1000          all 0.0164
recall_10       all 0.1825
recall_100      all 0.4126
recall_1000     all 0.4808
ndcg            all 0.4347
ndcg_cut_10     all 0.4321
ndcg_cut_100    all 0.4112

LTI-rewriter-g¶

summary (trec_eval)

map             all 0.2277
bpref           all 0.3217
recip_rank      all 0.6584
P_10            all 0.4304
P_100           all 0.1258
P_1000          all 0.0159
recall_10       all 0.1590
recall_100      all 0.3825
recall_1000     all 0.4591
ndcg            all 0.3869
ndcg_cut_10     all 0.3593
ndcg_cut_100    all 0.3591

mono-duo-rerank¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.3988
bpref           all 0.5415
recip_rank      all 0.7745
P_10            all 0.5886
P_100           all 0.2053
P_1000          all 0.0283
recall_10       all 0.2172
recall_100      all 0.6292
recall_1000     all 0.8290
ndcg            all 0.6393
ndcg_cut_10     all 0.5219
ndcg_cut_100    all 0.5688

umd2021_run4den¶

summary (trec_eval)

map             all 0.2848
bpref           all 0.4221
recip_rank      all 0.6505
P_10            all 0.4297
P_100           all 0.1615
P_1000          all 0.0247
recall_10       all 0.1580
recall_100      all 0.5036
recall_1000     all 0.7315
ndcg            all 0.5167
ndcg_cut_10     all 0.3655
ndcg_cut_100    all 0.4358

dense_manual¶

summary (trec_eval)

map             all 0.1841
bpref           all 0.3930
recip_rank      all 0.7136
P_10            all 0.4057
P_100           all 0.1170
P_1000          all 0.0207
recall_10       all 0.1341
recall_100      all 0.3515
recall_1000     all 0.5876
ndcg            all 0.4372
ndcg_cut_10     all 0.3660
ndcg_cut_100    all 0.3491

sparse_manual¶

summary (trec_eval)

map             all 0.2599
bpref           all 0.4607
recip_rank      all 0.7251
P_10            all 0.4563
P_100           all 0.1497
P_1000          all 0.0255
recall_10       all 0.1708
recall_100      all 0.4704
recall_1000     all 0.7493
ndcg            all 0.5238
ndcg_cut_10     all 0.3858
ndcg_cut_100    all 0.4227

hybrid_manual¶

summary (trec_eval)

map             all 0.2051
bpref           all 0.4384
recip_rank      all 0.7405
P_10            all 0.4247
P_100           all 0.1220
P_1000          all 0.0248
recall_10       all 0.1418
recall_100      all 0.3677
recall_1000     all 0.7197
ndcg            all 0.4888
ndcg_cut_10     all 0.3828
ndcg_cut_100    all 0.3679

bm25_automatic¶

summary (trec_eval)

map             all 0.1899
bpref           all 0.3515
recip_rank      all 0.6140
P_10            all 0.3582
P_100           all 0.1254
P_1000          all 0.0218
recall_10       all 0.1218
recall_100      all 0.3793
recall_1000     all 0.6351
ndcg            all 0.4179
ndcg_cut_10     all 0.2931
ndcg_cut_100    all 0.3281

DPH-auto-rye¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.1518
bpref           all 0.3146
recip_rank      all 0.5074
P_10            all 0.2753
P_100           all 0.1087
P_1000          all 0.0211
recall_10       all 0.0908
recall_100      all 0.3302
recall_1000     all 0.6061
ndcg            all 0.3699
ndcg_cut_10     all 0.2185
ndcg_cut_100    all 0.2692

uogTrTCT¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.2303
bpref           all 0.3542
recip_rank      all 0.5678
P_10            all 0.3892
P_100           all 0.1385
P_1000          all 0.0193
recall_10       all 0.1321
recall_100      all 0.4105
recall_1000     all 0.5419
ndcg            all 0.4151
ndcg_cut_10     all 0.3181
ndcg_cut_100    all 0.3651

CNR-run1¶

summary (trec_eval)

map             all 0.1122
bpref           all 0.1453
recip_rank      all 0.6179
P_10            all 0.3278
P_100           all 0.0558
P_1000          all 0.0056
recall_10       all 0.1058
recall_100      all 0.1637
recall_1000     all 0.1637
ndcg            all 0.1956
ndcg_cut_10     all 0.2622
ndcg_cut_100    all 0.1956

CNR-run2¶

summary (trec_eval)

map             all 0.1218
bpref           all 0.1518
recip_rank      all 0.5980
P_10            all 0.3437
P_100           all 0.0575
P_1000          all 0.0057
recall_10       all 0.1118
recall_100      all 0.1675
recall_1000     all 0.1675
ndcg            all 0.2018
ndcg_cut_10     all 0.2732
ndcg_cut_100    all 0.2018

uogTrADT¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.3089
bpref           all 0.4177
recip_rank      all 0.7116
P_10            all 0.4861
P_100           all 0.1670
P_1000          all 0.0224
recall_10       all 0.1781
recall_100      all 0.5113
recall_1000     all 0.6529
ndcg            all 0.5037
ndcg_cut_10     all 0.4107
ndcg_cut_100    all 0.4539

uogTrMDT¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.4268
bpref           all 0.5583
recip_rank      all 0.8794
P_10            all 0.6475
P_100           all 0.2106
P_1000          all 0.0267
recall_10       all 0.2417
recall_100      all 0.6601
recall_1000     all 0.7918
ndcg            all 0.6667
ndcg_cut_10     all 0.5832
ndcg_cut_100    all 0.6210

CNR-run4¶

summary (trec_eval)

map             all 0.1279
bpref           all 0.1615
recip_rank      all 0.6008
P_10            all 0.3791
P_100           all 0.0632
P_1000          all 0.0063
recall_10       all 0.1201
recall_100      all 0.1811
recall_1000     all 0.1813
ndcg            all 0.2201
ndcg_cut_10     all 0.3050
ndcg_cut_100    all 0.2203

CNR-run3¶

summary (trec_eval)

map             all 0.1256
bpref           all 0.1585
recip_rank      all 0.6247
P_10            all 0.3652
P_100           all 0.0611
P_1000          all 0.0061
recall_10       all 0.1175
recall_100      all 0.1752
recall_1000     all 0.1757
ndcg            all 0.2218
ndcg_cut_10     all 0.3039
ndcg_cut_100    all 0.2218

DPH-manual-rye¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.0704
bpref           all 0.1226
recip_rank      all 0.5243
P_10            all 0.2456
P_100           all 0.0408
P_1000          all 0.0052
recall_10       all 0.0828
recall_100      all 0.1421
recall_1000     all 0.1775
ndcg            all 0.1718
ndcg_cut_10     all 0.2070
ndcg_cut_100    all 0.1595

historyonly¶

summary (trec_eval)

map             all 0.0915
bpref           all 0.1710
recip_rank      all 0.4269
P_10            all 0.2506
P_100           all 0.0751
P_1000          all 0.0083
recall_10       all 0.0770
recall_100      all 0.2182
recall_1000     all 0.2393
ndcg            all 0.1981
ndcg_cut_10     all 0.1876
ndcg_cut_100    all 0.1893

uogTrTDT¶

Metadata | Participants| Input| Summary| Appendix

summary (trec_eval)

map             all 0.2254
bpref           all 0.3420
recip_rank      all 0.5795
P_10            all 0.3886
P_100           all 0.1314
P_1000          all 0.0187
recall_10       all 0.1329
recall_100      all 0.4027
recall_1000     all 0.5413
ndcg            all 0.4109
ndcg_cut_10     all 0.3266
ndcg_cut_100    all 0.3613

t5colbert¶

summary (trec_eval)

map             all 0.1015
bpref           all 0.2852
recip_rank      all 0.4097
P_10            all 0.2215
P_100           all 0.0942
P_1000          all 0.0198
recall_10       all 0.0614
recall_100      all 0.2756
recall_1000     all 0.5652
ndcg            all 0.3142
ndcg_cut_10     all 0.1547
ndcg_cut_100    all 0.2044

UiS_raft¶

summary (trec_eval)

map             all 0.4083
bpref           all 0.5319
recip_rank      all 0.8586
P_10            all 0.6506
P_100           all 0.2053
P_1000          all 0.0254
recall_10       all 0.2362
recall_100      all 0.6349
recall_1000     all 0.7492
ndcg            all 0.6374
ndcg_cut_10     all 0.5738
ndcg_cut_100    all 0.5976

historyonlyKILT¶

summary (trec_eval)

map             all 0.1023
bpref           all 0.1838
recip_rank      all 0.4327
P_10            all 0.2576
P_100           all 0.0828
P_1000          all 0.0091
recall_10       all 0.0771
recall_100      all 0.2479
recall_1000     all 0.2703
ndcg            all 0.2144
ndcg_cut_10     all 0.1897
ndcg_cut_100    all 0.2049

Rewritt5_monot5¶

summary (trec_eval)

map             all 0.1898
bpref           all 0.2555
recip_rank      all 0.6586
P_10            all 0.4285
P_100           all 0.1109
P_1000          all 0.0111
recall_10       all 0.1451
recall_100      all 0.3321
recall_1000     all 0.3321
ndcg            all 0.3319
ndcg_cut_10     all 0.3594
ndcg_cut_100    all 0.3323

t5_monot5¶

summary (trec_eval)

map             all 0.1957
bpref           all 0.2588
recip_rank      all 0.6757
P_10            all 0.4430
P_100           all 0.1108
P_1000          all 0.0111
recall_10       all 0.1486
recall_100      all 0.3323
recall_1000     all 0.3323
ndcg            all 0.3369
ndcg_cut_10     all 0.3720
ndcg_cut_100    all 0.3374

t5_doc2query¶

summary (trec_eval)

map             all 0.1970
bpref           all 0.2639
recip_rank      all 0.6693
P_10            all 0.4513
P_100           all 0.1139
P_1000          all 0.0114
recall_10       all 0.1447
recall_100      all 0.3391
recall_1000     all 0.3391
ndcg            all 0.3355
ndcg_cut_10     all 0.3648
ndcg_cut_100    all 0.3357

CFDA_CLIP_MRUN1¶

summary (trec_eval)

map             all 0.4395
bpref           all 0.5743
recip_rank      all 0.8677
P_10            all 0.6506
P_100           all 0.2145
P_1000          all 0.0266
recall_10       all 0.2523
recall_100      all 0.6576
recall_1000     all 0.7750
ndcg            all 0.6693
ndcg_cut_10     all 0.5998
ndcg_cut_100    all 0.6284

CFDA_CLIP_MRUN2¶

summary (trec_eval)

map             all 0.4449
bpref           all 0.5895
recip_rank      all 0.8567
P_10            all 0.6468
P_100           all 0.2172
P_1000          all 0.0284
recall_10       all 0.2477
recall_100      all 0.6675
recall_1000     all 0.8213
ndcg            all 0.6866
ndcg_cut_10     all 0.5937
ndcg_cut_100    all 0.6316

CFDA_CLIP_ARUN1¶

summary (trec_eval)

map             all 0.3288
bpref           all 0.4491
recip_rank      all 0.7167
P_10            all 0.5139
P_100           all 0.1782
P_1000          all 0.0235
recall_10       all 0.1853
recall_100      all 0.5420
recall_1000     all 0.6800
ndcg            all 0.5392
ndcg_cut_10     all 0.4416
ndcg_cut_100    all 0.4900

CFDA_CLIP_ARUN2¶

summary (trec_eval)

map             all 0.3193
bpref           all 0.4316
recip_rank      all 0.7217
P_10            all 0.5146
P_100           all 0.1730
P_1000          all 0.0217
recall_10       all 0.1877
recall_100      all 0.5251
recall_1000     all 0.6319
ndcg            all 0.5177
ndcg_cut_10     all 0.4403
ndcg_cut_100    all 0.4800

RUIR1_TURN-FT¶

summary (trec_eval)

map             all 0.3783
bpref           all 0.5179
recip_rank      all 0.8181
P_10            all 0.6127
P_100           all 0.2015
P_1000          all 0.0256
recall_10       all 0.2162
recall_100      all 0.5997
recall_1000     all 0.7459
ndcg            all 0.6178
ndcg_cut_10     all 0.5423
ndcg_cut_100    all 0.5675

RUIR2_TURN¶

summary (trec_eval)

map             all 0.3899
bpref           all 0.5270
recip_rank      all 0.8290
P_10            all 0.6285
P_100           all 0.2032
P_1000          all 0.0256
recall_10       all 0.2237
recall_100      all 0.6041
recall_1000     all 0.7459
ndcg            all 0.6256
ndcg_cut_10     all 0.5575
ndcg_cut_100    all 0.5769

RUIR4_HIST¶

summary (trec_eval)

map             all 0.3611
bpref           all 0.5046
recip_rank      all 0.8107
P_10            all 0.5905
P_100           all 0.2003
P_1000          all 0.0256
recall_10       all 0.2127
recall_100      all 0.6029
recall_1000     all 0.7459
ndcg            all 0.5929
ndcg_cut_10     all 0.4883
ndcg_cut_100    all 0.5403