Results - Conversational Assistance 2022¶

MLIA_DAC_splade¶

summary (lenient)

map             all 0.1948
bpref           all 0.3495
recip_rank      all 0.6576
P_10            all 0.4485
recall_10       all 0.0727
recall_100      all 0.2894
recall_1000     all 0.6136
ndcg            all 0.4327
ndcg_cut_10     all 0.3172

summary (strict)

map             all 0.1619
bpref           all 0.2406
recip_rank      all 0.5143
P_10            all 0.3043
recall_10       all 0.0884
recall_100      all 0.3108
recall_1000     all 0.6384
ndcg            all 0.4327
ndcg_cut_10     all 0.3172

splade_t5mse¶

summary (lenient)

map             all 0.1514
bpref           all 0.3263
recip_rank      all 0.5153
P_10            all 0.3828
recall_10       all 0.0576
recall_100      all 0.2266
recall_1000     all 0.6136
ndcg            all 0.3933
ndcg_cut_10     all 0.2601

summary (strict)

map             all 0.1270
bpref           all 0.2220
recip_rank      all 0.4101
P_10            all 0.2626
recall_10       all 0.0651
recall_100      all 0.2504
recall_1000     all 0.6384
ndcg            all 0.3933
ndcg_cut_10     all 0.2601

splade_t5mm_ens¶

summary (lenient)

map             all 0.2594
bpref           all 0.3850
recip_rank      all 0.7276
P_10            all 0.5497
recall_10       all 0.0940
recall_100      all 0.3758
recall_1000     all 0.6136
ndcg            all 0.4832
ndcg_cut_10     all 0.3990

summary (strict)

map             all 0.2193
bpref           all 0.2855
recip_rank      all 0.5923
P_10            all 0.3877
recall_10       all 0.1173
recall_100      all 0.4038
recall_1000     all 0.6384
ndcg            all 0.4832
ndcg_cut_10     all 0.3990

splade_t5mm¶

summary (lenient)

map             all 0.2398
bpref           all 0.3736
recip_rank      all 0.6926
P_10            all 0.5227
recall_10       all 0.0874
recall_100      all 0.3530
recall_1000     all 0.6136
ndcg            all 0.4704
ndcg_cut_10     all 0.3790

summary (strict)

map             all 0.2018
bpref           all 0.2717
recip_rank      all 0.5742
P_10            all 0.3638
recall_10       all 0.1068
recall_100      all 0.3822
recall_1000     all 0.6384
ndcg            all 0.4704
ndcg_cut_10     all 0.3790

DEI-run1¶

summary (lenient)

map             all 0.1483
bpref           all 0.2661
recip_rank      all 0.5926
P_10            all 0.4147
recall_10       all 0.0716
recall_100      all 0.2694
recall_1000     all 0.4262
ndcg            all 0.3312
ndcg_cut_10     all 0.2845

summary (strict)

map             all 0.1294
bpref           all 0.1979
recip_rank      all 0.4696
P_10            all 0.2828
recall_10       all 0.0831
recall_100      all 0.2922
recall_1000     all 0.4452
ndcg            all 0.3312
ndcg_cut_10     all 0.2845

DEI-run2¶

summary (lenient)

map             all 0.1505
bpref           all 0.2663
recip_rank      all 0.5666
P_10            all 0.4104
recall_10       all 0.0726
recall_100      all 0.2777
recall_1000     all 0.4393
ndcg            all 0.3368
ndcg_cut_10     all 0.2777

summary (strict)

map             all 0.1286
bpref           all 0.1976
recip_rank      all 0.4433
P_10            all 0.2785
recall_10       all 0.0853
recall_100      all 0.3046
recall_1000     all 0.4584
ndcg            all 0.3368
ndcg_cut_10     all 0.2777

uis_duoboat¶

summary (lenient)

map             all 0.1718
bpref           all 0.2570
recip_rank      all 0.6172
P_10            all 0.4577
recall_10       all 0.0809
recall_100      all 0.2518
recall_1000     all 0.3694
ndcg            all 0.3195
ndcg_cut_10     all 0.3336

summary (strict)

map             all 0.1514
bpref           all 0.2046
recip_rank      all 0.4767
P_10            all 0.3233
recall_10       all 0.1002
recall_100      all 0.2599
recall_1000     all 0.3646
ndcg            all 0.3195
ndcg_cut_10     all 0.3336

UWCmanual22¶

summary (lenient)

map             all 0.3252
bpref           all 0.4519
recip_rank      all 0.7956
P_10            all 0.6644
recall_10       all 0.1244
recall_100      all 0.4739
recall_1000     all 0.6456
ndcg            all 0.5497
ndcg_cut_10     all 0.4999

summary (strict)

map             all 0.2936
bpref           all 0.3681
recip_rank      all 0.7111
P_10            all 0.4975
recall_10       all 0.1638
recall_100      all 0.5251
recall_1000     all 0.6756
ndcg            all 0.5497
ndcg_cut_10     all 0.4999

UWCcano22¶

summary (lenient)

map             all 0.2453
bpref           all 0.3640
recip_rank      all 0.7284
P_10            all 0.5577
recall_10       all 0.0947
recall_100      all 0.3665
recall_1000     all 0.5341
ndcg            all 0.4453
ndcg_cut_10     all 0.4051

summary (strict)

map             all 0.2153
bpref           all 0.2856
recip_rank      all 0.6243
P_10            all 0.4018
recall_10       all 0.1238
recall_100      all 0.3975
recall_1000     all 0.5564
ndcg            all 0.4453
ndcg_cut_10     all 0.4051

UWCauto22¶

summary (lenient)

map             all 0.2351
bpref           all 0.3375
recip_rank      all 0.6791
P_10            all 0.5012
recall_10       all 0.0861
recall_100      all 0.3453
recall_1000     all 0.5044
ndcg            all 0.4161
ndcg_cut_10     all 0.3644

summary (strict)

map             all 0.2063
bpref           all 0.2640
recip_rank      all 0.5658
P_10            all 0.3632
recall_10       all 0.1107
recall_100      all 0.3751
recall_1000     all 0.5171
ndcg            all 0.4161
ndcg_cut_10     all 0.3644

DEI-run4¶

summary (lenient)

map             all 0.1505
bpref           all 0.2666
recip_rank      all 0.5852
P_10            all 0.4178
recall_10       all 0.0744
recall_100      all 0.2769
recall_1000     all 0.4398
ndcg            all 0.3373
ndcg_cut_10     all 0.2805

summary (strict)

map             all 0.1281
bpref           all 0.1974
recip_rank      all 0.4508
P_10            all 0.2798
recall_10       all 0.0868
recall_100      all 0.2970
recall_1000     all 0.4560
ndcg            all 0.3373
ndcg_cut_10     all 0.2805

monot5¶

summary (lenient)

map             all 0.1390
bpref           all 0.2459
recip_rank      all 0.6103
P_10            all 0.4006
recall_10       all 0.0712
recall_100      all 0.2159
recall_1000     all 0.4467
ndcg            all 0.3146
ndcg_cut_10     all 0.2874

summary (strict)

map             all 0.1172
bpref           all 0.1833
recip_rank      all 0.4925
P_10            all 0.2718
recall_10       all 0.0796
recall_100      all 0.2224
recall_1000     all 0.4531
ndcg            all 0.3146
ndcg_cut_10     all 0.2874

duo_reranker¶

summary (lenient)

map             all 0.2003
bpref           all 0.2823
recip_rank      all 0.7752
P_10            all 0.5687
recall_10       all 0.1030
recall_100      all 0.2939
recall_1000     all 0.4467
ndcg            all 0.3798
ndcg_cut_10     all 0.4112

summary (strict)

map             all 0.1787
bpref           all 0.2333
recip_rank      all 0.6382
P_10            all 0.4025
recall_10       all 0.1280
recall_100      all 0.3080
recall_1000     all 0.4531
ndcg            all 0.3798
ndcg_cut_10     all 0.4112

gold¶

summary (lenient)

map             all 0.2174
bpref           all 0.3371
recip_rank      all 0.7314
P_10            all 0.5540
recall_10       all 0.0999
recall_100      all 0.3510
recall_1000     all 0.5292
ndcg            all 0.4205
ndcg_cut_10     all 0.3829

summary (strict)

map             all 0.1787
bpref           all 0.2543
recip_rank      all 0.5745
P_10            all 0.3791
recall_10       all 0.1187
recall_100      all 0.3638
recall_1000     all 0.5566
ndcg            all 0.4205
ndcg_cut_10     all 0.3829

combine0.5¶

summary (lenient)

map             all 0.1974
bpref           all 0.2817
recip_rank      all 0.7643
P_10            all 0.5595
recall_10       all 0.1013
recall_100      all 0.2939
recall_1000     all 0.4467
ndcg            all 0.3765
ndcg_cut_10     all 0.3993

summary (strict)

map             all 0.1737
bpref           all 0.2295
recip_rank      all 0.6403
P_10            all 0.3908
recall_10       all 0.1256
recall_100      all 0.3080
recall_1000     all 0.4531
ndcg            all 0.3765
ndcg_cut_10     all 0.3993

CNC_AS-C¶

summary (lenient)

map             all 0.1705
bpref           all 0.3064
recip_rank      all 0.5015
P_10            all 0.4067
recall_10       all 0.0648
recall_100      all 0.2888
recall_1000     all 0.4978
ndcg            all 0.3688
ndcg_cut_10     all 0.2645

summary (strict)

map             all 0.1413
bpref           all 0.2203
recip_rank      all 0.3659
P_10            all 0.2724
recall_10       all 0.0821
recall_100      all 0.3214
recall_1000     all 0.5153
ndcg            all 0.3688
ndcg_cut_10     all 0.2645

CNC_AD-C¶

summary (lenient)

map             all 0.1244
bpref           all 0.2224
recip_rank      all 0.6231
P_10            all 0.4466
recall_10       all 0.0708
recall_100      all 0.2220
recall_1000     all 0.3096
ndcg            all 0.2857
ndcg_cut_10     all 0.3152

summary (strict)

map             all 0.1092
bpref           all 0.1771
recip_rank      all 0.4871
P_10            all 0.3018
recall_10       all 0.0842
recall_100      all 0.2423
recall_1000     all 0.3204
ndcg            all 0.2857
ndcg_cut_10     all 0.3152

CNC_MS-C¶

summary (lenient)

map             all 0.2945
bpref           all 0.4377
recip_rank      all 0.7043
P_10            all 0.5816
recall_10       all 0.1149
recall_100      all 0.4492
recall_1000     all 0.6603
ndcg            all 0.5371
ndcg_cut_10     all 0.4258

summary (strict)

map             all 0.2601
bpref           all 0.3421
recip_rank      all 0.5926
P_10            all 0.4325
recall_10       all 0.1563
recall_100      all 0.5120
recall_1000     all 0.7020
ndcg            all 0.5371
ndcg_cut_10     all 0.4258

CNC_MD-C¶

summary (lenient)

map             all 0.1753
bpref           all 0.2608
recip_rank      all 0.7999
P_10            all 0.6123
recall_10       all 0.1081
recall_100      all 0.2719
recall_1000     all 0.3223
ndcg            all 0.3504
ndcg_cut_10     all 0.4603

summary (strict)

map             all 0.1630
bpref           all 0.2330
recip_rank      all 0.7055
P_10            all 0.4380
recall_10       all 0.1364
recall_100      all 0.2950
recall_1000     all 0.3393
ndcg            all 0.3504
ndcg_cut_10     all 0.4603

DEI-run5.json¶

summary (lenient)

map             all 0.1355
bpref           all 0.2490
recip_rank      all 0.5350
P_10            all 0.3963
recall_10       all 0.0711
recall_100      all 0.2477
recall_1000     all 0.4077
ndcg            all 0.3120
ndcg_cut_10     all 0.2679

summary (strict)

map             all 0.1172
bpref           all 0.1854
recip_rank      all 0.4362
P_10            all 0.2638
recall_10       all 0.0832
recall_100      all 0.2644
recall_1000     all 0.4204
ndcg            all 0.3120
ndcg_cut_10     all 0.2679

CNC_AD¶

summary (lenient)

map             all 0.1344
bpref           all 0.2306
recip_rank      all 0.6446
P_10            all 0.4626
recall_10       all 0.0793
recall_100      all 0.2235
recall_1000     all 0.3096
ndcg            all 0.2939
ndcg_cut_10     all 0.3258

summary (strict)

map             all 0.1174
bpref           all 0.1838
recip_rank      all 0.5165
P_10            all 0.3067
recall_10       all 0.0947
recall_100      all 0.2409
recall_1000     all 0.3204
ndcg            all 0.2939
ndcg_cut_10     all 0.3258

CNC_AS¶

summary (lenient)

map             all 0.2103
bpref           all 0.3302
recip_rank      all 0.6724
P_10            all 0.4730
recall_10       all 0.0809
recall_100      all 0.3163
recall_1000     all 0.5131
ndcg            all 0.4111
ndcg_cut_10     all 0.3455

summary (strict)

map             all 0.1835
bpref           all 0.2490
recip_rank      all 0.5587
P_10            all 0.3301
recall_10       all 0.1028
recall_100      all 0.3428
recall_1000     all 0.5270
ndcg            all 0.4111
ndcg_cut_10     all 0.3455

uis_cargoboat¶

summary (lenient)

map             all 0.2000
bpref           all 0.3019
recip_rank      all 0.6537
P_10            all 0.4859
recall_10       all 0.0874
recall_100      all 0.2917
recall_1000     all 0.4521
ndcg            all 0.3705
ndcg_cut_10     all 0.3534

summary (strict)

map             all 0.1746
bpref           all 0.2298
recip_rank      all 0.5259
P_10            all 0.3399
recall_10       all 0.1075
recall_100      all 0.3077
recall_1000     all 0.4504
ndcg            all 0.3705
ndcg_cut_10     all 0.3534

CNR_run1¶

summary (lenient)

map             all 0.0796
bpref           all 0.1733
recip_rank      all 0.4242
P_10            all 0.2571
recall_10       all 0.0455
recall_100      all 0.1494
recall_1000     all 0.3239
ndcg            all 0.2187
ndcg_cut_10     all 0.1674

summary (strict)

map             all 0.0593
bpref           all 0.1116
recip_rank      all 0.2735
P_10            all 0.1534
recall_10       all 0.0460
recall_100      all 0.1553
recall_1000     all 0.3334
ndcg            all 0.2187
ndcg_cut_10     all 0.1674

CNR_run2¶

summary (lenient)

map             all 0.0951
bpref           all 0.2005
recip_rank      all 0.4723
P_10            all 0.2969
recall_10       all 0.0551
recall_100      all 0.1792
recall_1000     all 0.3746
ndcg            all 0.2555
ndcg_cut_10     all 0.1957

summary (strict)

map             all 0.0758
bpref           all 0.1355
recip_rank      all 0.3378
P_10            all 0.1859
recall_10       all 0.0609
recall_100      all 0.1820
recall_1000     all 0.3822
ndcg            all 0.2555
ndcg_cut_10     all 0.1957

uis_sparseboat¶

summary (lenient)

map             all 0.2005
bpref           all 0.3144
recip_rank      all 0.6740
P_10            all 0.4957
recall_10       all 0.0884
recall_100      all 0.2914
recall_1000     all 0.5024
ndcg            all 0.3983
ndcg_cut_10     all 0.3707

summary (strict)

map             all 0.1779
bpref           all 0.2355
recip_rank      all 0.5669
P_10            all 0.3546
recall_10       all 0.1162
recall_100      all 0.3097
recall_1000     all 0.5070
ndcg            all 0.3983
ndcg_cut_10     all 0.3707

CNR_run3¶

summary (lenient)

map             all 0.0867
bpref           all 0.1894
recip_rank      all 0.4881
P_10            all 0.2693
recall_10       all 0.0457
recall_100      all 0.1529
recall_1000     all 0.3326
ndcg            all 0.2313
ndcg_cut_10     all 0.1806

summary (strict)

map             all 0.0679
bpref           all 0.1265
recip_rank      all 0.3395
P_10            all 0.1620
recall_10       all 0.0475
recall_100      all 0.1534
recall_1000     all 0.3321
ndcg            all 0.2313
ndcg_cut_10     all 0.1806

CNR_run4¶

summary (lenient)

map             all 0.0843
bpref           all 0.1860
recip_rank      all 0.4855
P_10            all 0.2699
recall_10       all 0.0479
recall_100      all 0.1481
recall_1000     all 0.3073
ndcg            all 0.2235
ndcg_cut_10     all 0.1888

summary (strict)

map             all 0.0717
bpref           all 0.1297
recip_rank      all 0.3667
P_10            all 0.1712
recall_10       all 0.0559
recall_100      all 0.1553
recall_1000     all 0.3163
ndcg            all 0.2235
ndcg_cut_10     all 0.1888

uis_mixedboat¶

summary (lenient)

map             all 0.2121
bpref           all 0.3084
recip_rank      all 0.6436
P_10            all 0.4933
recall_10       all 0.0866
recall_100      all 0.3063
recall_1000     all 0.4379
ndcg            all 0.3756
ndcg_cut_10     all 0.3628

summary (strict)

map             all 0.1880
bpref           all 0.2480
recip_rank      all 0.4988
P_10            all 0.3540
recall_10       all 0.1116
recall_100      all 0.3301
recall_1000     all 0.4449
ndcg            all 0.3756
ndcg_cut_10     all 0.3628

udinfo_best2021¶

summary (lenient)

map             all 0.2111
bpref           all 0.3558
recip_rank      all 0.6174
P_10            all 0.4436
recall_10       all 0.0825
recall_100      all 0.3375
recall_1000     all 0.6577
ndcg            all 0.4653
ndcg_cut_10     all 0.3232

summary (strict)

map             all 0.1811
bpref           all 0.2547
recip_rank      all 0.5139
P_10            all 0.3233
recall_10       all 0.1087
recall_100      all 0.3801
recall_1000     all 0.6811
ndcg            all 0.4653
ndcg_cut_10     all 0.3232

udinfo_mi_b2021¶

summary (lenient)

map             all 0.2816
bpref           all 0.4311
recip_rank      all 0.7499
P_10            all 0.5472
recall_10       all 0.1033
recall_100      all 0.4141
recall_1000     all 0.7490
ndcg            all 0.5565
ndcg_cut_10     all 0.4166

summary (strict)

map             all 0.2461
bpref           all 0.3169
recip_rank      all 0.6559
P_10            all 0.4055
recall_10       all 0.1352
recall_100      all 0.4551
recall_1000     all 0.7709
ndcg            all 0.5565
ndcg_cut_10     all 0.4166

udinfo_onlyd¶

summary (lenient)

map             all 0.2013
bpref           all 0.3516
recip_rank      all 0.6580
P_10            all 0.4613
recall_10       all 0.0849
recall_100      all 0.3229
recall_1000     all 0.6240
ndcg            all 0.4526
ndcg_cut_10     all 0.3399

summary (strict)

map             all 0.1780
bpref           all 0.2587
recip_rank      all 0.5309
P_10            all 0.3405
recall_10       all 0.1143
recall_100      all 0.3645
recall_1000     all 0.6505
ndcg            all 0.4526
ndcg_cut_10     all 0.3399

udinfo_onlyd_mi¶

summary (lenient)

map             all 0.2771
bpref           all 0.4326
recip_rank      all 0.7520
P_10            all 0.5613
recall_10       all 0.1047
recall_100      all 0.4081
recall_1000     all 0.7046
ndcg            all 0.5402
ndcg_cut_10     all 0.4280

summary (strict)

map             all 0.2432
bpref           all 0.3233
recip_rank      all 0.6456
P_10            all 0.4221
recall_10       all 0.1380
recall_100      all 0.4493
recall_1000     all 0.7290
ndcg            all 0.5402
ndcg_cut_10     all 0.4280

uogTr-MI-HB¶

summary (lenient)

map             all 0.1700
bpref           all 0.2698
recip_rank      all 0.6129
P_10            all 0.4620
recall_10       all 0.0781
recall_100      all 0.2781
recall_1000     all 0.3815
ndcg            all 0.3365
ndcg_cut_10     all 0.3289

summary (strict)

map             all 0.1516
bpref           all 0.2160
recip_rank      all 0.5028
P_10            all 0.3313
recall_10       all 0.1019
recall_100      all 0.3074
recall_1000     all 0.4130
ndcg            all 0.3365
ndcg_cut_10     all 0.3289

uogTr-AT¶

summary (lenient)

map             all 0.1448
bpref           all 0.2283
recip_rank      all 0.5869
P_10            all 0.4552
recall_10       all 0.0791
recall_100      all 0.2361
recall_1000     all 0.3001
ndcg            all 0.2902
ndcg_cut_10     all 0.3167

summary (strict)

map             all 0.1343
bpref           all 0.1955
recip_rank      all 0.4942
P_10            all 0.3233
recall_10       all 0.0999
recall_100      all 0.2548
recall_1000     all 0.3189
ndcg            all 0.2902
ndcg_cut_10     all 0.3167

uogTr-MT¶

summary (lenient)

map             all 0.2669
bpref           all 0.3717
recip_rank      all 0.8060
P_10            all 0.6209
recall_10       all 0.1181
recall_100      all 0.4097
recall_1000     all 0.5118
ndcg            all 0.4768
ndcg_cut_10     all 0.4733

summary (strict)

map             all 0.2489
bpref           all 0.3180
recip_rank      all 0.6954
P_10            all 0.4669
recall_10       all 0.1621
recall_100      all 0.4606
recall_1000     all 0.5534
ndcg            all 0.4768
ndcg_cut_10     all 0.4733