Runs - Deep Learning 2023¶

agg-cocondenser¶

Run ID: agg-cocondenser
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 35fa5aa6b12b7456f99e408be96fe333
Run description: Aggretriever is a dense retriever with semantic and lexical matching. We initialize with coCondenser and train with official MS MARCO training queries (with BM25 hard negatives) with a batch size of 64 for 3 epochs on single GPU. Detail is described in https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00556/116046/Aggretriever-A-Simple-Approach-to-Aggregate

bm25_splades¶

Run ID: bm25_splades
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 26123ed5ab46fe0d76f59f80fb5a3474
Run description: Ensemble of BM25 + SPLADE++SD + SPLADE++ED

cip_run_1¶

Run ID: cip_run_1
Participant: CIP
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: c70237e7dcd929608deeed04828ec743
Run description: Pipeline: 1. Unicoil+doct5query (pyserini) 2. MonoT5-3b (pygaggle) 3. GPT-3.5-Turbo top-50 prf reranking 4. GPT-3.5-Turbo top-40 reranking 5. GPT-3.5-Turbo top-30 reranking with different window size 6. GPT-4 top-20 reranking

cip_run_2¶

Run ID: cip_run_2
Participant: CIP
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 0c5ffb8832c3717998bd51be105fe8c8
Run description: Pipeline: 1. Unicoil+doct5query (pyserini) 2. MonoT5-3b (pygaggle) 3. GPT-3.5-Turbo top-50 prf reranking 4. GPT-3.5-Turbo top-40 reranking 5. GPT-3.5-Turbo top-30 reranking with different window size 6. GPT-4 top-20 reranking 7. Score reassignment

cip_run_3¶

Run ID: cip_run_3
Participant: CIP
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 344f93a132d35f12b38b98a10c08ac89
Run description: Pipeline: 1. Unicoil+doct5query (pyserini) 2. MonoT5-3b (pygaggle) 3. GPT-3.5-Turbo top-50 prf reranking 4. GPT-3.5-Turbo top-40 reranking 5. GPT-3.5-Turbo top-30 reranking with different window size (fuse)

cip_run_4¶

Run ID: cip_run_4
Participant: CIP
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: cc884c1191cb6b6ecff0a587fb022b19
Run description: Pipeline: 1. Unicoil+doct5query (pyserini) 2. MonoT5-3b (pygaggle) 3. GPT-3.5-Turbo top-50 prf reranking 4. GPT-3.5-Turbo top-40 reranking

cip_run_5¶

Run ID: cip_run_5
Participant: CIP
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 2e5292385957103e82605e72ae9a9a5c
Run description: Pipeline: 1. Unicoil+doct5query (pyserini) 2. MonoT5-3b (pygaggle) 3. GPT-3.5-Turbo top-50 prf reranking 4. GPT-3.5-Turbo top-40 reranking 5. GPT-3.5-Turbo top-30 reranking with different window size

cip_run_6¶

Run ID: cip_run_6
Participant: CIP
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 21f45a56330d46b4644512a15d3a777e
Run description: Pipeline: 1. Unicoil+doct5query (pyserini) 2. MonoT5-3b (pygaggle) 3. GPT-3.5-Turbo top-50 prf reranking

cip_run_7¶

Run ID: cip_run_7
Participant: CIP
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: e9f8efe39405bb5a98a91baa1e69a739
Run description: Pipeline: 1. Unicoil+doct5query (pyserini) 2. MonoT5-3b (pygaggle) 3. GPT-3.5-Turbo top-50 prf reranking 4. GPT-3.5-Turbo top-40 reranking 5. GPT-3.5-Turbo top-30 reranking with different window size

colbertv2¶

Results | Participants | Input | Summary | Appendix

Run ID: colbertv2
Participant: InfoSense
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: docs
MD5: e068c29e4d8e8cbb276da4a9606e3bc2
Run description: Using Colbert-v2 and using pre-trained checkpoint and do continue training on MS MARCO v2 training data

D_bm25_splades¶

Run ID: D_bm25_splades
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: docs
MD5: 31d0e3e09e1326ecac424b2c29574cdf
Run description: Ensemble of BM25 + SPLADE++SD + SPLADE++ED

D_naverloo-frgpt4¶

Run ID: D_naverloo-frgpt4
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: docs
MD5: 3290f826729d8e377e4a0bf89b99fa99
Run description: First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever Second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro Third step is an ensemble of 3 duo rankers over the top50 duoT5 PRP-FlanT5-3b PRP-FlanT5-UL2 Fourth step is RankGPT4 over the top30, which is then ensemble with the 3rd stage

D_naverloo_bm25_RR¶

Run ID: D_naverloo_bm25_RR
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: docs
MD5: dcd5a5ef98526d1d0627b02380587262
Run description: First stage is BM25, second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro

D_naverloo_bm_splade_RR¶

Run ID: D_naverloo_bm_splade_RR
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: docs
MD5: 8fd7ab922d5206f3bc095a6de51c15aa
Run description: First stage is BM25+SPLADE++ED+SPLADE++SD, second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro

naverloo-frgpt4¶

Run ID: naverloo-frgpt4
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 4f04190938971d286d1edf8f09c215d2
Run description: First stage is a very large ensemble of BM25+DOCT5+SPLADEPP+SPLADESD+AGG+SLIM, second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro We then do a third step with an ensemble of 3 duo rankers over the top50, duoT5, PRP-FlanT5-3b and PRP-FlanT5-UL2. We finally finish by applying RankGPT4 over the top30 and ensembling with the previous step.

naverloo-rgpt4¶

Run ID: naverloo-rgpt4
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: ae6ba03d1aaf61f3aeecb3b4d37a510f
Run description: First stage is a very large ensemble of BM25+DOCT5+SPLADEPP+SPLADESD+AGG+SLIM, second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro We then do a third step with an ensemble of 3 duo rankers over the top50, duoT5, PRP-FlanT5-3b and PRP-FlanT5-UL2. We finally finish by applying RankGPT4 over the top30.

naverloo_bm25_RR¶

Run ID: naverloo_bm25_RR
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 55553801dd8862dffec6d75a6b6a0fe0
Run description: First stage is BM25, second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro

naverloo_bm25_splades_RR¶

Run ID: naverloo_bm25_splades_RR
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 64c59aa9bba0142857376ef7ff03d3e6
Run description: First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED Second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro

naverloo_fs¶

Run ID: naverloo_fs
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 1331c0c525a626c61a7df6a41c4db51d
Run description: Ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever

naverloo_fs_RR¶

Run ID: naverloo_fs_RR
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: bda95ea90ed615d1fcc5c47a29b105f6
Run description: First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever. Second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro

naverloo_fs_RR_duo¶

Run ID: naverloo_fs_RR_duo
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 299dcb3e07daf943a746119917a2bc29
Run description: First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever. Second stage is an ensemble of 5 rerankers: naver/trecdl22-crossencoder-albert naver/trecdl22-crossencoder-debertav2 naver/trecdl22-crossencoder-debertav3 naver/trecdl22-crossencoder-electra naver/trecdl22-crossencoder-rankT53b-repro Third step is an ensemble of 3 duo rankers over the top50: duoT5 PRP-FlanT5-3b PRP-FlanT5-UL2

slim-pp-0shot-uw¶

Run ID: slim-pp-0shot-uw
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 631835ca0a45a3c19d28bf12929917b7
Run description: https://arxiv.org/pdf/2302.06587.pdf

splade_pp_ensemble_distil¶

Run ID: splade_pp_ensemble_distil
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 0cbab324dbfebca7faf233901c9a92d9
Run description: Splade++ Ensemble distil available on huggingface

splade_pp_self_distil¶

Run ID: splade_pp_self_distil
Participant: h2oloo
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 50f8933fbff1008c97db3ef69493fa4b
Run description: Splade++ Self distil available on huggingface

uogtr_b_grf_e¶

Run ID: uogtr_b_grf_e
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 01ec324fe73dc072c726429e7859936c
Run description: BM25 over entire msmarco-passage-v2 inverted index, Generative relevance feedback using google/flant5-xxl (8 bit quantized), reranking using crystina-z/monoELECTRA_LCE_nneg31

uogtr_b_grf_e_gb¶

Run ID: uogtr_b_grf_e_gb
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 9b16c2da7d2198c3c8a3c7afdc7b665d
Run description: Generative query expansion using google/flant5-xxl (8-bit quantized), BM25 over entire msmarco-passage-v2 inverted index, adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph

uogtr_be¶

Run ID: uogtr_be
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 4dfa2a0d7956dd5cc70146b330827355
Run description: BM25 retrieval, re-ranked using crystina-z/monoELECTRA_LCE_nneg31

uogtr_be_gb¶

Run ID: uogtr_be_gb
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: d200f64351d1efac7c775e4307c0279a
Run description: BM25 over entire msmarco-passage-v2 inverted index,adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph

uogtr_dph¶

Run ID: uogtr_dph
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 6916de76e58cac8efa85b3a61f9dce83
Run description: Performs DPH on the entire msmarco-passage-v2 inverted index.

uogtr_dph_bo1¶

Run ID: uogtr_dph_bo1
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 3d392481e3127b27969eb7265c4b8cf9
Run description: Performs DPH with Bo1 query expansion on the entire msmarco-passage-v2 inverted index.

uogtr_qr_be¶

Run ID: uogtr_qr_be
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: f9267814b5b855fc628abdb26a76c708
Run description: Generative query expansion using google/flant5-xxl (8-bit quantized), BM25 over entire msmarco-passage-v2 inverted index, reranking using crystina-z/monoELECTRA_LCE_nneg31

uogtr_qr_be_gb¶

Run ID: uogtr_qr_be_gb
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 8d457d9392e936e00dc7187a9232a277
Run description: Generative query expansion using google/flant5-xxl (8-bit quantized), BM25 over entire msmarco-passage-v2 inverted index, adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph

uogtr_s¶

Run ID: uogtr_s
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 23aec2a3cac327addba6f93b63f4ba6e
Run description: SPLADE retrieval using naver/splade-cocondenser-ensembledistil

uogtr_se¶

Run ID: uogtr_se
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 6a3200e762c38ab4b5909c809bcb65a0
Run description: SPLADE retrieval using naver/splade-cocondenser-ensembledistil, re-ranked using crystina-z/monoELECTRA_LCE_nneg31

uogtr_se_gb¶

Run ID: uogtr_se_gb
Participant: uogTr
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 9710da3e2e4a1e5da0664b8c4cd7c915
Run description: SPLADE retrieval using naver/splade-cocondenser-ensembledistil, adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph

uot-yj_LLMs-blender¶

Results | Participants | Input | Summary | Appendix

Run ID: uot-yj_LLMs-blender
Participant: uot-yj
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 061d46bce792570d42c6bfc7fca67224
Run description: This run uses multiple LLM models to judge candidate document pairs in a pair-wise approach and finally aggregates the judgments of all models to generate the final ranking result. This is a Zero-shot learning approach.

uot-yj_rankgpt35¶

Results | Participants | Input | Summary | Appendix

Run ID: uot-yj_rankgpt35
Participant: uot-yj
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 61734f54e04fb7acb387923b2a7f968b
Run description: This run utilized the GPT3.5 model to generate the re-ranking results via a List-wise approach. This is a Zero-shot learning approach.

uot-yj_rankgpt4¶

Results | Participants | Input | Summary | Appendix

Run ID: uot-yj_rankgpt4
Participant: uot-yj
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 0b8e9713e24e4be4ab2d28d91ad37278
Run description: This run utilized the GPT4 model to generate the re-ranking results via a List-wise approach. This is a Zero-shot learning approach.

WatS-Augmented-BM25¶

Run ID: WatS-Augmented-BM25
Participant: UWaterlooMDS
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: 6180f09edd151fb24f0140fd6fb94dde
Run description: Prompting a LLM to rewrite queries.

WatS-LLM-Rerank¶

Run ID: WatS-LLM-Rerank
Participant: UWaterlooMDS
Track: Deep Learning
Year: 2023
Submission: 8/1/2023
Type: auto
Task: passages
MD5: c1861e95677e9bc9b6739ed1bf964069
Run description: Prompting the llama model to assess passages.