Runs - Product Search 2023
Participants
Run ID: BM25-pyserini-metadata-collection Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: a3bbaa10ec5fd6b1b9ec3719f2267182 Run description: BM25 using Pyserini standard retrieval settings on the Product search corpus with all additional metadata
BM25-pyserini-simple-collection
Participants
Run ID: BM25-pyserini-simple-collection Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 01ef208c183283f24b299808d4983379 Run description: BM25 using Pyserini standard retrieval settings on the Product search corpus without any metadata
cfdaclip_ER_A
Participants
Run ID: cfdaclip_ER_A Participant: CFDA_CLIP Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 81973864aebb560b86654cfae7aac9ae Run description: This run includes three text retrieval models, including BM25, Splade and dense retrieval (contriever). And the result was reranked by the MiniLM.
cfdaclip_ER_B
Participants
Run ID: cfdaclip_ER_B Participant: CFDA_CLIP Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 3a916a022f8dd486fc698a5343df88fc Run description: This run includes three text retrieval models, including BM25, Splade and dense retrieval (contriever). The document corpus were expanded with 5 predicted query by our fine-tuned T5-product2query model. The result was reranked by the MiniLM.
cfdaclip_MR_A
Participants
Run ID: cfdaclip_MR_A Participant: CFDA_CLIP Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: multimodal MD5: f60d0e347cca160399b40c12b5d17e30 Run description: This run includes three text retrieval models, including image retrieval (CLIP), learned sparse retrieval (Splade) and dense text retrieval (contriever). And the result was reranked by the MiniLM.
cfdaclip_MR_B
Participants
Run ID: cfdaclip_MR_B Participant: CFDA_CLIP Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: multimodal MD5: 9bf6accf1ce9463500ac14bcdba0a398 Run description: This run includes three text retrieval models, BM25, learned sparse retrieval (Splade) and dense text retrieval (contriever). The document corpus was expanded by the generated image captions. And the final result was reranked by the MiniLM.
f_gpt_rerank
Participants
Run ID: f_gpt_rerank Participant: h2oloo Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: multimodal MD5: a4f0a2d4b5154e9489be9946a592f323 Run description: fusion of splade, bm25, clip rerank by gpt-3.5-turbo
f_splade_bm25
Participants
Run ID: f_splade_bm25 Participant: h2oloo Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 0efe5f41cd692382265d0964f9bdc1a5 Run description: splade,bm25
f_splade_clip_bm25
Participants
Run ID: f_splade_clip_bm25 Participant: h2oloo Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: multimodal MD5: 7392a5771fab40b5137864cbe275d7a6 Run description: splade,clip,bm25 fusion
JBNU-1
Participants
Run ID: JBNU-1 Participant: jbnu Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: reranking MD5: 310bf02973bb0e41721a9526c8f823a2 Run description: 1. Translation for cases where the query is in a language other than English, 2. Spellchecker for handling query typos, 3. Reranking using DeBERTa deep learning: - Addition of special tokens "brand, color, size, price, material, category, model name, author, title, bullets" - Calculation of Loss using representations of [CLS] and [Special Token], along with model training - Model training through weighted computation of Loss for special tokens appearing in the query, 4. Application of score calculation for predictions "E, S, C, I".
JBNU-2
Participants
Run ID: JBNU-2 Participant: jbnu Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: reranking MD5: 233a7bcf96101d5514949d28c0f4e78d Run description: 1. Translation for cases where the query is in a language other than English, 2. Spellchecker for handling query typos, 3. Reranking using DeBERTa deep learning: - Addition of special tokens "brand, color, size, price, material, category, model name, author, title, bullets" - Calculation of Loss using representations of [CLS] and [Special Token], along with model training - Model training through weighted computation of Loss for special tokens appearing in the query - Category names as a single token 4. Application of score calculation for predictions "E, S, C, I".
JBNU-A
Participants
Run ID: JBNU-A Participant: jbnu Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 105436b70ebdd25b68e824f24c47a7d7 Run description: 1. Translation for cases where the query is not in English, 2. Spellchecker for processing query typos, 3. Creation of a candidate pool using ElasticSearch: - Checking the query's entities and applying reranking for cases where "brand, color, material, author" are present - Applying boosting in cases of ambiguous queries using ElasticSearch 4. Reranking using DeBERTa deep learning: - Addition of special tokens "brand, color, size, price, material, category, model name, author, title, bullets" - Calculation of Loss and model training using representations of [CLS] and [Special Token] - Model training by computing Loss weights for special tokens appearing in the query 5. Application of score calculation for predictions "E, S, C, I" based on the prediction.
JBNU-B
Participants
Run ID: JBNU-B Participant: jbnu Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 6defaad4c8848c1e4e2d02543ab8cdff Run description: 1. Translation for cases where the query is not in English, 2. Spellchecker for processing query typos, 3. Creation of a candidate pool using ElasticSearch: - Checking the query's entities and applying reranking for cases where "brand, color, material, author" are present - Applying boosting in cases of ambiguous queries using ElasticSearch 4. Reranking using DeBERTa deep learning: - Addition of special tokens "brand, color, size, price, material, category, model name, author, title, bullets" - Calculation of Loss and model training using representations of [CLS] and [Special Token] - Model training by computing Loss weights for special tokens appearing in the query - Category names as a single token 5. Application of score calculation for predictions "E, S, C, I" based on the prediction.
JBNU-C
Participants
Run ID: JBNU-C Participant: jbnu Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 099e1920f9aa16617c77901a42f3e96a Run description: 1. Translation for cases where the query is in a language other than English 2. Asin to title 3. pseudo relevance feedback and using ElasticSearch boost
Participants
Run ID: metadata-enhanced-all-MiniLM-L12-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 1150766acaa90fde5d2725a1b1a1d58d Run description: zero shot sentence-transformers/all-MiniLM-L12-v2e dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-all-MiniLM-L6-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 604124b705b6e1ef348e072e8d0b4cc7 Run description: zero shot all-MiniLM-L6-v2 dense retrieval using the enhanced (metadata)
Participants
Run ID: metadata-enhanced-all-mpnet-base-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 30548fe29ebd5cd16ad614636607d3f1 Run description: zero shot mpnet dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-bge-base-en-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 810f28c087a7502756c3c1407e633c89 Run description: zero shot bge-base-en dense retrieval using the enhanced (metadata)
Participants
Run ID: metadata-enhanced-bge-large-en-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 361f7665ff4985debf1286bcc76fadfe Run description: zero shot bge-large-en dense retrieval using the enhanced (metadata)
Participants
Run ID: metadata-enhanced-contriever-base-msmarco Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 001ef3e6864320166565996983174f61 Run description: zero shot contriever dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-e5-base-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: edf44477a285637c67457ac2c43393eb Run description: zero shot e5-base-v2 dense retrieval using the enhanced (metadata)
Participants
Run ID: metadata-enhanced-e5-small-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 0d9768745afbc006b436617c5de8421e Run description: zero shot e5-small-v2 dense retrieval using the enhanced (metadata)
Participants
Run ID: metadata-enhanced-gte-base-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: fb12bca3b2b2b8b3e725a145114b815d Run description: zero shot gte-base dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-gte-large-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: c1288dc70c7d9efb61553cf183842cb6 Run description: zero shot gte-large dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-gte-small-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 7acbcbc74df8597aec76bbe0421c661b Run description: zero shot gte-small dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-bge-base-en Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 20717c1d9f72b800d58e734ef751473e Run description: fine-tune bge-base-en dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-bge-large-en Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 98d049db3db227e05d5dba1c26c80bcd Run description: fine-tune bge-large-en dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-bge-small-en Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 9c039c0eae7c630a8c23dae334cfe5c9 Run description: fine-tune bge-small-en dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-dpr-bert Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 6dafad811bff342970f0fabb69d65f92 Run description: fine-tune bert-base-uncased dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-e5-base-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 47e4fca04ac64cf7ac0839246339486b Run description: zero shot spacemanidol/trec-product-search-e5-base-v2 dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-e5-large-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 894021f4e101c257e229d699fc2d4900 Run description: finetuned e5-large dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-e5-small-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 1da94625ca3ac081881fabfb812a095b Run description: finetuned e5-small dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-gte-base Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 38cadd0da9c188911b9cc3c0bbf1b3e7 Run description: fine-tune thenlper/gte-base dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-gte-large Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: cb0c510ba157b10dc9f20b9b0dc3f356 Run description: fine-tune thenlper/gte-large dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-enhanced-trec-product-search-gte-small Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 6dc41f29cdf7a1218a364ef9cbde64c4 Run description: zero shot spacemanidol/trec-product-search-gte-small dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-trec-product-search-all-miniLM-L12-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: 06d8c0feaf929311d1b39d27d82141f1 Run description: fine-tune -all-miniLM-L12-v2 dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-trec-product-search-all-miniLM-L6-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: e76337a26f7df0f803c843848b8e99e2 Run description: fine-tune -all-miniLM-L6-v2l dense retrieval using the enhanced (metadata) collection
Participants
Run ID: metadata-trec-product-search-all-mpnet-base-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/11/2023 Type: automatic Task: retrieval MD5: b7e7355e805bf0f63d8df9aafb6447c0 Run description: fine-tune all-mpnet-base-v2dense retrieval using the enhanced (metadata) collection
r_gpt3d5_turbo
Participants
Run ID: r_gpt3d5_turbo Participant: h2oloo Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: reranking MD5: 1e28249109255fee55ab71f1402a66c6 Run description: gpt-3.5-turbo, top-50 reranking
search-dpr-bert-base
Participants
Run ID: search-dpr-bert-base Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 4eedc8e0077499e16b6bbae0a3c803d7 Run description: finetuned bert-base-uncased dense retrieval using the simple (non-metadata) enhanced corpus)
simple-all-MiniLM-L12-v2-zero-shot
Participants
Run ID: simple-all-MiniLM-L12-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 6fcd6bbd0474f3fce4d57a6c6149daac Run description: zero shot all-MiniLM-L6 V2 dense retrieval using the simple (non metadata) enhanced corpus)
simple-all-MiniLM-L6-v2-zero-shot
Participants
Run ID: simple-all-MiniLM-L6-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 549d7f8a0344097799b30f030ef7afd6 Run description: zero shot all-MiniLM-L6 V2 dense retrieval.
simple-all-mpnet-base-v2-zero-shot
Participants
Run ID: simple-all-mpnet-base-v2-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 52e44bfb2f3002b508245b3d087f1b09 Run description: zero shot all-mpnet base V2 dense retrieval using the simple (non-metadata) enhanced corpus)
simple-bert-base-uncased-zero-shot
Participants
Run ID: simple-bert-base-uncased-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: c3a42ba9cae281ec4b3795b8b4564417 Run description: zero shot bert-base-uncased dense retrieval using the simple (non-metadata) enhanced corpus)
simple-bge-base-zero-shot
Participants
Run ID: simple-bge-base-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: e67ebbe7841894a1ad67e3c87808d858 Run description: zero shot bge-base-en dense retrieval using the simple (non-metadata) enhanced corpus)
simple-bge-large-zero-shot
Participants
Run ID: simple-bge-large-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: beb9a9eb08e3a1907cb3d7ae50c9ed35 Run description: zero shot bge-large-en dense retrieval using the simple (non-metadata) enhanced corpus)
simple-bge-small-zero-shot
Participants
Run ID: simple-bge-small-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 03ebfbd067b0ef770df5991f3eea737b Run description: zero shot bge-small-en dense retrieval using the simple (non-metadata) enhanced corpus)
simple-contriever-base-zero-shot
Participants
Run ID: simple-contriever-base-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 0b03f14a4a9e2313fe99feb2efba10fb Run description: zero shot contriever-base-msmarco dense retrieval using the simple (non-metadata) enhanced corpus)
simple-e5-base-zero-shot
Participants
Run ID: simple-e5-base-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: b6a389d1ab14c1cf8fa569d85deebedd Run description: zero shot e5-base-v2 dense retrieval using the simple (non-metadata) enhanced corpus)
simple-e5-large-zero-shot
Participants
Run ID: simple-e5-large-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: a2ceb28004019b8918c46b6e9521d7ca Run description: zero shot e5-large-v2 dense retrieval using the simple (non-metadata) enhanced corpus)
simple-e5-small-zero-shot
Participants
Run ID: simple-e5-small-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: d3c4919a31f0840a2b506cf23938d244 Run description: zero shot e5-small-v2 dense retrieval using the simple (non-metadata) enhanced corpus)
simple-gte-base-zero-shot
Participants
Run ID: simple-gte-base-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 3813f812708f1de1ceb226e1c384d629 Run description: zero shot gte-base dense retrieval using the simple (non-metadata) enhanced corpus)
simple-gte-large-zero-shot
Participants
Run ID: simple-gte-large-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 7d7f5407d90b00ecbc1994e6ec12beb7 Run description: zero shot gte-large dense retrieval using the simple (non-metadata) enhanced corpus)
simple-gte-small-zero-shot
Participants
Run ID: simple-gte-small-zero-shot Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: bac93995556f43d657d2822c8f8564d9 Run description: zero shot gte-small dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-all-miniLM-L12-v2
Participants
Run ID: simple-trec-product-search-all-miniLM-L12-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: da0738b37356a914bbe42f9c0130ccf9 Run description: finetuned shot all-miniLM-L12-v2 dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-all-miniLM-L6-v2
Participants
Run ID: simple-trec-product-search-all-miniLM-L6-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: d30bd2ff559e05268c3c5025da9088ce Run description: finetuned shot all-miniLM-L6-v2 dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-all-mpnet-base-v2
Participants
Run ID: simple-trec-product-search-all-mpnet-base-v2 Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: c6dd4a2995b3cd92ff851b4782006566 Run description: finetuned shot all-mpnet-v2 dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-bge-base-en
Participants
Run ID: simple-trec-product-search-bge-base-en Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 711d0f3dc225b8ace03d9bfff491b616 Run description: finetuned bge-base-en dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-bge-small-en
Participants
Run ID: simple-trec-product-search-bge-small-en Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: c6508fa3631c6473686f994d56fe89bc Run description: finetuned bge-small-en dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-gte-base
Participants
Run ID: simple-trec-product-search-gte-base Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 988f6979f09f25080f5a4b7b0de2e165 Run description: finetuned gte-base dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-gte-large
Participants
Run ID: simple-trec-product-search-gte-large Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: c55c207042924a41f0d81f01b1c326f1 Run description: finetuned gte-large dense retrieval using the simple (non-metadata) enhanced corpus)
simple-trec-product-search-gte-small
Participants
Run ID: simple-trec-product-search-gte-small Participant: NIST Track: Product Search Year: 2023 Submission: 8/10/2023 Type: automatic Task: retrieval MD5: 8696bcd97050531a34c5be04dd44f1fa Run description: finetuned gte-small dense retrieval using the simple (non-metadata) enhanced corpus)