Runs - Health Misinformation 2022¶

bm25¶

Results | Participants | Input | Summary | Appendix

Run ID: bm25
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 1fef723cf7718a7b10c0ec33a4bbcf28
Run description: Anserini/Pyserini BM25

citius.base¶

Run ID: citius.base
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/28/2022
Type: auto
Task: retrieval
MD5: 7e50c9f7b28074f46cd1a8a2c99c76c3
Run description: First, a Bm25 search over the index is performed using the question field to obtain the top 1000 documents per topic. Afterwards, we reorder the top 100 based on MonoT5 reranker technology and the question field.

citius.gpt-3¶

Run ID: citius.gpt-3
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: prediction
MD5: fc9a964b1c3d8b8e906b56dfc2cd726e
Run description: To predict the answer we utilised the encoder model GPT-3. We used no fine-tuned and no prompt. The model was fed with the topic question + "Yes or No?" and the answer was automatically analyzed and categorized into "yes", "no" or "inconclusive". Finally, the "yes" topics were assigned score 1, the "no", score 0, and the "inconclusive" turned into "no" with 0.5 probability.

citius.r1¶

Run ID: citius.r1
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: retrieval
MD5: 74264095f914293ccaa39decf939ed12
Run description: First, a Bm25 search over the index is performed using the question field to obtain the top 1000 documents per topic. Afterwards, we reorder the top 100 based on the fusion of two signals: a MonoT5 reranker technology using the question field and a RF classifier that identifies credibility. Those signals were weighted.

citius.r2¶

Run ID: citius.r2
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: retrieval
MD5: f7423aba3f448f9692e29d3a1eeb2452
Run description: First, a Bm25 search over the index is performed using the question field to obtain the top 1000 documents per topic. Afterwards, we reorder the top 100 based on the fusion of three signals: a MonoT5 reranker technology using the question field, a RF classifier that identifies credibility and a BERT model that discerns readable from non-readable excerpts. Those signals were weighted.

citius.r3¶

Run ID: citius.r3
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: retrieval
MD5: 729129ffd7979b3467e1370c0e86e5f6
Run description: First, a Bm25 search over the index is performed using the question field to obtain the top 1000 documents per topic. Afterwards, we reorder the top 100 based on MonoT5 reranker technology and the "predicted" correct sentence. To do so, we utilised a GPT-3 encoder model (see run citius.gpt-3 from Answer Prediction) to estimate the answer and for those identified as inconclusive we mantain the default Bm25 ordering.

citius.r4¶

Run ID: citius.r4
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: retrieval
MD5: 6ce645c221e3f6b1f980a045b6d168de
Run description: First, a Bm25 search over the index is performed using the question field to obtain the top 1000 documents per topic. Afterwards, we reorder the top 100 based on MonoT5 reranker technology and the "predicted" correct sentence. To do so, we utilised a GPT-3 encoder model (see run citius.gpt-3 from Answer Prediction) to estimate the answer and for those identified as inconclusive we kept the MonoT5 score based on the question field.

citius.r5¶

Run ID: citius.r5
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: retrieval
MD5: 4c5cbfc51796d9d67e4cefef5cee3377
Run description: First, a Bm25 search over the index is performed using the question field to obtain the top 1000 documents per topic. Afterwards, we reorder the top 100 based on MonoT5 reranker technology and the "predicted" correct sentence. To do so, we launched the query against a SE API and scraped the top 1 result. Finally, we extracted the most on topic passage and compared it with both variants (correct and incorrect) and we kept the one with highest probability.

citius.r6¶

Run ID: citius.r6
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: retrieval
MD5: c3f565a34c9f13496b04663c1a7f0617
Run description: First, a Bm25 search over the index is performed using the question field to obtain the top 1000 documents per topic. Afterwards, we reorder the top 100 based on MonoT5 reranker technology and the "predicted" correct sentence. To do so, we combined our GPT-3 estimator and our SE one. We simply apply an "and" operation. The inconclusive ones were kept like that, and they were reordered based on the MonoT5 score of the unmodified question.

citius.se¶

Run ID: citius.se
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: prediction
MD5: e336250c9d62fdb0f3a6a94934359381
Run description: To predict the answer, we launched the question into a SE API and we scraped the top 1 result. Then, using passage reranking technology, the most on topic passage was selected. Finally, the positive and negative versions of the sentences were compared against the passage and the highest probability determined the answer.

citius.se_gpt¶

Run ID: citius.se_gpt
Participant: CiTIUS
Track: Health Misinformation
Year: 2022
Submission: 7/31/2022
Type: auto
Task: prediction
MD5: b4eaaef007441b25ddcc3ba3d9c98036
Run description: To predict the answer, we applied an "and" operation to the output of our runs with GPT-3 and SE.

gpt3a¶

Results | Participants | Input | Summary | Appendix

Run ID: gpt3a
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: d8b5a8d34c85d994ab9b30f108c2f7e3
Run description: GPT3 prompted with 21 topics to generate label predictions

gpt3a_fc¶

Results | Participants | Input | Summary | Appendix

Run ID: gpt3a_fc
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: 851f0da9750f60da09b43d8c28d6846f
Run description: Rounds gpt3a to 1.0 or 0.0.

gpt3a_neg¶

Results | Participants | Input | Summary | Appendix

Run ID: gpt3a_neg
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: 5061028559b23d282824887135134e00
Run description: gpt3a but invert the predictions

gpt3b¶

Results | Participants | Input | Summary | Appendix

Run ID: gpt3b
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: 0da2ab7bf04b7b2a3baad0df578e51a9
Run description: gpt3a without normalized scoring scheme

gpt3b_neg¶

Results | Participants | Input | Summary | Appendix

Run ID: gpt3b_neg
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: c88a08173273391037baa53538107bba
Run description: gpt3b but invert the predictions

hm22.mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22.mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 5ad409a1e822bb954c53cf2566348dec
Run description: mdt5 - mdt5 on original

hm22.mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22.mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: ba5f438e0fd0d3e6cf8413ecc1572e85
Run description: mt5 - mt5 on original

hm22.vera¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22.vera
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: bbb5d2ec47a429cd390f3579e03294a0
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) vera - vera without linear combinations

hm22.vera_mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22.vera_mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 97d93032045771d82d2d749cbf1cfc65
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) vera_mdt5 - mdt5 on original followed by vera(0.95, duo1)

hm22.vera_mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22.vera_mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 2697b0f7b27b8499c0fe35498d47b945
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) vera_mt5 - mt5 on original followed by vera(0.95, mono)

hm22_ref.mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref.mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 75e283af8c8eefed518dfdb93f9a38bf
Run description: hm22_ref - label prediction by GPT3 (prompted with 2021 topics + stances) followed by reformulation by GPT3 mdt5 - mdt5

hm22_ref.mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref.mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: d94474cd58112fcdcae96155e347c7fe
Run description: hm22_ref - label prediction by GPT3 (prompted with 2021 topics + stances) followed by reformulation by GPT3 mt5 - mt5

hm22_ref.vera¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref.vera
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: aada542cf73adc2938f351d13b3eae1a
Run description: hm22_ref - label prediction by GPT3 (prompted with 2021 topics + stances) and reformulation by GPT3 vera - vera without linear combinations

hm22_ref.vera_mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref.vera_mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: a2313f2fceb7d5c3bfb65a819365a1cf
Run description: hm22_ref - label prediction by GPT3 (prompted with 2021 topics + stances) followed by reformulation by GPT3 vera_mdt5 - mdt5 on ref followed by vera(0.95, duo1)

hm22_ref.vera_mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref.vera_mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 8423f6d1dc925a34f12c93c1c76b3294
Run description: hm22_ref - label prediction by GPT3 (prompted with 2021 topics + stances) followed by reformulation by GPT3 vera_mdt5 - mdt5 on ref followed by vera(0.95, mono)

hm22_ref_comb.mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_comb.mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 5a33c1fc4d57d77fa0b76ce77fc4e3bc
Run description: hm22_ref_comb - combination of label prediction by GPT3 (prompted with 2021 topics + stances) and answer prediction run vera, followed by reformulation by GPT3 mdt5 - mdt5

hm22_ref_comb.mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_comb.mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 73ce4aa576bb1511f7ee87070520cb16
Run description: hm22_ref_comb - combination of label prediction by GPT3 (prompted with 2021 topics + stances) and answer prediction run vera, followed by reformulation by GPT3 mt5 - mt5

hm22_ref_comb.vera_mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_comb.vera_mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 1e2913d8a3af1fad8194c3bd169b2973
Run description: hm22_ref_comb - combination of label prediction by GPT3 (prompted with 2021 topics + stances) and answer prediction run vera, followed by reformulation by GPT3 vera_mdt5 - mdt5 on ref followed by vera(0.95, duo1)

hm22_ref_comb.vera_mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_comb.vera_mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: f2fd6f3bdfd63dba41326f8a5f02e203
Run description: hm22_ref_comb - combination of label prediction by GPT3 (prompted with 2021 topics + stances) and answer prediction run vera, followed by reformulation by GPT3 vera_mt5 - mt5 on ref followed by vera(0.95, mono)

hm22_ref_neg.mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_neg.mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: 492a95aaeb770afd391d4bdce102b32c
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) is inverted mdt5 - mdt5 on inverted reformulations

hm22_ref_neg.mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_neg.mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: d34df0734db8a6ee1d7a831db5380015
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) is inverted mt5 - mt5 on inverted reformulations

hm22_ref_neg.vera¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_neg.vera
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: b15d84c6534fb3697b5f3d74805ad124
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) is inverted vera - vera without linear combinations

hm22_ref_neg.vera_mdt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_neg.vera_mdt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: bcf868497d6e9be8837ea09d0ea753d9
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) is inverted vera_mdt5 - mdt5 on inverted reformulations followed by vera(0.95, duo1)

hm22_ref_neg.vera_mt5¶

Results | Participants | Input | Summary | Appendix

Run ID: hm22_ref_neg.vera_mt5
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: retrieval
MD5: b39a09cdf20f889cf0a004c1f85a62fc
Run description: hm22 - label prediction by GPT3 (prompted with 2021 topics + stances) is inverted vera_mdt5 - mt5 on inverted reformulations followed by vera(0.95, mono)

vera¶

Results | Participants | Input | Summary | Appendix

Run ID: vera
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: 4b491b70a8eaa83d5932daa38eebdb4b
Run description: Vera scores from top-50 Med-MonoT5 results

vera_abs¶

Results | Participants | Input | Summary | Appendix

Run ID: vera_abs
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: 6051cf5efb1533a86f14797cebccc0fa
Run description: Rounds vera to 1.0 or 0.0.

vera_gpt3¶

Results | Participants | Input | Summary | Appendix

Run ID: vera_gpt3
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: 7a790fe6029cd2974d5e05081b9bae0c
Run description: Mean of runs vera and gpt3

vera_gpt3_abs¶

Results | Participants | Input | Summary | Appendix

Run ID: vera_gpt3_abs
Participant: h2oloo
Track: Health Misinformation
Year: 2022
Submission: 8/28/2022
Type: auto
Task: prediction
MD5: 1a12140d60057d438661fbdd7138d703
Run description: Rounds vera_gpt3 to 1.0 or 0.0.

WatS-AP-Baseline¶

Run ID: WatS-AP-Baseline
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: prediction
MD5: e7f27ae8ef2473c1766809a6f1d6f7f3
Run description: Logistic Regression-based Trust Model

WatS-AP-Baseline-L1¶

Run ID: WatS-AP-Baseline-L1
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: prediction
MD5: 4449d20ab133385cf86539a94e8ebc14
Run description: Logistic Regression-based Trust Model with L1 regularization

WatS-AP-Manual¶

Run ID: WatS-AP-Manual
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/29/2022
Type: manual
Task: prediction
MD5: 4a0b1f9bbaf8983de6a2db953b09b934
Run description: This a manual run where the answers to topics were judged with google search results.

WatS-AP-MT5¶

Run ID: WatS-AP-MT5
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: prediction
MD5: 2c9dd2e73354b7073f3708b5774ec1d0
Run description: Logistic Regression-based Trust Model with mt5 for reranking

WatS-AP-MT5-L1¶

Run ID: WatS-AP-MT5-L1
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: prediction
MD5: 2469fd5a0c4d6a847b94e05c1de27fcb
Run description: Logistic Regression-based Trust Model with mt5 for reranking and L1 regularization

WatS-BB75-MT5-TA¶

Run ID: WatS-BB75-MT5-TA
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: prediction
MD5: c6b6a174880f1ef1d37d611252146290
Run description: "Initial Retrieval: Top 1k BM25 Rerank: MT5 Passage extraction: MeshQA QA: BERT , PubMedQA (+ tuning on 2019 2021 qrels) Answer prediction: mean_topk (bert(passage plus host plus hon))"

WatS-Bigbird2_75-MT5¶

Run ID: WatS-Bigbird2_75-MT5
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: f5ccf579ff71492183a633ae7fa32f1a
Run description: "Initial Retrieval: Top 1k BM25 Passage Extraction: Bigbird ss=0.75 Rerank: MT5"

WatS-Bigbird2_75-MT5-TA1¶

Run ID: WatS-Bigbird2_75-MT5-TA1
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: 46e1b9db6370f699bc31c2afe09cd13f
Run description: Same as TA2 but with no fine tuning on previous years topics

WatS-Bigbird2_75-MT5-TA2¶

Run ID: WatS-Bigbird2_75-MT5-TA2
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: fb248f61863db9fd79333d803a88583c
Run description: "Initial Retrieval: Top 1k BM25 Rerank: MT5 Passage extraction: MeshQA QA: BERT , PubMedQA (+ tuning on 2019 2021 qrels) Answer prediction: mean_topk (bert(passage plus host plus hon)) Second Rerank: MT5 * agreement alpha=.2"

WatS-BM25-Query¶

Run ID: WatS-BM25-Query
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: 9d8709d0421e2a7b72fc63a3ccdd5b3d
Run description: BM25 baseline (k1=0.9, b=0.4) using the query field

WatS-BM25-Question¶

Run ID: WatS-BM25-Question
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: fa40dd2a07944eed25884f1248e22871
Run description: BM25 baseline (k1=0.9, b=0.4) using the query field

WatS-Manual¶

Run ID: WatS-Manual
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/29/2022
Type: manual
Task: retrieval
MD5: 44e16c086d862c3316c192d976de8d1a
Run description: This a manual run where topics were judged by their snippets which were produced with MT5. the top 1k bm25 docs for each topic were reranked with MT5 and the first very useful correct 10 documents were found and placed first. incorrect documents were placed last.

WatS-manual-pred¶

Run ID: WatS-manual-pred
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: manual
Task: prediction
MD5: d55cde64c2cce1cc53a1be29e7eb72f3
Run description: Paste question into google, mean 28s to determine answer

WatS-MT5-MT5¶

Run ID: WatS-MT5-MT5
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: 8422bbb74041fb891fdf6bd921ffce12
Run description: "Initial Retrieval: Top 1k BM25 Passage Extraction: MT5 Rerank: MT5"

WatS-Trust¶

Run ID: WatS-Trust
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: 1d1b4e992658f4dd052342c5490b668b
Run description: Soft reranking BM25 using the predicted answers from WatS-AP-Baseline

WatS-Trust-L1¶

Run ID: WatS-Trust-L1
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: cb3eec9594d18527d82c513428e8b37a
Run description: Soft reranking BM25 using the predicted answer from WatS-AP-Baseline-L1

WatS-Trust-MT5¶

Run ID: WatS-Trust-MT5
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: eaa17132850030736ac19e36f75e4f58
Run description: Soft reranking MT5 using the predicted answer from WatS-AP-MT5

WatS-Trust-MT5-L1¶

Run ID: WatS-Trust-MT5-L1
Participant: UWaterlooMDS
Track: Health Misinformation
Year: 2022
Submission: 8/2/2022
Type: auto
Task: retrieval
MD5: f26ccb8ec2b902671e09143eeede5de7
Run description: Soft reranking MT5 using the predicted answer from WatS-AP-MT5-L1

webis-goo-boolq-abs¶

Run ID: webis-goo-boolq-abs
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: 758097d69ca42b400deaf1538b3c9575
Run description: Retrieve up to 20 abstracts through Google Custom Search Engine using provided questions. Apply pre-trained RoBERTa-base-BoolQ model to abstracts using questions. Aggregate results.

webis-goo-lbert-abs¶

Run ID: webis-goo-lbert-abs
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: 651ea09b1792730297bbaa72d9b8a6d3
Run description: Retrieve up to 20 abstracts through Google Custom Search Engine using provided questions. Apply pre-trained BioLinkBERT-large model to abstracts using questions. Aggregate results.

webis-goo-lbert-title-abs¶

Run ID: webis-goo-lbert-title-abs
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: 037a9a647a99ee499ab3f0fda03e3010
Run description: Retrieve up to 20 abstracts along with titles through Google Custom Search Engine using provided questions. Apply pre-trained BioLinkBERT-large model to titles and abstracts using questions. Aggregate results.

webis-longck-ax-com¶

Run ID: webis-longck-ax-com
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: aefaaa92b41fb358380b61e8e4904b02
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context). Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using LongChecker (fever_sci checkpoint, using abstract and title as context). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Combine retrieval score with similarity to predicted topic answer (tradeoff 0.75).

webis-longck-ax-lin¶

Run ID: webis-longck-ax-lin
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: 282443effae1b7623ed138a6c00786e6
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context). Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using LongChecker (fever_sci checkpoint, using abstract and title as context). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Boost retrieval score linearly based on similarity to predicted topic answer.

webis-longck-ax-pol¶

Run ID: webis-longck-ax-pol
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: 7781c7de503b293ebf2967e3ef947d4c
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context). Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using LongChecker (fever_sci checkpoint, using abstract and title as context). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Boost retrieval score polynomially (x^2) based on similarity to predicted topic answer.

webis-longck-dis¶

Run ID: webis-longck-dis
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: f26f35390a9d9ee3ffb25e0a4299bb8b
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context). Aggregate abstract-wise answer score with ranking position discount.

webis-longck-uniqa-ax-com¶

Run ID: webis-longck-uniqa-ax-com
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: 7cf56c247b69190c10388ce59f636ca8
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Combine retrieval score with similarity to predicted topic answer (tradeoff 0.75).

webis-longck-uniqa-ax-dis¶

Run ID: webis-longck-uniqa-ax-dis
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: cc701bf1f6596ae28655e2c082d145c0
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount.

webis-longck-uniqa-ax-lin¶

Run ID: webis-longck-uniqa-ax-lin
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: 88ee57623c25b308a4c914905d98843c
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Boost retrieval score linearly based on similarity to predicted topic answer.

webis-longck-uniqa-ax-pol¶

Run ID: webis-longck-uniqa-ax-pol
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: 1b65c66bdfd733e005e3034176185a79
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Boost retrieval score polynomially (x^2) based on similarity to predicted topic answer.

webis-longck-uniqa-dis¶

Run ID: webis-longck-uniqa-dis
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: 317c29d464cbd2a6590e166bd61cd1b9
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Aggregate abstract-wise answer score with ranking position discount.

webis-longck-uniqa-pol¶

Run ID: webis-longck-uniqa-pol
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: 05e81f442166286aed6f5eea889bccc6
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using LongChecker (fever_sci checkpoint, using abstract and title as context) and UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Boost retrieval score polynomially (x^2) based on similarity to predicted topic answer.

webis-nlm-boolq-abs¶

Run ID: webis-nlm-boolq-abs
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: manual
Task: prediction
MD5: 7da710bb0b9b11d7167cf8c0e77935df
Run description: Retrieve up to 20 abstracts through PubMed search API using provided keyword queries. Reformulate queries for which only 0 or 1 documents are retrieved. Apply pre-trained RoBERTa-base-BoolQ model to abstracts using keyword queries. Aggregate results.

webis-nlm-lbert-abs¶

Run ID: webis-nlm-lbert-abs
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: manual
Task: prediction
MD5: da1a5842a2ff933519664365f7c03e95
Run description: Retrieve up to 20 abstracts through PubMed search API using provided keyword queries. Reformulate queries for which only 0 or 1 documents are retrieved. Apply pre-trained BioLinkBERT-large model to abstracts using keyword queries. Aggregate results.

webis-uniqa-ax-com¶

Run ID: webis-uniqa-ax-com
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: ff7ac49642c2c9d283f70fb3a4dbb66c
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Combine retrieval score with similarity to predicted topic answer (tradeoff 0.75).

webis-uniqa-ax-lin¶

Run ID: webis-uniqa-ax-lin
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: 1dac5bdfe7f88109b9999a1897081785
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Boost retrieval score linearly based on similarity to predicted topic answer.

webis-uniqa-ax-pol¶

Run ID: webis-uniqa-ax-pol
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: retrieval
MD5: f4a92799bab23a747e3460169e135683
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Average answer score of both predictors. Resolve answer conflicts with axiomatic re-ranking (most recently published abstract first). Aggregate abstract-wise answer score with ranking position discount. Retrieve 1000 documents from C4 using BM25 (Elasticsearch). Predict answer per document using UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Boost retrieval score polynomially (x^2) based on similarity to predicted topic answer.

webis-uniqa-dis¶

Run ID: webis-uniqa-dis
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: c853271e838783c5dd6c3772998396ef
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using UnifiedQA (allenai/unifiedqa-t5-large, using abstract as context). Aggregate abstract-wise answer score with ranking position discount.

webis-verasent-dis¶

Run ID: webis-verasent-dis
Participant: Webis
Track: Health Misinformation
Year: 2022
Submission: 8/1/2022
Type: auto
Task: prediction
MD5: 746e38981983a918271feedd5f31146f
Run description: Retrieve 1000 abstracts from PubMed using BM25 (Elasticsearch). Re-rank top-1000 with monoT5 (castorini/monot5-3b-med-msmarco) and top-50 with duoT5 (castorini/duot5-3b-med-msmarco). Predict answer per abstract using Vera (gs://castorini/vera/experiments/3B, using abstract as context, select most 'relevant' sentences). Aggregate abstract-wise answer score with ranking position discount.