Skip to content

Runs - CrisisFACTs 2022

BM25_Heuristic_ILP

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: BM25_Heuristic_ILP
  • Participant: ohm_kiz
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: a3507adbba6da822745b49797641f7d6
  • Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: Heuristical re-ranking with selected entity concepts based on bag-of-entities and bag-of-keywords (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

BM25_QAasnq_ILP

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: BM25_QAasnq_ILP
  • Participant: ohm_kiz
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: 456ec14b698bef9b4ac71737f48ef028
  • Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: QA re-ranking with RoBERTa pretrained on ASNQ dataset (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

BM25_QAcrisis_ILP

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: BM25_QAcrisis_ILP
  • Participant: ohm_kiz
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: 3ce572e66ebbcc2c1962a2927c0fcde1
  • Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: QA re-ranking with RoBERTa pretrained on ASNQ dataset and finetuned on synthesized CrisisQA DocEE-dataset (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

ColBERT_ILP

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: ColBERT_ILP
  • Participant: ohm_kiz
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: aec521ba94511d6c500110089d877930
  • Run description: The system consists of two successive components: 1: Late interaction retrieval with ColBERTv2 pretrained on MS-MARCO dataset (top 25 per query) 2: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

combsum

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: combsum
  • Participant: umcp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/27/2022
  • MD5: b8f8229fa999c59432ac206bf4650d76

eXSum22_submission_01

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: eXSum22_submission_01
  • Participant: eXSum22
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/28/2022
  • MD5: 7cd5cc3b844d4f00d5849ca1097d29b0

eXSum22_submission_02

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: eXSum22_submission_02
  • Participant: eXSum22
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/28/2022
  • MD5: c505ef012404ccb8b76e4391e6e04e1a

IRIT_IRIS_mean_USE

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: IRIT_IRIS_mean_USE
  • Participant: IRIT_IRIS
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: f4632573ba9a20959d0c0a379c853c07
  • Run description: The system computes the importance as the similarity between an item and a potential Oracle summary automatically constructed. The potential Oracle summary is created using the mean of all the items of the daily stream, regarding a specific representation.

IRIT_IRIS_mean_USE_INeeds

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: IRIT_IRIS_mean_USE_INeeds
  • Participant: IRIT_IRIS
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: cee9540e286567c7b74ef58486939477
  • Run description: The system computes the importance as the similarity between an item and a potential Oracle summary automatically constructed. The potential Oracle summary is created using the mean of all the items of the daily stream, regarding a specific representation, for a particular query.

IRIT_IRIS_tssubert

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: IRIT_IRIS_tssubert
  • Participant: IRIT_IRIS
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: 71ba6f77d21a67daac9acf850412ac76
  • Run description: The system computes the importance using a neural pre-trained language model and the frequency of the tokens of the stream. Then, the system selects items to keep in the summary using redundancy removal in the manner of MMR.

mrr_all

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: mrr_all
  • Participant: umcp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/27/2022
  • MD5: b4a265bb9b2ce304b16260bd881cf848

mrr_main

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: mrr_main
  • Participant: umcp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/27/2022
  • MD5: 92d1646335d699d1fb4c3356b94bcdc6

mrr_no_dd

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: mrr_no_dd
  • Participant: umcp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/27/2022
  • MD5: 97a6fe205be3703d5db7189ec93da9f8

mrr_nobrf

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: mrr_nobrf
  • Participant: umcp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/27/2022
  • MD5: ef398813cc156941996634e058c6c253

mrr_sum

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: mrr_sum
  • Participant: umcp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/27/2022
  • MD5: 7256442e1de357513b9c8863b1a9a302

nazmultum11

Participants | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: nazmultum11
  • Participant: SiPEO
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: manual
  • MD5: f8852e87d24969fe1355a791b38249c4
  • Run description: for a single day run and single requestID, after downloading the dataframe it goes through text cleaning process (stop words, http links and emojis). For the queries, only stop words have removed. A SentenceTransformer model have been used to calculating the similiarity measurement ('sentence-transformers/all-mpnet-base-v2'). Then both the text and queries have sent to a function where against each query, the top 5 best maching sentecne has picked based on cosine similiarity. Then all of the sentences-->query pair has sorted based on the 'importance score' ( = cosine similiarity), then top 100 pair has been selected as a facts after removing the duplication when applied (retained the highest score pair in case of duplication). And then for the each pair of output, a output dataframe have formed with the respective information regarding text and query. The final output is the concat form of all the day-->requestID pair. And finally the dataframe transform to the submitted json format.

NM-1

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: NM-1
  • Participant: NM.unicamp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: d460ee1cfcb5236651f809c3920ae74c
  • Run description: We follow the traditional two-step retrieve-and-aggregate approach to solve the task. In the first step, we leverage a state-of-the-art search engine to retrieve candidate passages from the event collection of streams. In the second step, we use GPT-3 in a few-shot setting to generate a verbose explanation of the passages' contents before outputting an answer. We use the generated explanation as the fact text.

NM-2

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: NM-2
  • Participant: NM.unicamp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: b71a9d507349bdd9599484459dad770f
  • Run description: We follow the traditional two-step retrieve-and-aggregate approach to solve the task. In the first step, we leverage a state-of-the-art search engine to retrieve candidate passages from the event collection of streams. In the second step, we use GPT-3 in a few-shot setting to generate a verbose explanation of the passages' contents before outputting an answer. We use the generated explanation as the fact text. In this run, we applied a pos-processing step for removing extra white spaces

rr_now

Participants | Proceedings | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: rr_now
  • Participant: umcp
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/27/2022
  • MD5: ea593fc83c7f2733cf5677e427c01a92

submission_final.json

Participants | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: submission_final.json
  • Participant: IISER22
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: c76c670a350e2c664b71b541e5abb3bb
  • Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used BM25 Similarity to retrieval relevant document for every query.Here system was selected first five document for each query.

submission_final_4

Participants | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: submission_final_4
  • Participant: IISER22
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: d068480a0681ea24b4765eca2d90c005
  • Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMDirichlet Similarity and BM25 similarity to retrieval relevant document for every query.Here system was selected first five document for each query. After that system found common sentences that are present in two json file.Then again calculte their importance and then normalize that.

submission_LM_DS_3

Participants | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: submission_LM_DS_3
  • Participant: IISER22
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: 5314a08fca0717d17ef9089f9ae293bf
  • Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMDirichlet Similarity to retrieval relevant document for every query.Here system was selected first five document for each query.

submission_LM_JM_2

Participants | Input | Summary (auto) | Summary (manual) | Appendix

  • Run ID: submission_LM_JM_2
  • Participant: IISER22
  • Track: CrisisFACTs
  • Year: 2022
  • Submission: 9/30/2022
  • Type: automatic
  • MD5: 0ad4b26f3c44e9307f9eaa1ef31a5751
  • Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMJelinekMercer Similarity to retrieval relevant document for every query.Here system was selected first three document for each query.