Runs - CrisisFACTs 2022¶

BM25_Heuristic_ILP¶

Run ID: BM25_Heuristic_ILP
Participant: ohm_kiz
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: a3507adbba6da822745b49797641f7d6
Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: Heuristical re-ranking with selected entity concepts based on bag-of-entities and bag-of-keywords (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

BM25_QAasnq_ILP¶

Run ID: BM25_QAasnq_ILP
Participant: ohm_kiz
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: 456ec14b698bef9b4ac71737f48ef028
Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: QA re-ranking with RoBERTa pretrained on ASNQ dataset (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

BM25_QAcrisis_ILP¶

Run ID: BM25_QAcrisis_ILP
Participant: ohm_kiz
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: 3ce572e66ebbcc2c1962a2927c0fcde1
Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: QA re-ranking with RoBERTa pretrained on ASNQ dataset and finetuned on synthesized CrisisQA DocEE-dataset (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

ColBERT_ILP¶

Run ID: ColBERT_ILP
Participant: ohm_kiz
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: aec521ba94511d6c500110089d877930
Run description: The system consists of two successive components: 1: Late interaction retrieval with ColBERTv2 pretrained on MS-MARCO dataset (top 25 per query) 2: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)

combsum¶

Run ID: combsum
Participant: umcp
Track: CrisisFACTs
Year: 2022
Submission: 9/27/2022
MD5: b8f8229fa999c59432ac206bf4650d76

eXSum22_submission_01¶

Run ID: eXSum22_submission_01
Participant: eXSum22
Track: CrisisFACTs
Year: 2022
Submission: 9/28/2022
MD5: 7cd5cc3b844d4f00d5849ca1097d29b0

eXSum22_submission_02¶

Run ID: eXSum22_submission_02
Participant: eXSum22
Track: CrisisFACTs
Year: 2022
Submission: 9/28/2022
MD5: c505ef012404ccb8b76e4391e6e04e1a

IRIT_IRIS_mean_USE¶

Run ID: IRIT_IRIS_mean_USE
Participant: IRIT_IRIS
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: f4632573ba9a20959d0c0a379c853c07
Run description: The system computes the importance as the similarity between an item and a potential Oracle summary automatically constructed. The potential Oracle summary is created using the mean of all the items of the daily stream, regarding a specific representation.

IRIT_IRIS_mean_USE_INeeds¶

Run ID: IRIT_IRIS_mean_USE_INeeds
Participant: IRIT_IRIS
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: cee9540e286567c7b74ef58486939477
Run description: The system computes the importance as the similarity between an item and a potential Oracle summary automatically constructed. The potential Oracle summary is created using the mean of all the items of the daily stream, regarding a specific representation, for a particular query.

IRIT_IRIS_tssubert¶

Run ID: IRIT_IRIS_tssubert
Participant: IRIT_IRIS
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: 71ba6f77d21a67daac9acf850412ac76
Run description: The system computes the importance using a neural pre-trained language model and the frequency of the tokens of the stream. Then, the system selects items to keep in the summary using redundancy removal in the manner of MMR.

mrr_all¶

Run ID: mrr_all
Participant: umcp
Track: CrisisFACTs
Year: 2022
Submission: 9/27/2022
MD5: b4a265bb9b2ce304b16260bd881cf848

mrr_main¶

Run ID: mrr_main
Participant: umcp
Track: CrisisFACTs
Year: 2022
Submission: 9/27/2022
MD5: 92d1646335d699d1fb4c3356b94bcdc6

mrr_no_dd¶

Run ID: mrr_no_dd
Participant: umcp
Track: CrisisFACTs
Year: 2022
Submission: 9/27/2022
MD5: 97a6fe205be3703d5db7189ec93da9f8

mrr_nobrf¶

Run ID: mrr_nobrf
Participant: umcp
Track: CrisisFACTs
Year: 2022
Submission: 9/27/2022
MD5: ef398813cc156941996634e058c6c253

mrr_sum¶

Run ID: mrr_sum
Participant: umcp
Track: CrisisFACTs
Year: 2022
Submission: 9/27/2022
MD5: 7256442e1de357513b9c8863b1a9a302

nazmultum11¶

Participants | Input | Summary (auto) | Summary (manual) | Appendix

Run ID: nazmultum11
Participant: SiPEO
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: manual
MD5: f8852e87d24969fe1355a791b38249c4
Run description: for a single day run and single requestID, after downloading the dataframe it goes through text cleaning process (stop words, http links and emojis). For the queries, only stop words have removed. A SentenceTransformer model have been used to calculating the similiarity measurement ('sentence-transformers/all-mpnet-base-v2'). Then both the text and queries have sent to a function where against each query, the top 5 best maching sentecne has picked based on cosine similiarity. Then all of the sentences-->query pair has sorted based on the 'importance score' ( = cosine similiarity), then top 100 pair has been selected as a facts after removing the duplication when applied (retained the highest score pair in case of duplication). And then for the each pair of output, a output dataframe have formed with the respective information regarding text and query. The final output is the concat form of all the day-->requestID pair. And finally the dataframe transform to the submitted json format.

NM-1¶

Run ID: NM-1
Participant: NM.unicamp
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: d460ee1cfcb5236651f809c3920ae74c
Run description: We follow the traditional two-step retrieve-and-aggregate approach to solve the task. In the first step, we leverage a state-of-the-art search engine to retrieve candidate passages from the event collection of streams. In the second step, we use GPT-3 in a few-shot setting to generate a verbose explanation of the passages' contents before outputting an answer. We use the generated explanation as the fact text.

NM-2¶

Run ID: NM-2
Participant: NM.unicamp
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: b71a9d507349bdd9599484459dad770f
Run description: We follow the traditional two-step retrieve-and-aggregate approach to solve the task. In the first step, we leverage a state-of-the-art search engine to retrieve candidate passages from the event collection of streams. In the second step, we use GPT-3 in a few-shot setting to generate a verbose explanation of the passages' contents before outputting an answer. We use the generated explanation as the fact text. In this run, we applied a pos-processing step for removing extra white spaces

rr_now¶

Run ID: rr_now
Participant: umcp
Track: CrisisFACTs
Year: 2022
Submission: 9/27/2022
MD5: ea593fc83c7f2733cf5677e427c01a92

submission_final.json¶

Participants | Input | Summary (auto) | Summary (manual) | Appendix

Run ID: submission_final.json
Participant: IISER22
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: c76c670a350e2c664b71b541e5abb3bb
Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used BM25 Similarity to retrieval relevant document for every query.Here system was selected first five document for each query.

submission_final_4¶

Participants | Input | Summary (auto) | Summary (manual) | Appendix

Run ID: submission_final_4
Participant: IISER22
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: d068480a0681ea24b4765eca2d90c005
Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMDirichlet Similarity and BM25 similarity to retrieval relevant document for every query.Here system was selected first five document for each query. After that system found common sentences that are present in two json file.Then again calculte their importance and then normalize that.

submission_LM_DS_3¶

Participants | Input | Summary (auto) | Summary (manual) | Appendix

Run ID: submission_LM_DS_3
Participant: IISER22
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: 5314a08fca0717d17ef9089f9ae293bf
Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMDirichlet Similarity to retrieval relevant document for every query.Here system was selected first five document for each query.

submission_LM_JM_2¶

Participants | Input | Summary (auto) | Summary (manual) | Appendix

Run ID: submission_LM_JM_2
Participant: IISER22
Track: CrisisFACTs
Year: 2022
Submission: 9/30/2022
Type: automatic
MD5: 0ad4b26f3c44e9307f9eaa1ef31a5751
Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMJelinekMercer Similarity to retrieval relevant document for every query.Here system was selected first three document for each query.