Runs - CrisisFACTs 2022¶
BM25_Heuristic_ILP¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: BM25_Heuristic_ILP
- Participant: ohm_kiz
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
a3507adbba6da822745b49797641f7d6
- Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: Heuristical re-ranking with selected entity concepts based on bag-of-entities and bag-of-keywords (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)
BM25_QAasnq_ILP¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: BM25_QAasnq_ILP
- Participant: ohm_kiz
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
456ec14b698bef9b4ac71737f48ef028
- Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: QA re-ranking with RoBERTa pretrained on ASNQ dataset (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)
BM25_QAcrisis_ILP¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: BM25_QAcrisis_ILP
- Participant: ohm_kiz
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
3ce572e66ebbcc2c1962a2927c0fcde1
- Run description: The system consists of three successive components: 1: Lexical retrieval with BM25 based on indicative terms + query text (top 100 per query) 2: QA re-ranking with RoBERTa pretrained on ASNQ dataset and finetuned on synthesized CrisisQA DocEE-dataset (top 25 per query) 3: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)
ColBERT_ILP¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: ColBERT_ILP
- Participant: ohm_kiz
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
aec521ba94511d6c500110089d877930
- Run description: The system consists of two successive components: 1: Late interaction retrieval with ColBERTv2 pretrained on MS-MARCO dataset (top 25 per query) 2: ILP-system for diversified sentence selection in terms of covered entities and queries + MMR for re-ranking (top 150 stream items)
combsum¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: combsum
- Participant: umcp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/27/2022
- MD5:
b8f8229fa999c59432ac206bf4650d76
eXSum22_submission_01¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: eXSum22_submission_01
- Participant: eXSum22
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/28/2022
- MD5:
7cd5cc3b844d4f00d5849ca1097d29b0
eXSum22_submission_02¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: eXSum22_submission_02
- Participant: eXSum22
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/28/2022
- MD5:
c505ef012404ccb8b76e4391e6e04e1a
IRIT_IRIS_mean_USE¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: IRIT_IRIS_mean_USE
- Participant: IRIT_IRIS
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
f4632573ba9a20959d0c0a379c853c07
- Run description: The system computes the importance as the similarity between an item and a potential Oracle summary automatically constructed. The potential Oracle summary is created using the mean of all the items of the daily stream, regarding a specific representation.
IRIT_IRIS_mean_USE_INeeds¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: IRIT_IRIS_mean_USE_INeeds
- Participant: IRIT_IRIS
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
cee9540e286567c7b74ef58486939477
- Run description: The system computes the importance as the similarity between an item and a potential Oracle summary automatically constructed. The potential Oracle summary is created using the mean of all the items of the daily stream, regarding a specific representation, for a particular query.
IRIT_IRIS_tssubert¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: IRIT_IRIS_tssubert
- Participant: IRIT_IRIS
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
71ba6f77d21a67daac9acf850412ac76
- Run description: The system computes the importance using a neural pre-trained language model and the frequency of the tokens of the stream. Then, the system selects items to keep in the summary using redundancy removal in the manner of MMR.
mrr_all¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: mrr_all
- Participant: umcp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/27/2022
- MD5:
b4a265bb9b2ce304b16260bd881cf848
mrr_main¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: mrr_main
- Participant: umcp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/27/2022
- MD5:
92d1646335d699d1fb4c3356b94bcdc6
mrr_no_dd¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: mrr_no_dd
- Participant: umcp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/27/2022
- MD5:
97a6fe205be3703d5db7189ec93da9f8
mrr_nobrf¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: mrr_nobrf
- Participant: umcp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/27/2022
- MD5:
ef398813cc156941996634e058c6c253
mrr_sum¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: mrr_sum
- Participant: umcp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/27/2022
- MD5:
7256442e1de357513b9c8863b1a9a302
nazmultum11¶
Participants
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: nazmultum11
- Participant: SiPEO
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: manual
- MD5:
f8852e87d24969fe1355a791b38249c4
- Run description: for a single day run and single requestID, after downloading the dataframe it goes through text cleaning process (stop words, http links and emojis). For the queries, only stop words have removed. A SentenceTransformer model have been used to calculating the similiarity measurement ('sentence-transformers/all-mpnet-base-v2'). Then both the text and queries have sent to a function where against each query, the top 5 best maching sentecne has picked based on cosine similiarity. Then all of the sentences-->query pair has sorted based on the 'importance score' ( = cosine similiarity), then top 100 pair has been selected as a facts after removing the duplication when applied (retained the highest score pair in case of duplication). And then for the each pair of output, a output dataframe have formed with the respective information regarding text and query. The final output is the concat form of all the day-->requestID pair. And finally the dataframe transform to the submitted json format.
NM-1¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: NM-1
- Participant: NM.unicamp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
d460ee1cfcb5236651f809c3920ae74c
- Run description: We follow the traditional two-step retrieve-and-aggregate approach to solve the task. In the first step, we leverage a state-of-the-art search engine to retrieve candidate passages from the event collection of streams. In the second step, we use GPT-3 in a few-shot setting to generate a verbose explanation of the passages' contents before outputting an answer. We use the generated explanation as the fact text.
NM-2¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: NM-2
- Participant: NM.unicamp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
b71a9d507349bdd9599484459dad770f
- Run description: We follow the traditional two-step retrieve-and-aggregate approach to solve the task. In the first step, we leverage a state-of-the-art search engine to retrieve candidate passages from the event collection of streams. In the second step, we use GPT-3 in a few-shot setting to generate a verbose explanation of the passages' contents before outputting an answer. We use the generated explanation as the fact text. In this run, we applied a pos-processing step for removing extra white spaces
rr_now¶
Participants
| Proceedings
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: rr_now
- Participant: umcp
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/27/2022
- MD5:
ea593fc83c7f2733cf5677e427c01a92
submission_final.json¶
Participants
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: submission_final.json
- Participant: IISER22
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
c76c670a350e2c664b71b541e5abb3bb
- Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used BM25 Similarity to retrieval relevant document for every query.Here system was selected first five document for each query.
submission_final_4¶
Participants
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: submission_final_4
- Participant: IISER22
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
d068480a0681ea24b4765eca2d90c005
- Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMDirichlet Similarity and BM25 similarity to retrieval relevant document for every query.Here system was selected first five document for each query. After that system found common sentences that are present in two json file.Then again calculte their importance and then normalize that.
submission_LM_DS_3¶
Participants
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: submission_LM_DS_3
- Participant: IISER22
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
5314a08fca0717d17ef9089f9ae293bf
- Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMDirichlet Similarity to retrieval relevant document for every query.Here system was selected first five document for each query.
submission_LM_JM_2¶
Participants
| Input
| Summary (auto)
| Summary (manual)
| Appendix
- Run ID: submission_LM_JM_2
- Participant: IISER22
- Track: CrisisFACTs
- Year: 2022
- Submission: 9/30/2022
- Type: automatic
- MD5:
0ad4b26f3c44e9307f9eaa1ef31a5751
- Run description: Atfirst I used Lucene to index all the documents event-day wise.After that I used LMJelinekMercer Similarity to retrieval relevant document for every query.Here system was selected first three document for each query.