Skip to content

Runs - Incident Streams 2019

cbnuC1

Participants | Proceedings | Appendix

  • Run ID: cbnuC1
  • Participant: cbnu
  • Track: Incident Streams
  • Year: 2019
  • Submission: 10/15/2019
  • Task: main

cbnuS1

Participants | Proceedings | Appendix

  • Run ID: cbnuS1
  • Participant: cbnu
  • Track: Incident Streams
  • Year: 2019
  • Submission: 10/15/2019
  • Task: main

DLR_BERT_R

Participants | Proceedings | Appendix

  • Run ID: DLR_BERT_R
  • Participant: DLR_DW
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 08cfb6527ee117771e9dad75d48d14cb
  • Run description: CNN classifier with BERT word embeddings

DLR_Fusion

Participants | Proceedings | Appendix

  • Run ID: DLR_Fusion
  • Participant: DLR_DW
  • Track: Incident Streams
  • Year: 2019
  • Submission: 10/1/2019
  • Type: auto
  • Task: main
  • MD5: 37a3a2fcea078c4db52b95c3549d2e59
  • Run description: Last year's CNN model that combines sub-models trained on CrisisLex and CrisisNLP, and one trained from scratch

DLR_MeanMaxAAE_Regression

Participants | Proceedings | Appendix

  • Run ID: DLR_MeanMaxAAE_Regression
  • Participant: DLR_DW
  • Track: Incident Streams
  • Year: 2019
  • Submission: 10/15/2019
  • Task: main

DLR_SIF_R

Participants | Proceedings | Appendix

  • Run ID: DLR_SIF_R
  • Participant: DLR_DW
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 014f82d98ce3291790c0f8d0dcb666b4
  • Run description: DNN with CrisisNLP word embeddings and SIF sentence embeddings

DLR_USE_R

Participants | Proceedings | Appendix

  • Run ID: DLR_USE_R
  • Participant: DLR_DW
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: c11ff719282e437e8bb213a730cabf05
  • Run description: DNN with pre-trained Universal Sentence Encoder embeddings

ict_dl

Participants | Proceedings | Appendix

  • Run ID: ict_dl
  • Participant: ICTNET
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/27/2019
  • Type: manual
  • Task: main
  • MD5: 4340335dba6c706d05da64e467d8ac30
  • Run description: use lstm to predict the test incidents

IITBHU_run1

Participants | Proceedings | Appendix

  • Run ID: IITBHU_run1
  • Participant: IIT_BHU
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 75c93d35f3220824c2bebea2b35b58fb
  • Run description: Used TF-IDF vectorizer for feature extraction and multi label k-nearest neighbours to find nearest examples to test class which uses bayesian inference to select assigned labels.

IITBHU_run2

Participants | Proceedings | Appendix

  • Run ID: IITBHU_run2
  • Participant: IIT_BHU
  • Track: Incident Streams
  • Year: 2019
  • Submission: 10/1/2019
  • Type: auto
  • Task: main
  • MD5: f689606d035e627466527d4d1f568335
  • Run description: Used TF-IDF vectorizer for feature extraction and build kNN graph to learn embeddings. Specifically, divide the dataset into several clusters, and in each cluster, it detects embedding vectors by capturing non-linear label correlation and preserving the pairwise distance between labels.

Informedia-nb

Participants | Proceedings | Appendix

  • Run ID: Informedia-nb
  • Participant: CMUInformedia
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/27/2019
  • Type: auto
  • Task: main
  • MD5: dfc6128b475fee43ba72cf28bd4dec0d
  • Run description: Use the Naive Bayes model on features including statistical features and textual features (BERT/GloVe/SkipThought).

Informedia-rf1

Participants | Proceedings | Appendix

  • Run ID: Informedia-rf1
  • Participant: CMUInformedia
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/27/2019
  • Type: auto
  • Task: main
  • MD5: 6aa071e74e2eadbb14efb716f5d54c69
  • Run description: Use the Random Forest model on features including statistical features and textual features (BERT/GloVe/SkipThought). Use regression to get the priority score.

Informedia-rf2

Participants | Proceedings | Appendix

  • Run ID: Informedia-rf2
  • Participant: CMUInformedia
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/27/2019
  • Type: auto
  • Task: main
  • MD5: 2f3d78941297f7ab83ce5b323e52db02
  • Run description: Use the Random Forest model on features including statistical features and textual features (BERT/GloVe/SkipThought). Use a higher weight for the actionable categories during training, and merge the score from classification labels and from regression.

Informedia-rf3

Participants | Proceedings | Appendix

  • Run ID: Informedia-rf3
  • Participant: CMUInformedia
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/27/2019
  • Type: auto
  • Task: main
  • MD5: b484c23b047b96ea2edd2636e9081902
  • Run description: Use the Random Forest model on features including statistical features and textual features (BERT/GloVe/SkipThought). Use a very large weight for the actionable categories during training.

IRITrun1

Participants | Appendix

  • Run ID: IRITrun1
  • Participant: IRIT
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 6d1da6d201a4e2e9bc8813903ddfb9bf
  • Run description: Classic pre-processing (stopwords removing,...) and resampling for imbalanced classes. We used combination of Gradient Boosting and Random Forest classifiers. Binary Relevance is used to deal with multi-label classification.

IRITrun2

Participants | Appendix

  • Run ID: IRITrun2
  • Participant: IRIT
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 7c0d5d0e6d8c0aa8d074940fa32c9956
  • Run description: Classic pre-processing (stopwords removing,...). We used combination of Gradient Boosting and Random Forest classifiers. Binary Relevance is used to deal with multi-label classification.

IRITrun3

Participants | Appendix

  • Run ID: IRITrun3
  • Participant: IRIT
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: a1b2c4c0813ff9b81d6040ca965d3e14
  • Run description: Classic pre-processing (stopwords removing,...) and resampling for imbalanced classes. We used combination of Gradient Boosting and Random Forest classifiers. A threshold is used to deal with multi-label classification.

IRITrun4

Participants | Appendix

  • Run ID: IRITrun4
  • Participant: IRIT
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 2dc35c3ee1367fdf5b7cbac7ab405596
  • Run description: Classic pre-processing (stopwords removing,...). We used combination of Gradient Boosting and Random Forest classifiers. A threshold is used to deal with multi-label classification.

nyu.base.multi

Participants | Appendix

  • Run ID: nyu.base.multi
  • Participant: nyu-smapp
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: a39103f96689551a20a447e9a7b870e3
  • Run description: This method uses a tf-idf-based vectorizer with multiple classes, similar to nyu 2019a with the tweet source.

nyu.base.sing

Participants | Appendix

  • Run ID: nyu.base.sing
  • Participant: nyu-smapp
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 930d6c7bf339adc4ed260884f25b555d
  • Run description: This method uses a tfi-idf-based vectorizer essentially equivalent to the umd hcil-baseline version from 2018 and NYU 2019a baseline with tweaks on the training process and addition of tweet source (e.g., Twitter for iPhone) in features.

nyu.fast.multi

Participants | Appendix

  • Run ID: nyu.fast.multi
  • Participant: nyu-smapp
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: ef96b8dfebedda059c2a2b5809b4af72
  • Run description: This method uses a fasttext-based embedding with multiple classes, similar to nyu 2019a with a different classifier and including the tweet source.

nyu.fast.sing

Participants | Appendix

  • Run ID: nyu.fast.sing
  • Participant: nyu-smapp
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 68f063beb1721c4358e4c9d7c6f8b6e6
  • Run description: This method uses a fasttext-based embedding essentially equivalent to the umd hcil-fasttext version from 2018 and nyu 2019a with a different classifier and including the tweet source

run1_baseline

Participants | Appendix

  • Run ID: run1_baseline
  • Participant: UAGPLSI
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: a69a80f81b5a75fc3d328fbd91e613ce
  • Run description: This run is our baseline. For each information type and tweet, a value of similarity is calculated. This value takes into account the name of the information type, its description, and its low level types.

run2_negative

Participants | Appendix

  • Run ID: run2_negative
  • Participant: UAGPLSI
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: e6245bb8fa8689208e818b6f7bc42b15
  • Run description: This run uses the same technique as our baseline, but it only takes account those tweets with negative polarity. To detect the polarity of the tweets, a sentiment analysis approach is employed, trained with an external corpus in a different domain.

run3_irn

Participants | Appendix

  • Run ID: run3_irn
  • Participant: UAGPLSI
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: 1e75f51b4e8df1b88ce33cf72e427aef
  • Run description: This run uses the same technique as our baseline, but some tweets have their scores increased using the IR-n information retrieval system.

run4_all

Participants | Appendix

  • Run ID: run4_all
  • Participant: UAGPLSI
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: auto
  • Task: main
  • MD5: a719e60005ac72d1d9a5633300ed1f8c
  • Run description: This run combines the other three runs. It uses the same technique as our baseline, but some tweets have their scores increased using the IR-n information retrieval system. In addition, it only takes account those tweets with negative polarity. To detect the polarity of the tweets, a sentiment analysis approach is employed, trained with an external corpus in a different domain.

UCDbaseline

Participants | Proceedings | Appendix

  • Run ID: UCDbaseline
  • Participant: CS-UCD
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/28/2019
  • Type: auto
  • Task: main
  • MD5: bd44c97401e358af4b7d635555ee3a7f
  • Run description: This run is from run 3 at trec-is 2019-a edition. Feature matrix is constructed by a pre-trained word2vec model(2016, Muhammad) in domain (300 word2vec features) and 21 hand-crafted features for performance boosting in actionable types classification. The model combining Logistic Regression with Naive Bayes is trained on previous labeled dataset. SMOTE is applied to leverage the imbalanced classes in training set. Priority is estimated by a linear combination of quantitative analysis and a priority classifier.

UCDbcnelmo

Participants | Proceedings | Appendix

  • Run ID: UCDbcnelmo
  • Participant: CS-UCD
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/28/2019
  • Type: auto
  • Task: main
  • MD5: 6f11767e1f52c348795ddc5edef88d0c
  • Run description: This run is based on BCN+ELMo. GPT-2 is applied to leverage the imbalanced classes in training set. Priority is estimated by a linear combination of quantitative analysis and the trained bi-lstm prediction model for priority.

UCDbilstmalpha

Participants | Proceedings | Appendix

  • Run ID: UCDbilstmalpha
  • Participant: CS-UCD
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/28/2019
  • Type: auto
  • Task: main
  • MD5: 88eb57beb1665d679e5c2dc33038d493
  • Run description: This run is based on a simple bi-lstm model. Texts are embedded with glove word2vec first and then encoded by a bi-lstm encoder and finally fed to a fedforward network. GPT-2 is applied to leverage the imbalanced classes in training set. Priority is estimated by a linear combination of quantitative analysis and the trained bi-lstm prediction model for priority.

UCDbilstmbeta

Participants | Proceedings | Appendix

  • Run ID: UCDbilstmbeta
  • Participant: CS-UCD
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/28/2019
  • Type: auto
  • Task: main
  • MD5: de996b5ac5abf34fb558ec04cbf4f265
  • Run description: The main difference from UCDbilstmalpha.run are follows: 1. char-cnn applied in embedding layer 2. no data augmentation with gpt-2 3. loss weight for objective/loss function to conquer class imbalance

UPB-BERT

Participants | Proceedings | Appendix

  • Run ID: UPB-BERT
  • Participant: DICE_UPB
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/27/2019
  • Type: auto
  • Task: main
  • MD5: 3f08028c2870ff4fdf65adc3a8438279
  • Run description: We use a fine-tuned BERT model to classify tweets into multi-label information types. First, we clean tweets from URLs, usernames, hashtags and special characters. Then, we use the contextualized word embedding from BERT to represent tweets. Finally, we feed these BERT embedding features to our model to general a list of relevant information types.

UPB-FOCAL

Participants | Proceedings | Appendix

  • Run ID: UPB-FOCAL
  • Participant: DICE_UPB
  • Track: Incident Streams
  • Year: 2019
  • Submission: 9/30/2019
  • Type: manual
  • Task: main
  • MD5: 43732f0b1c1ccae23da9454d0388b068
  • Run description: We use a fine-tuned BERT model with focal loss function to cateogrize tweets as multi-label information types.