Overview - Incident Streams 2018

Proceedings | Data | Runs | Participants

The Text Retrieval Conference (TREC) Incident Streams track is a new initiative that aims to mature social media-based emergency response technology. This initiative advances the state of the art in this area through an evaluation challenge, which attracts researchers and developers from across the globe. The 2018 edition of the track provides a standardized evaluation methodology, an ontology of emergency-relevant social media information types, proposes a scale for information criticality, and releases a dataset containing fifteen test events and approximately 20,000 labeled tweets. Analysis of this dataset reveals a significant amount of actionable information on social media during emergencies (> 10%). While this data is valuable for emergency response efforts, analysis of the 39 state-of-the-art systems demonstrate a performance gap in identifying this data. We therefore find the current state-of-the-art is insufficient for emergency responders’ requirements, particularly for rare actionable information for which there is little prior training data available.

Track coordinator(s):

• Richard McCreadie, University of Glasgow
• Cody Buntain, New York University
• Ian Soborof, National Institute of Standards and Technology (NIST)

Track Web Page: http://trecis.org/

---