Skip to content

Proceedings - Crowdsourcing 2013

Overview of the TREC 2013 Crowdsourcing Track

Mark D. Smucker, Gabriella Kazai, Matthew Lease

Abstract

n 2013, the Crowdsourcing track partnered with the TREC Web Track and had a single task to crowdsource relevance judgments for a set of Web pages and search topics shared by the Web Track. This track overview describes the track and provides analysis of the track's results.

Bibtex 
@inproceedings{DBLP:conf/trec/SmuckerKRL13,
    author = {Mark D. Smucker and Gabriella Kazai and Matthew Lease},
    editor = {Ellen M. Voorhees},
    title = {Overview of the {TREC} 2013 Crowdsourcing Track},
    booktitle = {Proceedings of The Twenty-Second Text REtrieval Conference, {TREC} 2013, Gaithersburg, Maryland, USA, November 19-22, 2013},
    series = {{NIST} Special Publication},
    volume = {500-302},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2013},
    url = {http://trec.nist.gov/pubs/trec22/papers/CROWD.OVERVIEW.pdf},
    timestamp = {Thu, 12 Mar 2020 00:00:00 +0100},
    biburl = {https://dblp.org/rec/conf/trec/SmuckerKRL13.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Northeastern University Runs at the TREC13 Crowdsourcing Track

Maryam Bashir, Jesse Anderton, Virgil Pavlu, Javed A. Aslam

Abstract

The goal of the TREC 2013 Crowdsourcing Track was to evaluate approaches to crowdsourcing high quality relevance judgments for web pages and search topics. This paper describes our submission to Crowdsourcing track. Participants of this track were required to assess documents judged on a six-point scale. Our approach is based on collecting a linear number of preference judgements, and combining these into nominal grades using a modified version of QuickSort algorithm.

Bibtex 
@inproceedings{DBLP:conf/trec/BashirAPA13,
    author = {Maryam Bashir and Jesse Anderton and Virgil Pavlu and Javed A. Aslam},
    editor = {Ellen M. Voorhees},
    title = {Northeastern University Runs at the {TREC13} Crowdsourcing Track},
    booktitle = {Proceedings of The Twenty-Second Text REtrieval Conference, {TREC} 2013, Gaithersburg, Maryland, USA, November 19-22, 2013},
    series = {{NIST} Special Publication},
    volume = {500-302},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2013},
    url = {http://trec.nist.gov/pubs/trec22/papers/northeastern-crowd.pdf},
    timestamp = {Thu, 12 Mar 2020 00:00:00 +0100},
    biburl = {https://dblp.org/rec/conf/trec/BashirAPA13.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}

Hrbust in TREC 2013: Crowdsourcing Track

Li Peng, Sun Bo-yu, Liu Yang, Zhang Tingting

Abstract

In the practical application of crowdsourcing, some unreliable workers have emerged due to profit driven. Their results seriously reduce the quality and bring about the initiator's judgment biases. In this paper, we creatively put forward a crowdsourcing fraud detection method based on psychological behavior analysis to find out the spammer according to the psychological difference between deception and reliable behavior by means of Ebbinghaus forgetting curve. Furthermore, we constructed an online crowdsourcing experiment platform to verify the validity of our method. In addition, we participated in TREC 2013 Crowdsourcing Track and the organizer provided the evaluation results for our run. As a result, APCorr, RMSE and GAP attained 0.480, 0.135 and 0.392 respectively. Evaluation and xperimental results show that our method is effective and feasible.

Bibtex 
@inproceedings{DBLP:conf/trec/PengBYT13,
    author = {Li Peng and Sun Bo{-}yu and Liu Yang and Zhang Tingting},
    editor = {Ellen M. Voorhees},
    title = {Hrbust in {TREC} 2013: Crowdsourcing Track},
    booktitle = {Proceedings of The Twenty-Second Text REtrieval Conference, {TREC} 2013, Gaithersburg, Maryland, USA, November 19-22, 2013},
    series = {{NIST} Special Publication},
    volume = {500-302},
    publisher = {National Institute of Standards and Technology {(NIST)}},
    year = {2013},
    url = {http://trec.nist.gov/pubs/trec22/papers/Hrbust-crowd.pdf},
    timestamp = {Sun, 03 Jan 2021 00:00:00 +0100},
    biburl = {https://dblp.org/rec/conf/trec/PengBYT13.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
}