Runs - Crowdsourcing 2012¶

BUPTPRISZHS¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: BUPTPRISZHS
Participant: pris411
Track: Crowdsourcing
Year: 2012
Submission: 8/29/2012
Task: trat
MD5: 006e998ac6bbf33cc0e08ef50c8c6b13
Run description: This is our first run.Results are obtained on the CrowdFlower platform.

INFLB2012¶

Participants | Input | Summary | Appendix

Run ID: INFLB2012
Participant: INFLBSTF
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 330d193a67ef520c2c9417258087370e
Run description: Our approach includes hybrid strategy using computer calculated results and crowdsourced results, and some interface issues on TRAT task.

NEUElo2¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: NEUElo2
Participant: NEU
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: b3cff6197ff1376211d1653b1b8b6f7b
Run description: We have used crowd sourcing to generate relevance judgements for pairs of documents in form of preference pair judgements. Elo rating algorithm is used to combine the preference judgements from crowds. The Elo rating system is a method for calculating the relative skill levels of players in two-player games such as chess.

NEUElo3¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: NEUElo3
Participant: NEU
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: f393c26eed407caf592ac03d0bf1a90b
Run description: We have used crowd sourcing to generate relevance judgements for pairs of documents in form of preference pair judgements. Elo rating algorithm is used to combine the preference judgements from crowds. The Elo rating system is a method for calculating the relative skill levels of players in two-player games such as chess.

NEUElo4¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: NEUElo4
Participant: NEU
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: bc5fc1aeda9e9da915d280b0cc2b873d
Run description: We have used crowd sourcing to generate relevance judgements for pairs of documents in form of preference pair judgements. Elo rating algorithm is used to combine the preference judgements from crowds. The Elo rating system is a method for calculating the relative skill levels of players in two-player games such as chess.

NEUElo5¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: NEUElo5
Participant: NEU
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: d815ef0ddd29c2cc91bbb4b09784bda7
Run description: We have used crowd sourcing to generate relevance judgements for pairs of documents in form of preference pair judgements. Elo rating algorithm is used to combine the preference judgements from crowds. The Elo rating system is a method for calculating the relative skill levels of players in two-player games such as chess.

NEUEM1¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: NEUEM1
Participant: NEU
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: c1d06c302304323af38de35709a95775
Run description: This is the EM algorithm for acquiring relevant judgements through crowdsourcing.

NEUNugget12¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: NEUNugget12
Participant: NEU
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 3f15c04bbb021b169ecbbb877a8db395
Run description: 1 assessor takes about 45 min/query to judge documents and nuggets (extracted automatically from relevant documents) in an iterative process. When done, all documents are ranked given their match with the assessed nuggets.

Orc2G¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: Orc2G
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: c790edf41c0822d8e227243df383773f
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. A simplified version of Independent Bayesian Classifier Combination was applied, learning from Topic features extracted from the text. Reliability of workers is also learnt from test examples and used to weight crowdsourced labels.

Orc2GUL¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: Orc2GUL
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 10fcb21dd1e707b287ca921155f2016b
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. A simplified version of Independent Bayesian Classifier Combination was applied, learning from Topic features extracted from the text. Reliability of workers is also learnt from test examples and used to weight crowdsourced labels. Crowdsourced labels are taken into account when creating the final classification, in contrast to Orc2G, which only uses them to train a classifier.

Orc2GULConf¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: Orc2GULConf
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 4ffc76fee627b82ed3023e582004fd85
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. A simplified version of Independent Bayesian Classifier Combination was applied, learning from Topic features extracted from the text. Reliability of workers is also learnt from test examples and used to weight crowdsourced labels. Crowdsourced labels are taken into account when creating the final classification, in contrast to Orc2G, which only uses them to train a classifier. Confidence labels for the individual responses from the crowd are used to weight more confident responses more strongly when using the labels in classification.

Orc2Stage¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: Orc2Stage
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 44f3211a11fb75191f84863a023c6302
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. A simplified version of Independent Bayesian Classifier Combination was applied, learning from Topic features extracted from the text.

OrcVB1¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: OrcVB1
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 386975a8ed10e38d6df48df873a7a630
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. Independent Bayesian Classifier Combination was applied, treating all crowd members as equal and learning from Topic features extracted from the text.

OrcVB1Conf¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: OrcVB1Conf
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 768ff24c9d31a19e114ab09ada2965da
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. Independent Bayesian Classifier Combination was applied, treating all crowd members as equal and learning from Topic features extracted from the text. The crowd give estimates of their confidence when providing a label, which is taken into account by the classifier.

OrcVBW16Conf¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: OrcVBW16Conf
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: ddeb22e6987c081b7f9bf930be02ecdc
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. Independent Bayesian Classifier Combination was applied to crowdsourced labels, learning from Topic features extracted from the text. Reliability of workers is also learnt from the data and from test examples and used to weight crowdsourced labels. Confidence labels for the individual responses from the crowd are used to weight more confident responses more strongly. This version of the classifier uses middling priors that balance the prior belief that crowd members are accurate with the ability to spot the few that are not from patterns in the data. This allows us to disregard or treat as expert some of the responses from the crowd.

OrcVBW80¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: OrcVBW80
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 7147eae96835b94814faedb9cf4217d4
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. Independent Bayesian Classifier Combination was applied, learning the reliability of individual crowd members and learning from Topic features extracted from the text. The crowd give estimates of their confidence when providing a label, which is taken into account by the classifier as it learns how to use the confidence label from test data.

OrcVBW80Conf¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: OrcVBW80Conf
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: b57ba0f289868314e59098e2a78a9e34
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. Independent Bayesian Classifier Combination was applied, learning the reliability of individual crowd members and learning from Topic features extracted from the text. The crowd give estimates of their confidence when providing a label, which is taken into account by the classifier.

OrcVBW9Conf¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: OrcVBW9Conf
Participant: HAC
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 1e69d3c636c9f88ff66acf3c868d29c7
Run description: Using topic analysis to select files to crowdsource, we obtained 2600 labels from Amazon Mechanical Turk workers. Independent Bayesian Classifier Combination was applied, learning the reliability of individual crowd members and learning from Topic features extracted from the text. The crowd give estimates of their confidence when providing a label, which is taken into account by the classifier as it learns how to use the confidence label from test data. This uses weaker priors so that worker reliability is inferred in a semi-supervised manner from the data.

SetuServtest¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SetuServtest
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 8/31/2012
Task: trat
MD5: 589d715b34113b188e691f90b6c05327
Run description: To test submission

SSEC3excl¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSEC3excl
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: a1942c728d00fb41970e57e7f9674b7d
Run description: Post Error Correction (likely errors are identified using M/L), counting "not sures" as irrelevant documents

SSEC3incl¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSEC3incl
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: b9e4dc6823dbdd021e33cfcde266bf53
Run description: Post Error Correction (likely errors are identified using M/L), counting "not sures" as relevant documents

SSEC3inclML¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSEC3inclML
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 43140dd7564b760e145b49df4baa1499
Run description: Post Error Correction (likely errors are identified using M/L), counting "not sures" with ML score > 50% as relevant docs & remaining "not sures" as irrelevant

SSECML2to99¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSECML2to99
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: 57163b7d0abc98876a0f61d38ab94207
Run description: Post Error Correction (likely errors are identified using M/L for document with confidence scores between 2% and 99%), counting "not sures" as relevant documents

SSECML50pct¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSECML50pct
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: f2073d1363b6ddd35f2b4031ac5400c1
Run description: Post Error Correction (likely errors are identified using M/L for document with confidence score of >50%), counting "not sures" as relevant documents

SSECML75pct¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSECML75pct
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/4/2012
Task: trat
MD5: afd2624c3ec54f02d1e4b73af5b63fbe
Run description: Post Error Correction (likely errors are identified using M/L for document with confidence score of >75%), counting "not sures" as relevant documents

SSML2pct¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSML2pct
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/3/2012
Task: trat
MD5: e87e2d05b7f58c02f2d3bf8dbf6de3cd
Run description: Crowdsourced using Skierarchy approach for pairs with machine learning score of greater than 2%. Remaining documents are marked as not relevant

SSNoEC¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSNoEC
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/3/2012
Task: trat
MD5: b9f8cb3cb05ef991c35c7b78d1ae4f83
Run description: Crowdsourced using Skierarchy approach for all pairs (no error correction & no plurality)

SSPostEC¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSPostEC
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/15/2012
Task: irat
MD5: 1e66d0d472b86052e04fa395d1e1edb8
Run description: Post Error Correction

SSPostECv2¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSPostECv2
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/16/2012
Task: irat
MD5: 8022401c7c17dd08fd34a3676824e372
Run description: Final - derived using 2 annotators and 1 person for error correction

SSPreEC¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: SSPreEC
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/15/2012
Task: irat
MD5: ad781a89b1611187f7479f498291e6c3
Run description: Prior to Error Correction

testrun¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: testrun
Participant: SetuServ
Track: Crowdsourcing
Year: 2012
Submission: 9/13/2012
Task: irat
MD5: c6ed210b6460e8b89497904c276b26a8
Run description: Test run to see if the submission goes through

UIowaS01r¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: UIowaS01r
Participant: UIowaS
Track: Crowdsourcing
Year: 2012
Submission: 9/3/2012
Task: trat
MD5: f90b2b642a2239c81c0016dcf437d68a
Run description: Ranked by a weighted score. Partitioned into fixed sizes and sent to the crowd in ranked order.

UIowaS02r¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: UIowaS02r
Participant: UIowaS
Track: Crowdsourcing
Year: 2012
Submission: 9/3/2012
Task: trat
MD5: 182d4b56504a06b3c9abc93784d75d4d
Run description: Used k-means text clustering. Each cluster ranked by an average of each document's weighted score. Sent to the crowd in ranked order first by cluster, then by order within the cluster.

UIowaS03r¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: UIowaS03r
Participant: UIowaS
Track: Crowdsourcing
Year: 2012
Submission: 9/3/2012
Task: trat
MD5: 89199dfeb48c12deda26e2020b7f96b7
Run description: Used k-means text clustering. Each cluster ranked by an average of each document's weighted score. Sent to the crowd in ranked order by cluster, then by a random sample of documents within the cluster.

UTAustinM¶

Participants | Input | Summary | Appendix

Run ID: UTAustinM
Participant: UTAustin
Track: Crowdsourcing
Year: 2012
Submission: 9/15/2012
Task: irat
MD5: b5d94b813eb76df004d3b4bb9814e2b6
Run description: Incremental Crowd-sourcing by improving the level of agreement per each HIT and increasing a pool of trustful workers.

yorku12cs01¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: yorku12cs01
Participant: york
Track: Crowdsourcing
Year: 2012
Submission: 8/28/2012
Task: trat
MD5: 35d4a682ea23892690ceb2b57097f14e
Run description: Judgements partially based on fusion with TF and partially from CrowdFlower.

yorku12cs02¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: yorku12cs02
Participant: york
Track: Crowdsourcing
Year: 2012
Submission: 8/28/2012
Task: trat
MD5: cad8f20805028ce5be02cade3276a845
Run description: Judgements based on fusion with extracted expansion words from crowdFlower.

yorku12cs03¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: yorku12cs03
Participant: york
Track: Crowdsourcing
Year: 2012
Submission: 8/28/2012
Task: trat
MD5: 63542a908ff906d6805ff32ce1fc5480
Run description: Judgements based on fusion with expansion words from term proximity.

yorku12cs04¶

Participants | Proceedings | Input | Summary | Appendix

Run ID: yorku12cs04
Participant: york
Track: Crowdsourcing
Year: 2012
Submission: 8/28/2012
Task: trat
MD5: 6a971cb0e2f35fe6db8d958c7f280e99
Run description: Judgements based on fusion with expansion words from pseudo feedback and partially from crowdFlower.