Skip to content

Runs - Fair Ranking 2021

1step_pair

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: 1step_pair
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/3/2021
  • Task: coordinators
  • MD5: 163470b4bb08462ed28f9a9858509942
  • Run description: A static method with one filtering step: first 4500 top-ranked docs are selected.

1step_pair_list

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: 1step_pair_list
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/3/2021
  • Task: coordinators
  • MD5: 33b6d4de3944ec51609ff42bf2e5a128
  • Run description: A static method with one filtering step: first 4500 top-ranked docs are selected. The best-performing method from two different rankers is selected.

2step_pair

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: 2step_pair
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/3/2021
  • Task: coordinators
  • MD5: 6c51fa83c313431785e214465044f5ce
  • Run description: A static method with two filtering steps: first 4500 top-ranked docs are selected, then the union of the output of two different fair rankers is considered (around 1200 docs).

2step_pair_list

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: 2step_pair_list
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/3/2021
  • Task: coordinators
  • MD5: 8c4a6c44843e984f5c52eeb4f93646ad
  • Run description: A static method with two filtering steps: first 4500 top-ranked docs are selected, then the union of the output of two different fair rankers is considered (around 1200 docs).

pl_control_0.6

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: pl_control_0.6
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: editors
  • MD5: 71868ba69e56875f73389538962f94d2
  • Run description: 1) The method for this run is randomized. 2) For a given query, across various stages, ranking at the current stage depends on the ranked list from the previous stage. In each iteration we compute a score for each of the top-K documents in the current state, based on the expected exposure of each group so far (we compute an advantage) and the original estimated relevance score. Then we sample documents with respect to this score. 3) It uses the article's quality score in the ranking.

pl_control_0.8

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: pl_control_0.8
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: editors
  • MD5: 6e83058d006e8ccdecf828b14944981a
  • Run description: 1) The method for this run is randomized. 2) For a given query, across various stages, ranking at the current stage depends on the ranked list from the previous stage. In each iteration we compute a score for each of the top-K documents in the current state, based on the expected exposure of each group so far (we compute an advantage) and the original estimated relevance score. Then we sample documents with respect to this score. 3) It uses the article's quality score in the ranking.

pl_control_0.92

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: pl_control_0.92
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: editors
  • MD5: f71ca48e5c289573bfd273bad883a013
  • Run description: 1) The method for this run is randomized. 2) For a given query, across various stages, ranking at the current stage depends on the ranked list from the previous stage. In each iteration we compute a score for each of the top-K documents in the current state, based on the expected exposure of each group so far (we compute an advantage) and the original estimated relevance score. Then we sample documents with respect to this score. 3) It uses the article's quality score in the ranking.

PL_IRLab_05

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: PL_IRLab_05
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/3/2021
  • Task: editors
  • MD5: 40b9ce698ba4a9f1cfec9a2a3842ba01
  • Run description: randomized method with a two-staged Plackett-Luce sampling. Work needed is multiplied by the relevance.

PL_IRLab_07

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: PL_IRLab_07
  • Participant: IRLab-Amsterdam
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/3/2021
  • Task: editors
  • MD5: 66f4ee469c3f25048e4427f37237d193
  • Run description: randomized method with a two-staged Plackett-Luce sampling. Work needed is multiplied by the relevance.

RMITRet

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: RMITRet
  • Participant: RMIT-IR
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: d819844c043d2156a02203aad96bc739
  • Run description: We used the pyserini package to index the documents with Lucene inverted indexes and also for retrieval with BM25.

RMITRetRerank_1

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: RMITRetRerank_1
  • Participant: RMIT-IR
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: 9a8f1d04faf01ce9f9bbc1b289faf6fc
  • Run description: All indexing was done using Lucene inverted indexes from the pyserini package. Retrieval was performed with BM25 from pyserini and re-ranked using implicit diversification technique (MMR) with default lambda value = 0.5. Since implicit diversification was used no groups such as geographic locations or topic categories were involved while re-ranking.

RMITRetRerank_2

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: RMITRetRerank_2
  • Participant: RMIT-IR
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: c43f6aa8623bd995c6e6e45071f03ac1
  • Run description: All indexing was done using Lucene inverted indexes from the pyserini package. Retrieval was performed with BM25 from pyserini and re-ranked using implicit diversification technique (MMR) with lambda value = 0.3. Since implicit diversification was used no groups such as geographic locations or topic categories were involved while re-ranking.

RUN1

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: RUN1
  • Participant: TKB48
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/1/2021
  • Task: coordinators
  • MD5: 01d1760804b67bd002814715c7c6c8fd
  • Run description: 1. Rerank the ranking from BM25 (2000 documents for init ranking) 2. Choose document one by one 3. Calculate fairness based on how many documents still need to be put into the ranking. More need higher score. 4. Put document with highest score one by one.

RUN_task2

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: RUN_task2
  • Participant: TKB48
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/1/2021
  • Task: editors
  • MD5: 8c521c31bd914b1bff090535ae436402
  • Run description: 1. Rerank the ranking from BM25 (5000 documents for init ranking) 2. Choose document one by one 3. Calculate fairness based on how many documents still need to be put into the ranking. More need higher score. 4. Because we need to return 100 rankings. We focus on first ranking, and choose one document with highest score (Which combine the relevance and fairness and quality score). Then we focus on second ranking, and choose one document with the highest score. ... After we put one document for 100 rankings, we decide the second document for rankings one by one.

UoGTrDExpDisLT1

Participants | Input | Summary | Appendix

  • Run ID: UoGTrDExpDisLT1
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: b34beaaa320df15f5b86475a55eef2fb
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both the geographic location attribute and an inferred demographic attribute. The approach combines the first-pass retrieval with a tailored diversification plus data fusion fairness component. The fairness component is optimised to take into consideration a protected groups distribution in the background collection and the total relevance of the group in the candidate results set. This run differs from our UoGTrDivExpDispT1 run in that it integrates a parameter for learning the importance of the group relevances and background distributions.

UoGTrDExpDisT1

Participants | Input | Summary | Appendix

  • Run ID: UoGTrDExpDisT1
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: 6099caca4e6c38505b0a1af46e05e957
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both the geographic location attribute and an inferred demographic attribute. The approach combines the first-pass retrieval scores with a tailored diversification plus data fusion fairness component. The fairness component is optimised to take into consideration a protected groups distribution in the background collection and the total relevance of the group in the candidate results set.

UoGTrDExpDisT2

Participants | Input | Summary | Appendix

  • Run ID: UoGTrDExpDisT2
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: editors
  • MD5: f1778245b833c08f378e5d2f5574e13c
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both the geographic location attribute and an inferred demographic attribute. The approach combines the first-pass retrieval with a tailored diversification and data fusion fairness component. The fairness component is optimised to take into consideration a protected groups distribution in the background collection and the total relevance of the group in the candidate results set. For a sequence of repeated queries, the approach continually optimises the selection of documents to minimise the disparity between a groups expected and actual exposures.

UoGTrDivPropT1

Participants | Input | Summary | Appendix

  • Run ID: UoGTrDivPropT1
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: 4c18eaf5cbdf6012022d63d6ea422152
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both geographic location and an inferred demographic attribute. The approach combines the first-pass retrieval relevance scores with a tailored diversification plus data fusion approach to prioritise highly relevant documents while matching the distributions of the protected groups in the generated ranking to their distributions in the background population.

UoGTrDivPropT2

Participants | Input | Summary | Appendix

  • Run ID: UoGTrDivPropT2
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: editors
  • MD5: afbe32a751e21a84fffa2627e7098205
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both geographic location and an inferred demographic attribute. The approach combines the first-pass retrieval relevance and work-needed scores with a tailored diversification plus data fusion approach to prioritise highly relevant documents while matching the distributions of the protected groups to their distributions in the background population. For repeated queries, the approach continually optimises the selection of documents to minimise the divergence, or skew, in the distributions of the protected groups over all of the rankings within a sequence, compared to the background population.

UoGTrDRelDiT1

Participants | Input | Summary | Appendix

  • Run ID: UoGTrDRelDiT1
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: 3bd8f532f283c07da392956adc410a56
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both the geographic location attribute and an inferred demographic attribute. The approach combines the first-pass retrieval relevance score with a tailored diversification plus data fusion approach that prioritises highly relevant documents, while allocating positions in the generated ranking to a protected group proportionally with respect to the total relevance score of the group within the candidate results set.

UoGTrDRelDiT2

Participants | Input | Summary | Appendix

  • Run ID: UoGTrDRelDiT2
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: editors
  • MD5: 2d972b5e2980ab0b7463daa2b0fa4555
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both the geographic location attribute and an inferred demographic attribute. The approach combines the first-pass retrieval relevance and work-needed scores with a tailored diversification plus data fusion approach that prioritises highly relevant documents, while allocating positions in the ranking to a protected group proportionally with respect to the total relevance score of the group within the candidate results set. For repeated queries, the approach continually optimises the selection of documents to

UoGTrLambT2

Participants | Input | Summary | Appendix

  • Run ID: UoGTrLambT2
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/3/2021
  • Task: editors
  • MD5: 34c69ac7024b61798f6e733a644d25b1
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to both the geographic location attribute and an inferred demographic attribute. The approach combines the first-pass retrieval with a tailored diversification and data fusion fairness component. The fairness component is optimised to take into consideration a protected groups distribution in the background collection and the total relevance of the group in the candidate results set. For a sequence of repeated queries, the approach continually optimises the selection of documents to minimise the disparity between a groups expected and actual exposures. This run differs from our UoGTrDivExpDispT2 run in that it integrates a parameter for learning the importance of the group relevances and background distributions.

UoGTrRelT1

Participants | Input | Summary | Appendix

  • Run ID: UoGTrRelT1
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: coordinators
  • MD5: 4a93e114c74c6a51d6a81ffd62311c0f
  • Run description: This run simply consists in ranking the documents according to their relevance with respect to the query. No fairness component is explicitly enforced.

UoGTrRelT2

Participants | Input | Summary | Appendix

  • Run ID: UoGTrRelT2
  • Participant: uogTr
  • Track: Fair Ranking
  • Year: 2021
  • Submission: 8/2/2021
  • Task: editors
  • MD5: 17f9e7e68841b308f4466363a54e0e19
  • Run description: This run simply consists in ranking, for each instance of a query in the sequence, the documents according to their relevance with respect to the query. No fairness component is explicitly enforced.