Skip to content

Runs - Fair Ranking 2020

Deltr-gammas

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: Deltr-gammas
  • Participant: MTG
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/25/2020
  • Task: rerank
  • MD5: 136501c4245817369a32e8ab9e3d4ed3
  • Run description: In this solution, we train two Deltr models using gamma values of zero and one. The models are trained using 'H class' as the protected attribute. We combine the results of both models using a parameter that makes different reranks for the same query.

LM-rel-groups

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: LM-rel-groups
  • Participant: MTG
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/25/2020
  • Task: rerank
  • MD5: 3064088722ff00e93209df9f1c9dd5da
  • Run description: In this solution, we use the relevance predicted with LambdaMART and rerank the results by only selecting one paper of each group until every group has a paper in the ranking. We compute the groups by clustering the authors from the graph of collaborations. To compute the clusters, we use the Louvain algorithm.

LM-rel-year-100

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: LM-rel-year-100
  • Participant: MTG
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/25/2020
  • Task: rerank
  • MD5: c82d9af3a67e66d9c398b7cf1b6e23de
  • Run description: This solution is the same as the previous "LM-relev-year" but using a smaller number of sequences for training LambdaMart (100 instead of 10000)

LM-relev-year

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: LM-relev-year
  • Participant: MTG
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/25/2020
  • Task: rerank
  • MD5: b2b918a3d839f70870822fc915dce378
  • Run description: This solution is the same as "LM-relevance" but including the attribute "year" as a feature for LambdaMART.

LM-relevance

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: LM-relevance
  • Participant: MTG
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/24/2020
  • Task: rerank
  • MD5: 1b1c71898ccdd2d4cc468a15077962e4
  • Run description: This submission uses the lambdamart to learn-to-rank and randomizes the results based on the predicted relevance

MacEwan-base

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: MacEwan-base
  • Participant: MacEwan_Biz
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: rerank
  • MD5: 13e19d7a8241840a4f04e23457afc0d8
  • Run description: Baseline run, similar to 2019 submission. Documents are ranked based on BM25 scores from query against title and abstract fields. The two rankings are merged. As the query repeats in the sequence, the weights used for merging the two lists are shifted.

MacEwan-norm

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: MacEwan-norm
  • Participant: MacEwan_Biz
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: rerank
  • MD5: d6407362bcdc74b1bf8864244a686295
  • Run description: The same as MacEwan-base, except normalized similarity scores are used for merging instead of ranks.

NLE_META_99_1

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: NLE_META_99_1
  • Participant: NLE
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/29/2020
  • Task: rerank
  • MD5: af839f3d174764ad17c4fe3b3418e587
  • Run description: Fairness is achieved by a P-controller that ensures that actual exposures for authors are as close as possible to their target exposures. This run uses the following controller gains: 0.99 for relevance and 0.01 for unfairness. The base re-ranker only use metadata (year, number of citations, ...) and content (i.e. text) information.

NLE_META_9_1

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: NLE_META_9_1
  • Participant: NLE
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/29/2020
  • Task: rerank
  • MD5: f4be06fddae406acdcf59b9230b3eef7
  • Run description: Fairness is achieved by a P-controller that ensures that actual exposures for authors are as close as possible to their target exposures. This run uses the following controller gains : 0.9 for relevance and 0.1 for unfairness. The base re-ranker only use metadata (year, number of citations, ...) and content (i.e. text) information.

NLE_META_PKL

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: NLE_META_PKL
  • Participant: NLE
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/29/2020
  • Task: rerank
  • MD5: 421d0c6a2ace139fc4328da9b272b9d8
  • Run description: It is our baseline run. Fairness is achieved by drawing rankings from a PK-Luce Distribution to randomize the PRP-based re-ranker. The base re-ranker uses metadata (year, number of citations, ...) and content (i.e. text) information.

NLE_TEXT_99_1

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: NLE_TEXT_99_1
  • Participant: NLE
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/29/2020
  • Task: rerank
  • MD5: 67e14bbabfb52fe56dad5efa136563d3
  • Run description: Fairness is achieved by a P-controller that ensures that actual exposures for authors are as close as possible to their target exposures. This run uses the following controller gains: 0.99 for relevance and 0.01 for unfairness. The base re-ranker only use content (i.e. text) information.

NLE_TEXT_9_1

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: NLE_TEXT_9_1
  • Participant: NLE
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/29/2020
  • Task: rerank
  • MD5: 2e9a11219fff82effabd9a9ff290392b
  • Run description: Fairness is achieved by a P-controller that ensures that actual exposures for authors are as close as possible to their target exposures. This run uses the following controller gains : 0.9 for relevance and 0.1 for unfairness.. The base re-ranker only uses content (i.e. text) information.

umd_relfair_ltr

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: umd_relfair_ltr
  • Participant: UMD_IR
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: rerank
  • MD5: 5d2fb3c535785b80f0f8074b6512e3a0
  • Run description: Listwise learning to rank trying to optimize a custom evaluation measure that balances between relevance and fairness. The method is static and no state between rankings.

UoGTrBComFu

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UoGTrBComFu
  • Participant: UoGTr
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/29/2020
  • Task: rerank
  • MD5: 1c54995218333d31cccd043fec53f327
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to a number of organically discovered communities that are generated using a graph embeddings approach. This variant of the approach deploys data fusion to promote different communities over time.

UoGTrBComPro

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UoGTrBComPro
  • Participant: UoGTr
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: rerank
  • MD5: d4a7ce8ace7ce3a98a99e1494e9121d7
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to a number of organically discovered communities that are generated using a graph embeddings approach. This variant of the approach ranks documents by combining the first-pass retrieval relevance score with community similarity scores to provide a fair exposure to the documents throughout the ranking.

UoGTrBComRel

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UoGTrBComRel
  • Participant: UoGTr
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: rerank
  • MD5: 58252b40c94923ab82282127059406d6
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to a number of organically discovered communities that are generated using a graph embeddings approach. This variant of the approach ranks documents by combining the first-pass retrieval relevance score with community similarity scores to provide a fair exposure to the documents in the ranking that are predicted to be most relevant.

UoGTrBRel

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UoGTrBRel
  • Participant: UoGTr
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: rerank
  • MD5: e1bd3a9b2fc60811dbe9f0a20bd9708d
  • Run description: This run simply consists in ranking, for each instance of a query in the sequence, the documents according to their relevance with respect to the query. No fairness component is explicitly enforced.

UoGTrComRel

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UoGTrComRel
  • Participant: UoGTr
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: rerank
  • MD5: 11c4542c8ec1c7aa16398c8236659d11
  • Run description: This run builds on the Terrier.org IR platform, to rank documents by relevance before applying a fairness component that aims to be fair to a number of organically discovered communities that are generated using a graph embeddings approach. This variant of the approach ranks documents by combining the first-pass retrieval relevance score with community similarity scores to provide a fair exposure to the documents in the ranking that are predicted to be most relevant. This approach differs from our UoGTrBComRel run in the first-pass retrieval model.

UW_bm25

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_bm25
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/27/2020
  • Task: rerank
  • MD5: 8fa716bda68877fb695c53e522d609a3
  • Run description: Used the output of OkapiBM25 directly for this run. OkapiBM25 how we judge the relevance of a document for a given query. As such, we predict this run will have the best utility.

UW_Kr_r0g0c100

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kr_r0g0c100
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/27/2020
  • Task: rerank
  • MD5: e77f30749ec10035f83a642fa993597f
  • Run description: Static method. Uses KL-Divergence to optimize fairness for predicted country groups (advanced/developing economies). Reference distribution is the overall country distribution for the query. Comparison distribution is the distribution of the documents up to rank i.

UW_Kr_r0g100c0

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kr_r0g100c0
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/27/2020
  • Task: rerank
  • MD5: d73f85fc4eed43d0801d42a1e0a80db2
  • Run description: Static method. Uses KL-Divergence to optimize fairness for predicted gender groups. Reference distribution is the overall gender distribution for the query. Comparison distribution is the distribution of the documents up to rank i.

UW_Kr_r25g25c50

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kr_r25g25c50
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/27/2020
  • Task: rerank
  • MD5: d86c19d7da127b523dc2f7027ac23f23
  • Run description: Static method. Reranked based on 3 parameters: relevance (from OkapiBM25), gender distribution of authors, and geographic distribution of authors (2 groups--advanced/developing nations). Weighted parameters at 25%, 25%, and 50%, respectively. Used KL-Divergence to assess fairness of author group distributions up to i. The reference distributions are the overall group distributions for a query.

UW_Kr_r60g20c20

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kr_r60g20c20
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/27/2020
  • Task: rerank
  • MD5: ea5504235b903c20d3dd132df769cb52
  • Run description: Static method. Reranked based on 3 parameters: relevance (from OkapiBM25), gender distribution of authors, and geographic distribution of authors (2 groups--advanced/developing nations). Weighted parameters at 60%, 20%, and 20%, respectively. Used KL-Divergence to assess fairness of author group distributions up to i. The reference distributions are the overall group distributions for a query.

UW_Kt_r0g0c100

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kt_r0g0c100
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: retrieval
  • MD5: 045981ed5b1cfb91473ebc3946b5c266
  • Run description: Static method. Retrieves top 100 results using OkapiBM25, then reranks based solely on predicted geographic groups of authors (advanced/developing nations). Document i in the reranked list results in the minimum KL-Divergence between group distribution up to i compared to the overall distribution for the query.

UW_Kt_r25g25c50

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kt_r25g25c50
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: retrieval
  • MD5: f955e1ee90e0b6d82dca0888203a1c85
  • Run description: Static method. Retrieves top 100 results using OkapiBM25, then reranks based on relevance (OkapiBM25 score), gender distribution, and country distribution (advanced/developing). Parameters weighted at 25%, 25%, and 50%, respectively.

UW_Kt_r60g20c20

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kt_r60g20c20
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: retrieval
  • MD5: 88c49ca236f445a525baea26c5c9f1d6
  • Run description: Static method. Retrieves top 100 results using OkapiBM25, then reranks based on relevance (OkapiBM25 score), gender distribution, and country distribution (advanced/developing). Parameters weighted at 60%, 20%, and 20%, respectively.

UW_Kt_r80g10c10

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_Kt_r80g10c10
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: retrieval
  • MD5: 1b1b6b5a26772f32ce66c2759e254de4
  • Run description: Static method. Retrieves top 100 results using OkapiBM25, then reranks based on relevance (OkapiBM25 score), gender distribution, and country distribution (advanced/developing). Parameters weighted at 80%, 10%, and 10%, respectively.

UW_t_bm25

Participants | Proceedings | Input | Summary | Appendix

  • Run ID: UW_t_bm25
  • Participant: InfoSeeking
  • Track: Fair Ranking
  • Year: 2020
  • Submission: 8/28/2020
  • Task: retrieval
  • MD5: 28ee2d36ed8e9a094b9e8c31e9dbb4ab
  • Run description: Static method. Result of running OkapiBM25 with no reranking of the top 100 results.