Runs - Legal 2007¶

catchup0701p¶

Run ID: catchup0701p
Participant: uamsterdam.deRijke
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: automatic
Task: main
Run description: For the basic run, we index for each document the WHOLE text, including the XML tags. They should not have much effect, since they are usually the same for each document. We just take the request as is and retrieve it from that (quite huge) index.

CMU07RBase¶

Run ID: CMU07RBase
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/6/2007
Type: manual
Task: routing
Run description: baseline run with ranking of final boolean query filtered results first, and then appending with results from bag of word boolean query terms.

CMU07RFBSVME¶

Run ID: CMU07RFBSVME
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: routing
Run description: expanding the query with terms extracted from highest weight SVM features learned from training documents. 40 terms are selected for expansion for each topic. This new query is used to rerank the documents matching the final negotiated Boolean query. To make up for 26000 documents, results from the expanded queries are used to patch up the Boolean result lists.

CMU07RSVMNP¶

Run ID: CMU07RSVMNP
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/6/2007
Type: manual
Task: routing
Run description: Queries with positive and negative term weights that approximates SVM classifiers learned from training documents. Top 100 weighted terms (with positive and negative weights) are selected for each topic. This new query is used to rerank the documents matching the final negotiated Boolean query. Those documents that do not match the boolean query are ranked and used to append the Boolean result lists.

CMUL07ibp¶

Run ID: CMUL07ibp
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: manual
Task: main
Run description: Convert final boolean queries to Indri structured queries with boolean constraints. The ranking function treats each phrase as a query term.

CMUL07ibs¶

Run ID: CMUL07ibs
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: manual
Task: main
Run description: Convert final boolean queries to Indri structured queries with boolean constraints. The ranking function treats terms connected by 'OR' as synonyms.

CMUL07ibt¶

Run ID: CMUL07ibt
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: manual
Task: main
Run description: Convert final boolean queries to Indri structured queries with boolean constraints. The ranking is based on bag-of-words representation without any operators.

CMUL07irs¶

Run ID: CMUL07irs
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: manual
Task: main
Run description: Convert final boolean queries to Indri structured queries without boolean constraints. The ranking function treats terms connected by 'OR' as synonyms.

CMUL07irt¶

Run ID: CMUL07irt
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: manual
Task: main
Run description: Convert final boolean queries to Indri structured queries without boolean constraints. The ranking is based on bag-of-words representation of the query without any operators.

CMUL07o1¶

Run ID: CMUL07o1
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: manual
Task: main
Run description: Okapi ranking based on bag-of-words representation of the final boolean query. Ignore boolean constraints.

CMUL07o3¶

Run ID: CMUL07o3
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/4/2007
Type: manual
Task: main
Run description: Okapi ranking based on bag-of-words representation of the final boolean query, initial proposal by defendant and rejoinder by plaintiff. Ignore boolean constraints.

CMUL07std¶

Run ID: CMUL07std
Participant: cmu.dir.callan
Track: Legal
Year: 2007
Submission: 8/3/2007
Type: manual
Task: main
Run description: This is the standard condition run. It uses terms in request text, excluding some headers that are not meaningful for the query (e.g. "Please produce any and all documents that discuss ...").

Dartmouth1¶

Run ID: Dartmouth1
Participant: dartmouth.thompson
Track: Legal
Year: 2007
Submission: 7/30/2007
Type: automatic
Task: main
Run description: It is a automatic run that only use request text as the query sentence. It is the result returned by Indri.

fdwim7rs¶

Run ID: fdwim7rs
Participant: fudanu.niu
Track: Legal
Year: 2007
Submission: 8/2/2007
Type: automatic
Task: main
Run description: On the base of run fdwim7ts, we write a program to remove meaningless words from as indri's input

fdwim7sl¶

Run ID: fdwim7sl
Participant: fudanu.niu
Track: Legal
Year: 2007
Submission: 8/2/2007
Type: automatic
Task: main
Run description: It is mostly same as fdwim7ss despite that we run the retrieval on an non-preprocessed index

fdwim7ss¶

Run ID: fdwim7ss
Participant: fudanu.niu
Track: Legal
Year: 2007
Submission: 8/2/2007
Type: automatic
Task: main
Run description: On the base of run fdwim7rs, we try a new method to extract representative words from fields "instruction" "definition" , "background" adding to the input of retrieval

fdwim7ts¶

Run ID: fdwim7ts
Participant: fudanu.niu
Track: Legal
Year: 2007
Submission: 8/2/2007
Type: automatic
Task: main
Run description: In spite of removing prepositions and "the", "a" (such meaningless words), run of fdwim7ts does not take any optimizing steps.

fdwim7xj¶

Run ID: fdwim7xj
Participant: fudanu.niu
Track: Legal
Year: 2007
Submission: 8/1/2007
Type: manual
Task: main
Run description: As the baseline, fdwim7xj does not make lots of pre-process. All steps mainly rely on indri search engine.

IowaSL0702¶

Run ID: IowaSL0702
Participant: uiowa.srinivasan
Track: Legal
Year: 2007
Submission: 8/2/2007
Type: automatic
Task: main
Run description: This run is processed the same as IowaSL07Ref, except that all three boolean queries are converted to term vectors, and added to the query.

IowaSL0703¶

Run ID: IowaSL0703
Participant: uiowa.srinivasan
Track: Legal
Year: 2007
Submission: 8/3/2007
Type: automatic
Task: main
Run description: This run is processed in the same manner as IowaSL0702, except that the wildcards in the Boolean queries, previously stripped, are evaluated, and the two candidates with the greatest document frequencies are added to the query. I should note that all scoring/reranking for the IowaSL07* runs are based on Okapi term weights.

IowaSL0704¶

Run ID: IowaSL0704
Participant: uiowa.srinivasan
Track: Legal
Year: 2007
Submission: 8/3/2007
Type: automatic
Task: main
Run description: This run is processed in the same manner as IowaSL0703, except that the top three documents from IowaSL0703 are scanned for terms that have a high local frequency relative to the rest of the corpus. The best five candidates are added to the query term vector, provided they are not already in the query, not on the SMART list of stop words, do not contain any punctuation or numeric characters, and that they have a valid entry in WordNet.

IowaSL0705¶

Run ID: IowaSL0705
Participant: uiowa.srinivasan
Track: Legal
Year: 2007
Submission: 8/3/2007
Type: automatic
Task: main
Run description: This run is processed in the same manner as IowaSL0704, except that the top three documents from a prior run with this configuration provide the basis for the query expansion terms. For this run, the system has filtered our terms from the RequestText topic field that exist in that field for more than 10 of the topics.

IowaSL0706¶

Run ID: IowaSL0706
Participant: uiowa.srinivasan
Track: Legal
Year: 2007
Submission: 8/3/2007
Type: automatic
Task: main
Run description: This run was conducted in the same manner as IowaSL0705, except that the two indicated Boolean query fields were dropped from the query term vector for the initial feedback retrieval and the final retrieval.

IowaSL0707¶

Run ID: IowaSL0707
Participant: uiowa.srinivasan
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: Same as IowaSL0706 except that the Request Text was edited manually instead of automatically.

IowaSL07Ref¶

Run ID: IowaSL07Ref
Participant: uiowa.srinivasan
Track: Legal
Year: 2007
Submission: 8/2/2007
Type: automatic
Task: main
Run description: The reference run. The index consists of the document title and ocr data indexed as one field, generated using Lucene's StandardAnalyzer. Lucene is used to retrieve max(25000, B) * 1.5 documents. We post-process, automatically assigning a new score to the documents (the submitted score) and reranking.

LIU¶

Run ID: LIU
Participant: liu.chu
Track: Legal
Year: 2007
Submission: 8/15/2007
Task: interactive

otL07db¶

Run ID: otL07db
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/28/2007
Type: manual
Task: main
Run description: boolean run based on initial proposal by defendant (same as otL07fb except that the defendant boolean query was used instead of the final query)

otL07fb¶

Run ID: otL07fb
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/14/2007
Type: manual
Task: main
Run description: boolean run based on final negotiated query (same as reference boolean run refL07B except re-ordered by relevance)

otL07fb2x¶

Run ID: otL07fb2x
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/28/2007
Type: manual
Task: main
Run description: boolean run based on doubling the word distances in the final negotiated query (same as otL07fb except that the word proximity distances were doubled, e.g. w/20 was changed to w/40)

otL07fbe¶

Run ID: otL07fbe
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/28/2007
Type: manual
Task: main
Run description: blind feedback expansion of the final boolean query, based 50% on queries from first 2 retrieved rows of otL07fb and 50% on original otL07fb query

otL07frw¶

Run ID: otL07frw
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/28/2007
Type: manual
Task: main
Run description: weighted fusion of boolean and vector runs, weight 3 on otL07fb, weight 2 on otL07rvl, weight 1 on otL07fv

otL07fv¶

Run ID: otL07fv
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/14/2007
Type: manual
Task: main
Run description: vector run based on final negotiated query (same as otL07fb except that all boolean, phrase and proximity operators were changed to the OR operator)

otL07pb¶

Run ID: otL07pb
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/31/2007
Type: manual
Task: main
Run description: boolean run based on rejoinder by plaintiff (same as otL07fb except that the plaintiff boolean query was used instead of the final query); note, for some topics, the depth-25000 limit prevented some lower-ranking matches from being included

otL07rvl¶

Run ID: otL07rvl
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 7/28/2007
Type: manual
Task: main
Run description: vector run based on request text terms (same as otL07fv except (1) the terms were taken from the request text field instead of the final boolean field, (2) English inflections were matched, (3) common instruction words (e.g. "please", "produce", "documents") were manually removed)

otRF07fb¶

Run ID: otRF07fb
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 8/3/2007
Type: manual
Task: routing
Run description: boolean run based on final negotiated query (same as reference boolean run refL06B except re-ordered by relevance, and for some non-interactive topics, the depth-26000 limit prevented some lower-ranking matches from being included)

otRF07fv¶

Run ID: otRF07fv
Participant: open-text.tomlinson
Track: Legal
Year: 2007
Submission: 8/3/2007
Type: manual
Task: routing
Run description: vector run based on final negotiated query (same approach as otL07fv)

sab07legrf1¶

Run ID: sab07legrf1
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 7/18/2007
Type: manual
Task: routing
Run description: Optimize weights using up to top 50 Rocchio weighted terms. Limit number of expansion terms to avoid overfitting by automatic determination from split collection.

sab07legrf2¶

Run ID: sab07legrf2
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: routing
Run description: Optimized weights from legal 06 judgements, using all top 100 Rocchio terms for all topics (no attempt to avoid overfitting).

sab07legrf3¶

Run ID: sab07legrf3
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: routing
Run description: Same as run rf1, except rf1 inadvertantly used only judgments on odd-numbered docs from legal06, this uses all judgements, and is to be preferred in judging if possible.

SabL07ab1¶

Run ID: SabL07ab1
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: blind feedback, expansion by 20 terms from top 30 docs

SabL07ar1¶

Run ID: SabL07ar1
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: Basic tfidf vector run on request field only. Run on full OCR collection.

SabL07ar2¶

Run ID: SabL07ar2
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: Basic tfidf vector run on request field only. Same as ar1 except run on collection with most OCR terms deleted.

SabL07arbn¶

Run ID: SabL07arbn
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: Simple vector run using request plus boolean queries. Initial search corresponding to blind feedback run bf1

sableg07i1¶

Run ID: sableg07i1
Participant: sabir.buckley
Track: Legal
Year: 2007
Submission: 8/15/2007
Task: interactive

UIowa07LegE0¶

Results | Participants | Input | Summary | Appendix

Run ID: UIowa07LegE0
Participant: uiowa.eichman
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: baseline run - standard analysis of OCR and request text

UIowa07LegE1¶

Results | Participants | Input | Summary | Appendix

Run ID: UIowa07LegE1
Participant: uiowa.eichman
Track: Legal
Year: 2007
Submission: 8/6/2007
Type: automatic
Task: main
Run description: 3-4 ngram analysis on OCR and request text

UIowa07LegE2¶

Results | Participants | Input | Summary | Appendix

Run ID: UIowa07LegE2
Participant: uiowa.eichman
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: standard analysis on OCR, translation of boolean query

UIowa07LegE3¶

Results | Participants | Input | Summary | Appendix

Run ID: UIowa07LegE3
Participant: uiowa.eichman
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: 3-4 ngram analysis of OCR w/ boolean query

UIowa07LegE4¶

Results | Participants | Input | Summary | Appendix

Run ID: UIowa07LegE4
Participant: uiowa.eichman
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: automatic
Task: main
Run description: standard analysis on OCR with pseudo-relevance feedback on persons (authors, recipients and mentions) and production boxes

UIowa07LegE5¶

Results | Participants | Input | Summary | Appendix

Run ID: UIowa07LegE5
Participant: uiowa.eichman
Track: Legal
Year: 2007
Submission: 8/6/2007
Type: automatic
Task: main
Run description: ngram analysis on OCR with pseudo-relevance feedback on persons (authors, recipients and mentions) and production boxes

UMass10¶

Run ID: UMass10
Participant: umass.allan
Track: Legal
Year: 2007
Submission: 7/16/2007
Type: manual
Task: main
Run description: This is a hand edited version of the automatic queries generated for UMass11. It uses only the request text field.

UMass11¶

Run ID: UMass11
Participant: umass.allan
Track: Legal
Year: 2007
Submission: 7/16/2007
Type: automatic
Task: main
Run description: This run is automatically generated from the RequestText field only. The only metadata field used is date.

UMass12¶

Run ID: UMass12
Participant: umass.allan
Track: Legal
Year: 2007
Submission: 7/16/2007
Type: automatic
Task: main
Run description: This run is automatically generated from the final BooleanQuery field only. The only metadata field used is date.

UMass13¶

Run ID: UMass13
Participant: umass.allan
Track: Legal
Year: 2007
Submission: 7/16/2007
Type: automatic
Task: main
Run description: This run combines the queries generated from the requestText field (UMass11) and the final booleanQuery (UMass12). The only metadata field used is date.

UMass14¶

Run ID: UMass14
Participant: umass.allan
Track: Legal
Year: 2007
Submission: 7/29/2007
Type: automatic
Task: main
Run description: A term dependence model that makes use of phrases and term proximity was used on request text.

UMass15¶

Run ID: UMass15
Participant: umass.allan
Track: Legal
Year: 2007
Submission: 7/29/2007
Type: automatic
Task: main
Run description: A term dependence model that makes use of phrases and term proximity was used on request text. In addition, a standard pseudo-relevance feedback technique was employed.

UMKC1¶

Run ID: UMKC1
Participant: umkc.zhao
Track: Legal
Year: 2007
Submission: 7/30/2007
Type: automatic
Task: main
Run description: PBDf9E80LMa8b9.run --> UMKC1.run

UMKC2¶

Run ID: UMKC2
Participant: umkc.zhao
Track: Legal
Year: 2007
Submission: 7/30/2007
Type: automatic
Task: main
Run description: RTf9E80LMa8b9.run --> UMKC2

UMKC3¶

Run ID: UMKC3
Participant: umkc.zhao
Track: Legal
Year: 2007
Submission: 7/30/2007
Type: automatic
Task: main
Run description: FQf9E80LMa8b9.run --> UMKC3

UMKC4¶

Run ID: UMKC4
Participant: umkc.zhao
Track: Legal
Year: 2007
Submission: 7/30/2007
Type: automatic
Task: main
Run description: PBDf9E80.run --> UMKC4

UMKC5¶

Run ID: UMKC5
Participant: umkc.zhao
Track: Legal
Year: 2007
Submission: 7/30/2007
Type: automatic
Task: main
Run description: RTf9E80.run --> UMKC5

UMKC6¶

Run ID: UMKC6
Participant: umkc.zhao
Track: Legal
Year: 2007
Submission: 7/30/2007
Type: automatic
Task: main
Run description: FQf9E80.run --> UMKC6

ursinus1¶

Run ID: ursinus1
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/22/2007
Type: automatic
Task: main
Run description: request text, fourth root norm, no ocr, query pruning w/ old query file

ursinus2¶

Run ID: ursinus2
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/23/2007
Type: automatic
Task: main
Run description: request text, cube root norm, no ocr, query pruning w/ old query file

ursinus3¶

Run ID: ursinus3
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/24/2007
Type: automatic
Task: main
Run description: sqrt norm, no ocr, query pruning

ursinus4¶

Run ID: ursinus4
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/24/2007
Type: automatic
Task: main
Run description: log norm, no ocr, query pruning

ursinus5¶

Run ID: ursinus5
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/24/2007
Type: automatic
Task: main
Run description: cos norm, no ocr, query pruning

ursinus6¶

Run ID: ursinus6
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/24/2007
Type: automatic
Task: main
Run description: cube rt norm, ocr on, query pruning

ursinus7¶

Run ID: ursinus7
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/25/2007
Type: manual
Task: main
Run description: cube rt norm, ocr off, query pruning, manual (played with boolean query)

ursinus8¶

Run ID: ursinus8
Participant: ursinus-college.kontostathis
Track: Legal
Year: 2007
Submission: 7/25/2007
Type: manual
Task: main
Run description: cos norm, ocr off, query pruning, manual (played with boolean query)

Washington¶

Run ID: Washington
Participant: uwash.efthimiadis
Track: Legal
Year: 2007
Submission: 8/15/2007
Task: interactive

wat1fuse¶

Run ID: wat1fuse
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: Fusion of 6 runs.

wat2nobool¶

Run ID: wat2nobool
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: Same as wat1fuse, but with boolean results excluded.

wat3desc¶

Run ID: wat3desc
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: Description only field.

wat4feed¶

Run ID: wat4feed
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: Pseudo-relevance feedback (only feedback terms - character 4grams)

wat5nofeed¶

Run ID: wat5nofeed
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: Fusion, no pseudo-relevance feedback.

wat6qap¶

Run ID: wat6qap
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: term proximity ranking

wat7bool¶

Run ID: wat7bool
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: ranked boolean

wat8gram¶

Run ID: wat8gram
Participant: uwaterloo.clarke
Track: Legal
Year: 2007
Submission: 8/5/2007
Type: manual
Task: main
Run description: character 4-grams