group: ITC-irst, Italy
task: Monolingual Italian Question Answering
subtask: exact answers
Statistics over 200 questions in the file irstex031mi.txt
run irstex031mi
This file is composed of 590 answers:
132 Right, 439 Wrong, 6 Unsupported and 13 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 103 (51.5%)
Number of questions where at least one correct response was returned: 97 (48.5%)
150 Mean reciprocal rank : 0.4225
103 Statistics (lenient):
99
100 Number of questions where no correct responses were returned: 99 (49.5%)
79 Number of questions where at least one correct response was returned: 101 (50.5%)
75
Mean reciprocal rank : 0.4425
50 Number of questions for which `NIL' was returned as a response : 4
Number of questions for which `NIL' was correctly returned : 2
13 13
9 9
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: ITC-irst, Italy
task: Monolingual Italian Question Answering
subtask: 50 bytes long answers
run irstst032mi Statistics over 200 questions in the file irstst032mi.txt
This file is composed of 582 answers:
200 153 Right, 410 Wrong, 7 Unsupported and 12 ineXact.
Statistics (strict):
Number of questions where no correct responses were returned: 101 (50.5%)
150 Number of questions where at least one correct response was returned: 99 (49.5%)
Mean reciprocal rank : 0.449166666666667
101 96
100 Statistics (lenient):
83 87
Number of questions where no correct responses were returned: 96 (48%)
Number of questions where at least one correct response was returned: 104 (52%)
Mean reciprocal rank : 0.471666666666667
50
Number of questions for which `NIL' was returned as a response : 5
Number of questions for which `NIL' was correctly returned : 2
9 10 7 7
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Alicante, Spain
task: Monolingual Spanish Question Answering
subtask: exact answers
Statistics over 200 questions in the file alicex031ms.txt
run alicex031ms
This file is composed of 547 answers:
91 Right, 437 Wrong, 9 Unsupported and 10 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 120 (60%)
Number of questions where at least one correct response was returned: 80 (40%)
150 Mean reciprocal rank : 0.3075
120 113
Statistics (lenient):
100 Number of questions where no correct responses were returned: 113 (56.5%)
Number of questions where at least one correct response was returned: 87 (43.5%)
Mean reciprocal rank : 0.320833333333333
49 49 Number of questions for which `NIL' was returned as a response : 21
50
Number of questions for which `NIL' was correctly returned : 5
23
13 15 18
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Alicante, Spain
task: Monolingual Spanish Question Answering
subtask: exact answers
run alicex032ms Statistics over 200 questions in the file alicex032ms.txt
This file is composed of 546 answers:
200 79 Right, 448 Wrong, 9 Unsupported and 10 ineXact.
Statistics (strict):
Number of questions where no correct responses were returned: 130 (65%)
150 Number of questions where at least one correct response was returned: 70 (35%)
130 Mean reciprocal rank : 0.296666666666667
123
100 Statistics (lenient):
Number of questions where no correct responses were returned: 123 (61.5%)
Number of questions where at least one correct response was returned: 77 (38.5%)
Mean reciprocal rank : 0.3175
53
51
50
Number of questions for which `NIL' was returned as a response : 21
Number of questions for which `NIL' was correctly returned : 5
12 15
7 9
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Amsterdam, The Netherlands
task: Monolingual Dutch Question Answering
subtask: exact answers
run uamsex031md
Statistics over 200 questions in file uamsex031md.txt.
200 This file is composed of 600 answers:
84 Right, 492 Wrong, 6 Unsupported and 18 ineXact.
Statistics (strict):
150 Number of questions where no correct responses were returned: 122 (61%)
Number of questions where at least one correct response was returned: 78 (39%)
122 118 Mean reciprocal rank : 0.298333333333333
100
Statistics (lenient):
Number of questions where no correct responses were returned: 118 (59%)
Number of questions where at least one correct response was returned: 82 (41%)
50
47 Mean reciprocal rank : 0.3175
50
17 Number of questions for which `NIL' was returned as a response : 200
14 17 15
Number of questions for which `NIL' was correctly returned : 17
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Amsterdam, The Netherlands
task: Monolingual Dutch Question Answering
subtask: exact answers
Statistics over 200 questions in file uamsex032md.txt.
run uamsex032md
This file is composed of 600 answers:
85 Right, 488 Wrong, 11 Unsupported and 16 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 118 (59%)
Number of questions where at least one correct response was returned: 82 (41%)
150 Mean reciprocal rank : 0.305833333333333
118
111
Statistics (lenient):
100 Number of questions where no correct responses were returned: 111 (55.5%)
Number of questions where at least one correct response was returned: 89 (44.5%)
Mean reciprocal rank : 0.335833333333333
50
50 46 Number of questions for which `NIL' was returned as a response : 200
25
Number of questions for which `NIL' was correctly returned : 17
19
17 14
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: DFKI, Germany
task: Bilingual German Question Answering
subtask: 50 bytes long answers
run dfkist031bg Statistics over 200 questions in the file dfkist031bg.txt
This file is composed of 564 answers:
200 41 Right, 522 Wrong, 1 Unsupported and 0 ineXact.
171 170 Statistics (strict):
Number of questions where no correct responses were returned: 171 (85.5%)
150 Number of questions where at least one correct response was returned: 29 (14.5%)
Mean reciprocal rank : 0.0983333333333333
100 Statistics (lenient):
Number of questions where no correct responses were returned: 170 (85%)
Number of questions where at least one correct response was returned: 30 (15%)
Mean reciprocal rank : 0.103333333333333
50
Number of questions for which `NIL' was returned as a response : 18
Number of questions for which `NIL' was correctly returned : 0
13 14
8 8 8 8
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: ITC-irst, Italy
task: Bilingual Italian Question Answering
subtask: exact answers
Statistics over 200 questions in the file irstex031bi.txt
run irstex031bi
This file is composed of 492 answers:
94 Right, 378 Wrong, 6 Unsupported and 14 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 123 (61.5%)
Number of questions where at least one correct response was returned: 77 (38.5%)
150 Mean reciprocal rank : 0.3225
123 119
Statistics (lenient):
100 Number of questions where no correct responses were returned: 119 (59.5%)
Number of questions where at least one correct response was returned: 81 (40.5%)
Mean reciprocal rank : 0.334166666666667
55 56
50 Number of questions for which `NIL' was returned as a response : 49
Number of questions for which `NIL' was correctly returned : 6
13 15
9 10
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: ITC-irst, Italy
task: Bilingual Italian Question Answering
subtask: exact answers
Statistics over 200 questions in the file irstex032bi.txt
run irstex032bi
This file is composed of 526 answers:
113 Right, 402 Wrong, 2 Unsupported and 9 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 110 (55%)
Number of questions where at least one correct response was returned: 90 (45%)
150 Mean reciprocal rank : 0.393333333333333
110 108 Statistics (lenient):
100 Number of questions where no correct responses were returned: 108 (54%)
Number of questions where at least one correct response was returned: 92 (46%)
70 71 Mean reciprocal rank : 0.400833333333333
50 Number of questions for which `NIL' was returned as a response : 28
Number of questions for which `NIL' was correctly returned : 5
12 13
8 8
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Limerick, Ireland
task: Bilingual French Question Answering
subtask: exact answers
Statistics over 200 questions in the file dltgex031bf.txt
run dltgex031bf This file is composed of 200 answers:
23 Right, 175 Wrong, 1 Unsupported and 1 ineXact.
200 Statistics (strict):
177 176 Number of questions where no correct responses were returned: 177 (88.5%)
Number of questions where at least one correct response was returned: 23 (11.5%)
Mean reciprocal rank : 0.115
150
Statistics (lenient):
Number of questions where no correct responses were returned: 176 (88%)
100 Number of questions where at least one correct response was returned: 24 (12%)
Mean reciprocal rank : 0.12
Number of questions for which `NIL' was returned as a response : 119
50 Number of questions for which `NIL' was correctly returned : 10
23 24
0 0 0 0
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Limerick, Ireland
task: Bilingual French Question Answering
subtask: exact answers
Statistics over 200 questions in the file dltgex032bf.txt
run dltgex032bf
This file is composed of 200 answers:
22 Right, 175 Wrong, 1 Unsupported and 2 ineXact.
200
178 177 Statistics (strict):
Number of questions where no correct responses were returned: 178 (89%)
Number of questions where at least one correct response was returned: 22 (11%)
150 Mean reciprocal rank : 0.11
Statistics (lenient):
100 Number of questions where no correct responses were returned: 177 (88.5%)
Number of questions where at least one correct response was returned: 23 (11.5%)
Mean reciprocal rank : 0.115
50 Number of questions for which `NIL' was returned as a response : 119
23
Number of questions for which `NIL' was correctly returned : 10
22
0 0 0 0
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: ISI, University of Southern California, USA
task: Bilingual Spanish Question Answering
subtask: exact answers
Statistics over 200 questions in the file isixex031bs.txt
run isixex031bs
This file is composed of 586 answers:
87 Right, 476 Wrong, 15 Unsupported and 8 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 131 (65.5%)
Number of questions where at least one correct response was returned: 69 (34.5%)
Mean reciprocal rank : 0.3025
150
131
123
Statistics (lenient):
Number of questions where no correct responses were returned: 123 (61.5%)
100 Number of questions where at least one correct response was returned: 77 (38.5%)
Mean reciprocal rank : 0.32833333333333
56
53 Number of questions for which `NIL' was returned as a response : 4
50 Number of questions for which `NIL' was correctly returned : 0
13 16
3 5
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: ISI, University of Southern California, USA
task: Bilingual Spanish Question Answering
subtask: exact answers
Statistics over 200 questions in the file isixex032bs.txt
run isixex032bs
This file is composed of 586 answers:
85 Right, 478 Wrong, 13 Unsupported and 10 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 132 (66%)
Number of questions where at least one correct response was returned: 68 (34%)
150 Mean reciprocal rank : 0.27166666666667
132
122
Statistics (lenient):
100 Number of questions where no correct responses were returned: 122 (61%)
Number of questions where at least one correct response was returned: 78 (39%)
Mean reciprocal rank : 0.3075
48 Number of questions for which `NIL' was returned as a response : 4
50 43
Number of questions for which `NIL' was correctly returned : 0
18 21
7 9
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: Carnegie Mellon University, USA
task: Bilingual French Question Answering
subtask: exact answers
Statistics over 200 questions in the file lumoex031bf.txt
run lumoex031bf
This file is composed of 414 answers:
41 Right, 366 Wrong, 6 Unsupported and 1 ineXact.
200
Statistics (strict):
162
Number of questions where no correct responses were returned: 162 (81%)
158 Number of questions where at least one correct response was returned: 38 (19%)
150 Mean reciprocal rank : 0.153333333333333
Statistics (lenient):
100 Number of questions where no correct responses were returned: 158 (79%)
Number of questions where at least one correct response was returned: 42 (21%)
Mean reciprocal rank : 0.170833333333333
50 Number of questions for which `NIL' was returned as a response : 92
28 Number of questions for which `NIL' was correctly returned : 8
25
8 9
5 5
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: Carnegie Mellon University, USA
task: Bilingual French Question Answering
subtask: exact answers
Statistics over 200 questions in the file lumoex032bf.txt
run lumoex032bf
This file is composed of 416 answers:
34 Right, 375 Wrong, 6 Unsupported and 1 ineXact.
200
Statistics (strict):
169 Number of questions where no correct responses were returned: 169 (84.5%)
165
Number of questions where at least one correct response was returned: 31 (15.5%)
150 Mean reciprocal rank : 0.13166666666666667
Statistics (lenient):
100 Number of questions where no correct responses were returned: 165 (82.5%)
Number of questions where at least one correct response was returned: 35 (17.5%)
Mean reciprocal rank : 0.14916666666666667
50 Number of questions for which `NIL' was returned as a response : 91
Number of questions for which `NIL' was correctly returned : 7
25
22
8 9
1 1
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Montreal, Canada
task: Bilingual French Question Answering
subtask: 50 bytes long answers
Statistics over 200 questions in the file udemst031bf.txt
run udemst031bf This file is composed of 599 answers:
65 Right, 532 Wrong, 2 Unsupported and 0 ineXact.
200 Statistics (strict):
Number of questions where no correct responses were returned: 144 (72%)
Number of questions where at least one correct response was returned: 56 (28%)
Mean reciprocal rank : 0.2133333333333333
150 144 142
Statistics (lenient):
Number of questions where no correct responses were returned: 142 (71%)
100 Number of questions where at least one correct response was returned: 58 (29%)
Mean reciprocal rank : 0.2208333333333333
Number of questions for which `NIL' was returned as a response : 4
Number of questions for which `NIL' was correctly returned : 1
50
32 33
16 17
8 8
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank
group: University of Montreal, Canada
task: Bilingual French Question Answering
subtask: exact answers
Statistics over 200 questions in the file udemex032bf.txt
run udemex032bf
This file is composed of 599 answers:
39 Right, 539 Wrong, 5 Unsupported and 16 ineXact.
200
Statistics (strict):
Number of questions where no correct responses were returned: 162 (81%)
162 Number of questions where at least one correct response was returned: 38 (19%)
158
Mean reciprocal rank : 0.14
150
Statistics (lenient):
Number of questions where no correct responses were returned: 158 (79%)
100 Number of questions where at least one correct response was returned: 42 (21%)
Mean reciprocal rank : 0.16083333333333333
Number of questions for which `NIL' was returned as a response : 3
50 Number of questions for which `NIL' was correctly returned : 1
23
20
12 13
6 6
0
1st 2nd 3rd not found
strict lenient
Number of questions for which the first correct response was at a given rank