UniNEmulti3 UniNEmulti4 UniNEmulti5 UniNEpt1 UniNEpt2 UniNEpt3 UniNEru1 UniNEru2 UniNEru3
SICS SICS SICS CEA/LIC2M CEA/LIC2M CEA/LIC2M Universidad de Evora Universidad de Evora ILTG ILTG ILTG ILTG ILTG ILTG ILTG ILTG ILTG U Jaen/SINAI U Jaen/SINAI U Jaen/SINAI U Jaen/SINAI U Jaen/SINAI CLIPS CLIPS CLIPS CLIPS CLIPS CLIPS CLIPS CLIPS CLIPS USA USA USA USA USA USA USA USA USA USA Sweden Sweden Sweden France France France Portugal Portugal Spain Spain Spain Spain Spain France France France France France France France France France SICSsvfrQF ltrcmonofr1 ltrcmonofr2 ltrcmonofr3 ltrcmonofr4 UJAMLRL UJAMLRL2P UJAMLRL3P UJAMLRLRP UJAMLRSV2 CLIPS04ENFI1 CLIPS04ENRU1 CLIPS04ENRU2 CLIPS04FRFI1 CLIPS04FRRU1 CLIPS04FRRU2 CLIPS04MNFI1 CLIPS04MNFR1 CLIPS04MNRU1 humFI04t humFI04td humFI04tde humFR04t humFR04td humFR04tde humPT04t Bi-FR SV
EN
EN Multi Multi Bi-EN FR Bi-EN FR
EN EN EN EN EN EN
FR Multi Multi Multi Multi Multi Bi-FI Bi-FI TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD T T TD TD TD TD TD T TD TD T TD TD T TDN Auto Y TDN Auto Y Hummingbird Hummingbird Hummingbird Canada Canada Canada Spain Spain Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Netherlands Germany Germany Germany Germany Germany Germany Germany Germany Germany Germany Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Italy Italy Italy Italy Italy Italy Italy Italy Italy Italy Italy Italy Italy Italy Italy Italy humRU04t humRU04td humRU04tde uoviESEN uoviRU
T TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD
o
I am NA tlâe lffau tiyU rd I recP reogup lla− tifsv ec eeb R th tseAm /JaenS ceuhN BNU ubC
Y iln Min m r o f r e p p o
T 0 . 1 1.0 0.8 0.6 0.4 0.2 0.0
0.0 0.0 0.0 0.3470UAmsterdam 0.0644UOviedo 0.4
0.6
Recall 0.2 0.8 1.0 0.2 0.4
0.6 0.8 1.0 n o i s i c e r 0.8 0.0 0.0 Top performing experiments by the best five groups; TD topic fields; Automatic 0.3750 Thomson Legal 0.3505 Daedalus 0.3368 NII Group 0.3339 Dublin City U 0.4
0.6
Recall 0.2 0.8 1.0 Top performing experiments by the best five groups; TD topic fields; Automatic 0.4704 U Neuchâtel 0.4537 JHU/APL 0.4366 U Amsterdam 0.3243 U Alicante 0.2 0.4
0.6 0.8 1.0 Top performing experiments by the best five groups; TD topic fields; Automatic 0.3357 U Alicante 0.3291 UC Berkeley 0.3210 Dublin City U 0.3007 U Neuchâtel 0.1407 JHU/APL 0.0 0.0 0.2 0.4
0.6 0.8 1.0 1.0 0.8 0.8
Monolingual EN; Recall−Precision Graph 0.5167SUNYBuffalo 0.0 0.0 0.2 0.4
0.6 1.0 0.0 0.0 0.2 0.4
0.6 Recall 0.8 1.0 n o i s i c e r n o i s i c e r 0.8
Monolingual FR; Recall−Precision Graph Top performing experiments by the best five groups; TD topic fields; Automatic 0.4926 Hummingbird 0.4849 U Neuchâtel 0.4673 Daedalus 0.4629 SUNY Buffalo 0.0 0.0 0.2 0.4
0.6 0.8 1.0 0.0 0.0 0.2 0.4
0.6 Recall 0.8 1.0 Top performing experiments by the best five groups; TD topic fields; Automatic 0.5188 U Neuchâtel 0.4750 Hummingbird 0.4691 JHU/APL 0.4469 Thomson Legal 0.4326 U Amsterdam Top performing experiments by the best five groups; TD topic fields; Automatic 0.4809 U Alicante 0.4431 Hummingbird 0.4412 U Amsterdam 0.4204 UC Berkeley 0.3790 Dublin City U 0.0 0.0 0.2 0.4
0.6 0.8 1.0 0.8 0.6 n o i s i c e r P 0.4 0.4177UCBerkeley 0.2482FUHagen 0.3855IRIT 0.0369FUHagen 0.0 0.0 0.2 0.4
0.6 0.8 1.0 8 .
0 s s e 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 o, cis ): t t s u s e A i i
t r h, sta que s iEngl Overall (for50 : ) s Uni ison uerie City preci udalq Avg.prec.(non- forallre.ldoc 8 .
0 s s e 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 o, cis ): t t s u s e A i i
t r h, sta que s iEngl Overall (for50 : ) s Uni ison uerie City preci udalq Avg.prec.(non- forallre.ldoc 8 .
0 s s e 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 o, cis ): t t s u s e A i i
t r h, sta que s iEngl Overall (for50 l a u g n i l i t l u M , y t i s r e v : ) s Uni ison uerie City preci udalq Avg.prec.(non- forallre.ldoc P 8 .
0 s s e 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 o, cis ): t t s u s e A i i
t r h, sta que s iEngl Overall (for50 l a u g n i l i t l u M , y t i s r e v : ) s Uni ison uerie City preci udalq Avg.prec.(non- forallre.ldoc P 8 .
0 s s e 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 o, cis ): t t s u s e A i i
t r h, sta que s iEngl Overall (for50 l a u g n i l i t l u M , y t i s r e v : ) s Uni ison uerie City preci udalq Avg.prec.(non- forallre.ldoc P R p 8 . 0 n− reP IR ll−
a n ec u R ) noi si cer p egar eva( ecner effi D .50+ .00 .50− 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 ics ): t s s e i i t r sta que DTto, arOvell r(fo50 u A l a u g n i l i t : ) s Mul ison uerie nte, preci udalq Avg.prec.(non- forallre.ldoc P 8 . 0 p x se e lu n a − V u n o i M is − ec n r
P IR ll−
a n ce u R
R ) noi si cer p egar eva( ecner effi D .50+ .00 .05− 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 ics ): t s s e i i t r sta que DTto, arOvell r(fo50 u A l a u g n i l i t : ) s Mul ison uerie nte, preci udalq Avg.prec.(non- forallre.ldoc P 8 . 0 IR ll−
a n ce u R R p x se e lu P a − V u n o i M is − ec n rP 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 4 2 0 3 2 0 2 2 0 1 2 1 0 2 ics ): t s s e i i t r sta que DTto, arOvell r(fo50 u A l a u g n i l i t : ) s Mul ison uerie nte, preci udalq Avg.prec.(non- forallre.ldoc P 8 .
0 n on u is
r Rap m o C 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 ics ): t s s e i i t r sta que TDo, raOvell r(fo25 t u A o t IRGTn, isiconpre querieudal Avg.prec.(non- forallre.ldoc P 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 112: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 .
0 waV d n m o C 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 ics ): t s s e i i t r sta que TDo, raOvell r(fo25 t u A o t IRGTn, isiconpre querieudal Avg.prec.(non- forallre.ldoc P 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 112: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 .
0 waV d n m o C 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r sta que TDo, raOvell r(fo25 t u A o t IRGTn, isiconpre querieudal Avg.prec.(non- forallre.ldoc P 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 112: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 .
0 u ce 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r sta que o t n a m : ) s Avg.prec.(non- forallre.ldoc P 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 112: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R R 8 .
0 u ce 4 se 0 lu e a d V t n e io d is n ec re rP
− n lla ) noi si cer p egar eva( ecner effi D .50+ .00 .05− 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r sta que o t h s i l : ) s Avg.prec.(non- forallre.ldoc rRte ved)ie 867 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 112: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 1 L V 8 .
0 ) noi si cer p egar eva( ecner effi D .05+ .00 .50− u m Ro
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r TD sta que ,otAu arOvell r(fo24 o t : ) s eley,GI precisoin udalqureie Avg.prec.(non- forallre.ldoc rRte ved)ie 396 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 2 L V 8 .
0 ) noi si cer p egar eva( ecner effi D .50+ .00 .05− u m Ro
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r TD sta que ,otAu arOvell r(fo24 o t : ) s eley,GI precisoin udalqureie i u Avg.prec.(non- forallre.ldoc rRte ved)ie 717 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 3 L V 8 .
0 ) noi si cer p egar eva( ecner effi D .50+ .00 .50− u m Ro
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r TD sta que ,otAu arOvell r(fo24 o t : ) s eley,GI precisoin udalqureie U i u Avg.prec.(non- forallre.ldoc rRte ved)ie 659 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 4 L V 8 .
0 u m Ro
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r TD sta que ,otAu arOvell r(fo24 o t : ) s eley,GI precisoin udalqureie U i u Avg.prec.(non- forallre.ldoc rRte ved)ie 338 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 5 L V 8 .
0 u m Ro
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i t r TD sta que ,otAu arOvell r(fo24 o t : ) s eley,GI precisoin udalqureie U s t n e m u c o i u Avg.prec.(non- forallre.ldoc rRte ved)ie 927 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 8 .
0 L aV Kno u m
RoC 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 ics ): t s s e i i TD tats quer , o t h s i l g n E eley,GI precisoin udalqureie U s t n e m u c o i u Avg.prec.(non- forallre.ldoc rRte ved)ie 958 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 8 .
0 L aV Kno u m
RoC 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 h s i l g n E o t h s i l g n E eley,GI precisoin udalqureie U s t n e m u c o i u Avg.prec.(non- forallre.ldoc rRte ved)ie 676 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 . 0 c i p o t y 1 b e an
i Hd
e Um F to n n u so
i Rra p m o
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que o t : ) s IRGTn, isiconpre querieudal F s t n e m u c o i u Avg.prec.(non- forallre.ldoc rRte ved)ie 852 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 . 0 c i p o t y 2 b e an
i Hd
e Um F to n n u so
i Rra p m o
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que o t : ) s IRGTn, isiconpre querieudal s t n e m u c o berofd queires eved: ant: et: i u Avg.prec.(non- forallre.ldoc rRte ved)ie 355 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 . 0 c i p o t y 3 b e an
i Hd
e Um F to n n u so
i Rra p m o
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que o t n a m r e G : ) s IRGTn, isiconpre querieudal ge ge vi a a i HU reAv dn(i s t n e m u c o berofd queires eved: ant: et: i u Avg.prec.(non- forallre.ldoc rRte ved)ie 854 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 . 0 c i p o t y 4 b e an
i Hd
e Um F to n n u so
i Rra p m o
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que DTto, arOvell r(fo24 u A o t n a m r e G : ) s IRGTn, isiconpre querieudal ge ge vi a a i HU reAv dn(i s t n e m u c o berofd queires eved: ant: et: i u Avg.prec.(non- forallre.ldoc rRte ved)ie 355 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 .
0 c i p o t y 5 b e an
i Hd
e Um F to n n u so
i Rra p m o
C 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que DTto, arOvell r(fo24 u A o t n a m r e G : ) s IRGTn, isiconpre querieudal ge ge vi a a i HU reAv dn(i s t n e m u c o berofd queires eved: ant: et: i u Avg.prec.(non- forallre.ldoc rRte ved)ie 747 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 .
0 e
R 1 n n io
R lac
s Rir a p m o
C un ton l l a c e
R 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que o t h s i : ) s EnglIRT isiconpre querieudal s t n e m u c o berofd queires eved: ant: et: i u Avg.prec.(non- forallre.ldoc rRte ved)ie 497 erci mu_r Exa 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: R 8 . 0 2T onV c i p o t y Db
n Tia
r Ra p m o
C ru tom un ison
n2 ed 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que o t h s i : ) s EnglIRT isiconpre querieudal s t n e m u c o berofd queires eved: ant: et: i u Avg.prec.(non- forallre.ldoc rRte ved)ie 683 erci mu_r Exa 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: s 8 . 0 R eR fc leu
a
DV n so u ir
a Rp m o
C Dan Tid 3 e n m fc by 0 . 1 + 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 sta que o t h s i : ) s EnglIRT isiconpre querieudal G ge vi , a i RIT rAve d(in s t n e m u c o berofd queires eved: ant: et: i u Avg.prec.(non- forallre.ldoc 750 292 111 938 389 621 648 825 450 rRte ved)ie 781 erci mu_r Exa 101: 102: 103: 104: 105: 106: 107: 108: 109: 110: 111: 113: 114: 115: 116: 117: 118: 119: 120: 121: 122: 123: 124: 125: 8 . 0 R R un reP 4T iisco
s Neu
l Wa
V
Dn c i p o Nt y b Wn
a Di
d Te 4 m n to ru on
s n ir u ap Rm o
C 0 . 1 + 6 . 0 4 .
0 8 .
0 0 . 1 2 .
0 4 . 0 2 . 0 0 2 1 5 1 1 0 1 1 5 0 1 1 0 1 r e b m u n c i p o T ics ): t s s e i i t r sta que o t h s i : ) s EnglIRT isiconpre querieudal G ge vi , a i RIT rAve d(in s t n e m u c o berofd queires eved: ant: et: ) d e t a l 0 po s 64 r t 3 etn nem .0 i u Avg.prec.(non- forallre.ldoc 917 333 083 833 333 587 633 839 461 rRte ved)ie 366 erci mu_r Exa ru reP T on n4 iisc c i p o t Ny b n Wa
i Td 4 e n m u to r n 8 .
0 0 . 1 2 .
0 4 . 0 0 2 1 5 1 1 sta que h s i EnglIRT isiconpre querieudal G ge vi , a i s t n e m u c o
21907 1235 1014 berofd queires eved: ant: et: i u