$3URSRVDOIRUWKH(YDOXDWLRQRI$GDSWLYH3HUVRQDOL]HG ,QIRUPDWLRQ5HWULHYDO  6pDPXV/DZOHVV$OH[DQGHU2¶&RQQRU&DWKHULQH0XOZD &HQWUHIRU1H[W*HQHUDWLRQ/RFDOL]DWLRQ 6FKRRORI&RPSXWHU6FLHQFHDQG6WDWLVWLFV 7ULQLW\&ROOHJH'XEOLQ 'XEOLQ,UHODQG VHDPXVODZOHVV#VFFVWFGLHDOH[RFRQQRU#VFFVWFGLHPXOZDF#VFFVWFGLH   $%675$&7 $V SHUVRQDOLVHG LQIRUPDWLRQ UHWULHYDO DQG UHVSRQVH FRPSRVLWLRQ 3HUVRQDOLVDWLRQLQ,QIRUPDWLRQ5HWULHYDOLVDFKLHYHGXVLQJDUDQJH EHFRPHPRUHZLGHO\XVHGWUDGLWLRQDODSSURDFKHVWR,5HYDOXDWLRQ RIFRQWH[WXDOLQIRUPDWLRQVXFKDVLQIRUPDWLRQDERXWWKHXVHUWKH EHFRPH OHVV HIIHFWLYH DQG DSSOLFDEOH ,W LV WKH FRQWHQWLRQ RI WKLV WDVNEHLQJFRQGXFWHGDQGWKHGHYLFHEHLQJXVHG7KLVLQIRUPDWLRQ SDSHUWKDWFRQWH[WXDODQGVSHFLILFDOO\SHUVRQDOLVHGDSSURDFKHVWR LV XVHG WR GHYLVH WKH PRVW VXLWDEOH UHVSRQVH IRU WKH LQGLYLGXDO¶V ,5FRXOGEHQHILWIURPWKHH[SHULHQFHRIWKH$GDSWLYH+\SHUPHGLD QHHG $V VXFK SHUVRQDOLVHG ,QIRUPDWLRQ 5HWULHYDO DQG UHVSRQVH $+ FRPPXQLW\ FRPSRVLWLRQ DSSURDFKHV EHFRPH PRUH ZLGHO\ XVHG WUDGLWLRQDO 5HFHQWO\UHVHDUFKKDVEHHQXQGHUWDNHQH[SORULQJKRZWRHQKDQFH HYDOXDWLRQ PHDVXUHV EHFRPH OHVV HIIHFWLYH DQG DSSOLFDEOH 7KLV DQG FRPELQH NH\ DVSHFWV RI $+ UHVHDUFK ZLWK ,5 UHVHDUFK WR SDSHU SURSRVHV WKDW FRQWH[WXDO DQG VSHFLILFDOO\ SHUVRQDOLVHG SURYLGHDGYDQFHGDQQRWDWLRQVOLFLQJUHWULHYDODQGFRPSRVLWLRQRI DSSURDFKHV WR ,QIRUPDWLRQ 5HWULHYDO FRXOG EHQHILW IURP WKH PXOWLOLQJXDO GLJLWDO FRQWHQW GUDZQ IURP FRUSRUDWH GRFXPHQWV H[SHULHQFH RI WKH $GDSWLYH +\SHUPHGLD FRPPXQLW\ 7KH SDSHU UHSRVLWRULHV DV ZHOO DV RSHQ FRUSXV VRXUFHV >@>@ :H FDOO VXFK GHWDLOVWKHDSSURDFKHVWRHYDOXDWLRQFRPPRQO\XVHGE\WKH,5DQG V\VWHPV ZKLFK FRPELQH $+ DQG ,5 DSSURDFKHV WR GHOLYHU $+ FRPPXQLWLHV DQG SURSRVHV D PHDQV RI FRPELQLQJ DQG SHUVRQDOLVHG LQIRUPDWLRQ VHHNLQJ DQG DFFHVV $GDSWLYH HQKDQFLQJWKHVHGLVSDUDWHDSSURDFKHV LQDXQLILHGIUDPHZRUNIRU ,QIRUPDWLRQ5HWULHYDO6\VWHPV $,56  WKHHYDOXDWLRQRISHUVRQDOLVHG,5V\VWHPV $+ V\VWHPV KDYH WUDGLWLRQDOO\ IXQFWLRQHG XVLQJ ORZ YROXPHV RI &DWHJRULHVDQG6XEMHFW'HVFULSWRUV FRQWHQW ZLWK DVVRFLDWHG ULFK PHWDGDWD GHVFULSWLRQV LQ FRQWUDVW WR ZHEEDVHG ,5 V\VWHPV ZKLFK WHQG WR IXQFWLRQ XVLQJ VWDWLVWLFDO +>,QIRUPDWLRQ6WRUDJHDQG5HWULHYDO@,QIRUPDWLRQ6HDUFK PHWKRGVRQYHU\ODUJHVFDOHFROOHFWLRQVRIXQDQQRWDWHGFRQWHQW,Q DQG5HWULHYDO+>,QIRUPDWLRQ,QWHUIDFHVDQG3UHVHQWDWLRQ@ RUGHU WR IDFLOLWDWH HIIHFWLYH SHUVRQDOLVHG LQIRUPDWLRQ UHWULHYDO DW 0XOWLPHGLD,QIRUPDWLRQ6\VWHPV+>,QIRUPDWLRQ,QWHUIDFHV WKHVFDOHUHTXLUHGE\ZHEEDVHG,5V\VWHPVDK\EULGDSSURDFKWR DQG3UHVHQWDWLRQ@+\SHUWH[W+\SHUPHGLD ERWK WKH LPSOHPHQWDWLRQ DQG HYDOXDWLRQ RI WKHVH V\VWHPV LV *HQHUDO7HUPV QHFHVVDU\ 7KLV SDSHU SURSRVHV DQ DSSURDFK WR WKH HYDOXDWLRQ RI ([SHULPHQWDWLRQ0HDVXUHPHQW3HUIRUPDQFH $,56+DYLQJLQWURGXFHGWKHGLVFLSOLQHVRI,5DQG$+WKHSDSHU GHWDLOV WKH DSSURDFKHV WR HYDOXDWLRQ FRPPRQO\ XVHG E\ WKHVH .H\ZRUGV FRPPXQLWLHV7KHSDSHUWKHQFRQWLQXHVE\ SURSRVLQJ D PHDQV RI ,QIRUPDWLRQ 5HWULHYDO 3HUVRQDOLVDWLRQ $GDSWLYH +\SHUPHGLD FRPELQLQJDQGHQKDQFLQJWKHVHGLVSDUDWHDSSURDFKHVWRSURGXFHD (YDOXDWLRQ IUDPHZRUN IRU WKH HYDOXDWLRQ RI DGDSWLYH SHUVRQDOL]HG LQIRUPDWLRQUHWULHYDOV\VWHPV  ,1752'8&7,21 &RQWH[WXDO LQIRUPDWLRQ LV LQFUHDVLQJO\ EHLQJ XVHG WR IDFLOLWDWH  (9$/8$7,21$3352$&+(6 SHUVRQDOLVDWLRQ LQ ,QIRUPDWLRQ 5HWULHYDO ,5  7KH SHUVRQDOLVHG  ,QIRUPDWLRQ5HWULHYDO LGHQWLILFDWLRQUHWULHYDODQGSUHVHQWDWLRQ RIUHVRXUFHVFDQ SURYLGH 7KH &UDQILHOG PRGHO >@ UHPDLQV WKH GRPLQDQW DSSURDFK WR WKH WKH XVHU ZLWK D WDLORUHG LQIRUPDWLRQ VHHNLQJ H[SHULHQFH 6XFK HYDOXDWLRQ RI ,5 V\VWHPV 7KLV HYDOXDWLRQ PRGHO XVHV WHVW WDLORUHG H[SHULHQFHV FDQ SURGXFH D PRUH LQIRUPDWLYH UHVSRQVH FROOHFWLRQVWRDVVHVVDQGFRPSDUHWKHSHUIRUPDQFHRI,5V\VWHPV WKDQDWUDGLWLRQDOUDQNHGOLVWDSSURDFK 7\SLFDO WHVW FROOHFWLRQV LQFOXGH D GRFXPHQW FRUSXV D VHW OLVW RI VHDUFK TXHULHV DQG PDQXDOO\ DVVLJQHG UHOHYDQFH DVVHVVPHQWV IRU  HDFKTXHU\6XFKFROOHFWLRQVDUHXWLOLVHGLQ,5 FRPPXQLWLHVVXFK 3HUPLVVLRQ WR PDNH GLJLWDO RU KDUG FRSLHV RI DOO RU SDUW RI WKLV ZRUN IRU DV WKH 7H[W 5(WULHYDO &RQIHUHQFH 75(&  WKH &URVV /DQJXDJH SHUVRQDORUFODVVURRPXVHLVJUDQWHGZLWKRXWIHHSURYLGHGWKDWFRSLHVDUH QRWPDGHRUGLVWULEXWHGIRUSURILWRUFRPPHUFLDODGYDQWDJHDQGWKDWFRSLHV EHDUWKLVQRWLFHDQGWKHIXOOFLWDWLRQRQWKHILUVWSDJH7RFRS\RWKHUZLVH RU UHSXEOLVK WR SRVW RQ VHUYHUV RU WR UHGLVWULEXWH WR OLVWV UHTXLUHV SULRU Appears in the Proceedings of The 2nd International Workshop on Contex- VSHFLILFSHUPLVVLRQDQGRUDIHH tual Information &,56( ¶ :RUNVKRSAccess, RQ Seeking and Retrieval &RQWH[WXDO Evaluation ,QIRUPDWLRQ $FFHVV (CIRSE 2010), 6HHNLQJ  DQG5HWULHYDO(YDOXDWLRQ March 28, 2010, Milton Keynes, UK. WK0DUFK0LOWRQ.H\QHV8.  http://www.irit.fr/CIRSE/ &RS\ULJKW$&0« 7KH7H[W5HWULHYDO&RQIHUHQFH±KWWSWUHFQLVWJRY Copyright owned by the authors. (YDOXDWLRQ )RUXP &/()  DQG WKH 1DWLRQDO ,QVWLWXWH RI XVHG GXULQJ DQ\ LQWHUDFWLRQV ZLWK WKH XVHU LQ RUGHU WR DGDSW ,QIRUPDWLFV7HVW&ROOHFWLRQIRU,56\VWHPV 17&,5  DVSHFWVRIWKHV\VWHPVIXQFWLRQDOLW\RUSUHVHQWDWLRQVWUDWHJ\WRWKH QHHGVRI WKDWXVHU>@ $+DSSURDFKHVDUHDSSOLHG LQHGXFDWLRQDO 3UHFLVLRQDQGUHFDOO>@DUHWKHPHWULFVXSRQZKLFKWKHPDMRULW\RI K\SHUPHGLD RQOLQH LQIRUPDWLRQ V\VWHPV RQOLQH KHOS V\VWHPV WUDGLWLRQDO ,5 HYDOXDWLRQ DSSURDFKHV DUH EDVHG LQFOXGLQJ WKH LQVWLWXWLRQDO K\SHUPHGLD DQG V\VWHPV IRU PDQDJLQJ SHUVRQDOL]HG &UDQILHOG PRGHO3UHFLVLRQLVWKHIUDFWLRQRIGRFXPHQWVUHWULHYHG YLHZV LQ UHVSRQVH WR D TXHU\ WKDW DUH UHOHYDQW WR WKH XVHUV LQIRUPDWLRQ QHHGZKHQPDNLQJWKDWTXHU\5HFDOOLVWKHIUDFWLRQRIWKHWRWDOVHW  &XUUHQW(YDOXDWLRQRI$GDSWLYH6\VWHPV RI UHOHYDQW GRFXPHQWV LQ D FROOHFWLRQ ZKLFK DUH UHWXUQHG IRU D ,Q RUGHU WR SURGXFH HIIHFWLYH UHVXOWV HYDOXDWLRQ VKRXOG RFFXU JLYHQ TXHU\ $QRWKHU FRPPRQ PHDVXUH RI SHUIRUPDQFH LV YDQ WKURXJKRXW WKH HQWLUH GHVLJQ F\FOH DQG SURYLGH IHHGEDFN IRU 5LMVEHUJHQ¶V IPHDVXUH >@ ZKLFK FDOFXODWHV WKH ZHLJKWHG GHVLJQPRGLILFDWLRQ>@ KDUPRQLFPHDQRISUHFLVLRQDQGUHFDOO 8VHUFHQWHUHG DSSURDFK 8VHUFHQWHUHG HYDOXDWLRQ 8&(  FDQ VHUYH WKUHH JRDOV YHULI\LQJ WKH TXDOLW\ RI DQ $+6 GHWHFWLQJ +RZHYHU GLIIHUHQFHV EHWZHHQ WKH VWUXFWXUH RI WKH ::: DQG SUREOHPVLQWKHV\VWHP IXQFWLRQDOLW\ RULQWHUIDFHDQG VXSSRUWLQJ W\SLFDO GRFXPHQW FROOHFWLRQV DQG NH\ GLIIHUHQFHV EHWZHHQ WKH DGDSWLYLW\ GHFLVLRQV >@ 7KHVH IXQFWLRQV PDNH 8&( D YDOXDEOH XVHUV RI ZHEEDVHG ,5 V\VWHPV DQG WUDGLWLRQDO ,5 V\VWHPV WRRO IRU GHYHORSHUV RI DOO NLQGV RI V\VWHPV EHFDXVH WKH\ FDQ FKDOOHQJHPDQ\RIWKHDVVXPSWLRQVRIWKHVHPHWULFV7KHSULPDU\ MXVWLI\WKHLUHIIRUWVLPSURYHXSRQDV\VWHPRUKHOSGHYHORSHUVWR FRQFHUQ LV WKDW ERWK PHWULFV DUH KLJKO\ VXEMHFWLYH DV UHOHYDQF\ GHFLGH ZKLFK YHUVLRQ RI D V\VWHP WR UHOHDVH 7KH EHQHILWV RI WKH FDQRQO\EHDVVLJQHGEDVHGXSRQWKHXVHUVLQWHQWZKHQVXEPLWWLQJ XVHUFHQWHUHG DSSURDFK DUH VDYLQJV LQ WHUPV RI WLPH DQG FRVW DVSHFLILFTXHU\,QUHODWLRQWRWKH:::UHFDOOLVDQLQHIIHFWLYH HQVXULQJ WKH FRPSOHWHQHVV RI V\VWHP IXQFWLRQDOLW\ PLQLPL]LQJ PHDQV RI HYDOXDWLRQ DV UHFDOO UHTXLUHV WKH HQWLUH FROOHFWLRQ RI UHTXLUHGUHSDLUHIIRUWVDQGLPSURYLQJXVHUVDWLVIDFWLRQ>@ UHOHYDQW GRFXPHQWV WR EH NQRZQ LQ DGYDQFH RI HDFK TXHU\ SHUIRUPHG>@ (PSLULFDO DSSURDFK (PSLULFDO HYDOXDWLRQV DOVR NQRZQ DV FRQWUROOHG H[SHULPHQWV UHIHU WR WKH DSSUDLVDO RI D WKHRU\ E\ :KLOVW WKH SUHFLVLRQ PHDVXUH IRU D VHW RI UHVXOWV LV YDOXDEOH LW REVHUYDWLRQLQH[SHULPHQWV7KHVHHYDOXDWLRQVKHOSWRHVWLPDWHWKH GRHV QRW WDNH LQWR DFFRXQW WKH LPSRUWDQFH RI UDQNLQJ LQ PRGHUQ HIIHFWLYHQHVV HIILFLHQF\ DQG XVDELOLW\ RI D V\VWHP DQG PD\ ZHEEDVHG LQIRUPDWLRQ UHWULHYDO 7KH DYHUDJH SUHFLVLRQ LV D XQFRYHU FHUWDLQ W\SHV RI HUURUV LQ WKH V\VWHP WKDW ZRXOG UHPDLQ PHDVXUH XVHG WR HVWLPDWH WKH SHUIRUPDQFH RI DQ ,5 V\VWHP E\ RWKHUZLVHXQGLVFRYHUHG7KH NH\ WR JRRG HPSLULFDO HYDOXDWLRQLV SODFLQJHPSKDVLVRQUHOHYDQWUHVXOWVDSSHDULQJKLJKXSWKHUDQNHG WKH SURSHU GHVLJQ DQG H[HFXWLRQ RI WKH H[SHULPHQWV VR WKDW WKH OLVW$YHUDJHSUHFLVLRQLVFDOFXODWHGE\ILQGLQJWKHDYHUDJHRIWKH SDUWLFXODU IDFWRUV WR EH WHVWHG FDQ EH HDVLO\ VHSDUDWHG IURP RWKHU LQGLYLGXDOSUHFLVLRQFDOFXODWLRQVIRUHDFKUHOHYDQWUHVXOWLQWKHOLVW FRQIRXQGLQJ IDFWRUV 7KLV PHWKRG RI HYDOXDWLRQ LV GHULYHG IURP LI WKH OLVW ZHUH WUXQFDWHG GLUHFWO\ DIWHU WKDW UHVXOW 7KH 0HDQ HPSLULFDO VFLHQFH DQG FRJQLWLYH DQG H[SHULPHQWDO SV\FKRORJ\ $YHUDJH 3UHFLVLRQ 0$3 IRUDQ ,5V\VWHP LV WKHQFDOFXODWHG E\ >@ VXPPLQJ WKH DYHUDJH SUHFLVLRQ YDOXH IRU HDFK TXHU\ FRQGXFWHG /D\HUHG DSSURDFK 7KH OD\HUHG HYDOXDWLRQ DSSURDFK >@>@ DQG GLYLGLQJ E\ WKH WRWDO QXPEHU RI TXHULHV 7KLV JLYHV DQ VHSDUDWHV WKH µLQWHUDFWLRQ DVVHVVPHQW¶ DQG WKH µDGDSWDWLRQ LQGLFDWLRQRIWKHRYHUDOO SHUIRUPDQFH RIWKHV\VWHPZLWKUHODWLRQ GHFLVLRQ¶ %RWK OD\HUV VKRXOG EH HYDOXDWHG VHSDUDWHO\ LQ RUGHU WR WRSUHFLVLRQDQGUDQNLQJ HIIHFWLYHO\ LQWHUSUHW WKH HYDOXDWLRQ UHVXOWV (YDOXDWLQJ $+6 RQ D OD\HU E\ OD\HU EDVLV KDV EHHQ UHFRPPHQGHG DV D PRUH 7KH&UDQILHOGPRGHORIWHVWFROOHFWLRQVDLPVWR HQVXUHFRQWUROOHG FRPSUHKHQVLYH DSSURDFK >@>@ ,Q FRQWUDVW WR DSSURDFKHV WKDW H[SHULPHQWDWLRQ DFURVV V\VWHPV XVLQJ PHWULFV VXFK DV WKRVH IRFXV RQ WKH RYHUDOO XVHU¶V SHUIRUPDQFH DQG VDWLVIDFWLRQ >@ LQWURGXFHG DERYH VR WKDW FRQFOXVLRQV FDQ EH IRUPHG DERXW WKH OD\HUHGHYDOXDWLRQLQSDUWLFXODUDVVHVVHVWKHVXFFHVVRIDGDSWDWLRQ FRPSDUDWLYH SHUIRUPDQFH RI D YDULHW\ RI ,5 V\VWHPV +RZHYHU E\GHFRPSRVLQJLWLQWRGLIIHUHQWOD\HUVDQGHYDOXDWLQJHDFKOD\HU WKLVDSSURDFKLVOLPLWHGDVDVVXPSWLRQVPXVWEHPDGHDERXWXVHU LQGLYLGXDOO\ 7KLV KDV D QXPEHU RI DGYDQWDJHV RYHU RWKHU LQWHQWDQGEHKDYLRXU'XHWRWKHPDLQVWUHDPDFFHVVLELOLW\RIZHE DSSURDFKHV VXFK DV XVHIXO LQVLJKW LQWR WKH VXFFHVV RU IDLOXUH RI EDVHG,5V\VWHPVDVVXPSWLRQVFDQQRWEHPDGHUHJDUGLQJDXVHU¶V HDFK VHSDUDWH DGDSWDWLRQ VWDJH IDFLOLWDWLRQ RI LPSURYHPHQWV OHYHORINQRZOHGJHZLWKUHJDUGWRHLWKHUWKHV\VWHPRUWKHVXEMHFW JHQHUDOL]DWLRQ RI HYDOXDWLRQ UHVXOWV DQG UHXVH RI VXFFHVVIXO GRPDLQ7KH,5V\VWHPPXVWFRSHZLWKXVHUVDFURVVWKHVSHFWUXP SUDFWLFHV RINQRZOHGJHDQGDELOLW\>@ 8WLOLW\EDVHG DSSURDFK &XUUHQW HYDOXDWLRQ SUDFWLFHV DWWHPSW WR  $GDSWLYH+\SHUPHGLD HYDOXDWH DGDSWDWLRQ DV D ZKROH ZLWK XVHU VDWLVIDFWLRQ RU $GDSWLYH +\SHUPHGLD $+  LV DQ DOWHUQDWLYH WR WKH WUDGLWLRQDO SHUIRUPDQFHDVWKHRYHUDOOPHWULFIRUVXFFHVVEDVHGRQLGHQWLILHG ³RQHVL]HILWVDOO´ DSSURDFK WR WKH GHYHORSPHQW RI +\SHUPHGLD PHDVXUDEOH FULWHULD ,Q WKH XWLOLW\EDVHG DSSURDFK WKH HYDOXDWLRQ 6\VWHPV $GDSWLYH +\SHUPHGLD 6\VWHPV $+6  FDQ SRWHQWLDOO\ FDQEHVHHQDVDXWLOLW\IXQFWLRQ;WKDWPDSVDV\VWHPJLYHQVRPH DGDSW RQ DQ\ DWWULEXWHV RI D XVHU IRU H[DPSOH LQ $GDSWLYH XVHUFRQWH[WWRDTXDQWLWDWLYHUHSUHVHQWDWLRQRIXVHUVDWLVIDFWLRQRU (GXFDWLRQDO +\SHUPHGLD 6\VWHPV $(+6  PRGHOV FDQ EH SHUIRUPDQFH )RU H[DPSOH LI RQH FRPSDUHV DQ DGDSWLYH V\VWHP FRQVWUXFWHG ZKLFK GHVFULEH WKH JRDOV SUHIHUHQFHV NQRZOHGJH RU ZLWK LWV QRQDGDSWLYH FRXQWHUSDUW WKH YDOXH RI DGDSWDWLRQ LV WKH RWKHU DWWULEXWHV RI HDFK LQGLYLGXDO XVHU 7KLV PRGHO FDQ WKHQ EH GLIIHUHQFHLQXWLOLW\EHWZHHQWKHWZRV\VWHPV   $V GHVFULEHG DERYH WKH PDLQ DGYDQWDJH RI OD\HUHG HYDOXDWLRQ 7KH&URVV/DQJXDJH(YDOXDWLRQ)RUXPKWWSZZZFOHI PHWKRGV DUH WKDW WKH\ EUHDN WKH XWLOLW\ IXQFWLRQ LQWR VHYHUDO FDPSDLJQRUJ GLVWLQFW IXQFWLRQV )RU H[DPSOH VXSSRVH WKHUH LV D XWLOLW\ ; WKDW  7KH1DWLRQDO,QVWLWXWHRI,QIRUPDWLFV7HVW&ROOHFWLRQIRU,5 PDSVWKHLQWHUDFWLRQDVVHVVPHQWDQGWKHUHVXOWLQJXVHUPRGHOWRD 6\VWHPVKWWSUHVHDUFKQLLDFMSQWFLU UHDOQXPEHUWKDWUHSUHVHQWVLWVFRUUHFWQHVV6XSSRVHWKHUHLVDOVRD XWLOLW\IXQFWLRQ8WKDWPDSVDV\VWHPJLYHQVRPHXVHUPRGHOWR PHFKDQLVPZHFODVVLI\DFRQWHQWFRPSRVLWLRQDVWKHRXWSXWIURP DUHDOQXPEHUWKDWUHSUHVHQWVXVHUVDWLVIDFWLRQRUSHUIRUPDQFH,Q DQ$,56$FRQWHQWFRPSRVLWLRQLVDQDJJUHJDWHGVHWRIUHVRXUFHV WKLVFDVHWKHZKROHXWLOLW\IXQFWLRQFDQEHH[SUHVVHGDV; ;; RUGHUHGDFFRUGLQJWRWKHXVHU¶VQHHGVDQGSUHIHUHQFHV ,W LV FOHDU WKDW WKH ODWWHU XWLOLW\ IXQFWLRQ EHWWHU LQGLFDWHV WKH XVDELOLW\ RI DQ DGDSWLYH K\SHUPHGLD V\VWHP 8WLOLW\EDVHG ,Q RUGHU WR VXIILFLHQWO\ HYDOXDWH ERWK WKH DGDSWLYH IXQFWLRQDOLW\ HYDOXDWLRQRIDGDSWLYHV\VWHPV>@RIIHUVDSHUVSHFWLYHRIKRZWR DQG WKH UHWULHYDO SHUIRUPDQFH RI DQ $,56 D K\EULG DSSURDFK LV UHLQWHJUDWHWKHGLIIHUHQWOD\HUV QHFHVVDU\ 7KLV LQYROYHV XVHUFHQWULF DVVHVVPHQW OD\HUHG HYDOXDWLRQ RI WKH SHUVRQDOLVDWLRQ ZKLFK KDV EHHQ DSSOLHG DQG +HXULVWLFDSSURDFK$KHXULVWLFLVD JHQHUDOSULQFLSOHRUUXOHRI TXDQWLWDWLYHSHUIRUPDQFHPHWULFVUHODWLQJWRWKHFRQWHQWGHOLYHUHG WKXPE WKDW FDQ EH XVHG WR FULWLTXH H[LVWLQJ GHFLVLRQV RU JXLGH D GHVLJQGHFLVLRQ$QDSSURDFKZKLFKLQWHJUDWHVOD\HUHGHYDOXDWLRQ 7KHOD\HUHGDSSURDFKWRHYDOXDWLRQZRXOGDOORZWKHDVVHVVPHQWRI DQG KHXULVWLF HYDOXDWLRQ KDV EHHQ SURSRVHG >@ 7KH XVH RI HDFKLQGLYLGXDODVSHFWRISHUVRQDOLVDWLRQDSSOLHGZLWKLQWKH$,56 KHXULVWLFVHQVXUHVWKDWWKHHQWLUHV\VWHPFDQEHHYDOXDWHGLQGHSWK WR WDLORU WKH H[SHULHQFH GHOLYHUHG WR WKH XVHU %\ DVVHVVLQJ HDFK DQGVSHFLILF SUREOHPVFDQEHGLVFRYHUHGDWDQHDUO\ GHVLJQ VWDJH SLHFH RI IXQFWLRQDOLW\ LQ LVRODWLRQ LW FDQ EH GHWHUPLQHG ZKLFK EHIRUH UHOHDVLQJ D UXQQLQJ SURWRW\SH RI D V\VWHP >@ 7KLV SURYLGH WKH PRVW YDOXH LQ UHODWLRQ WR WKH H[SHULHQFH GHOLYHUHG WR DSSURDFK FDQ KHOS HYDOXDWRUV E\ LPSURYLQJ WKH GHWHFWLRQ DQG WKHXVHU7KHOD\HUVZKLFKPXVWEHHYDOXDWHGZLOOGLIIHUIRU HDFK GLDJQRVLVRISRWHQWLDOXVDELOLW\SUREOHPV V\VWHPDVWKHDGDSWDWLRQDQGSHUVRQDOL]DWLRQWHFKQLTXHVXVHGZLOO YDU\GHSHQGLQJRQWKHUDQJHRIPRGHOVDQGDGDSWLYHSUHVHQWDWLRQ  &KDOOHQJHVLQ(YDOXDWLQJRI$GDSWLYH6\VWHPV PRGDOLWLHVDYDLODEOHLQWKHV\VWHP 7KHHYDOXDWLRQRI$+6LVDGLIILFXOWWDVNGXHWRWKHFRPSOH[LW\RI 7KHUHDUHDVHWRIQHFHVVDU\HOHPHQWVRIDK\EULG$,56HYDOXDWLRQ VXFKV\VWHPVDVVKRZQE\PDQ\VWXGLHV>@>@,WLVRIFUXFLDO PRGHO ZKLFK FDQ EH GHILQHG LUUHVSHFWLYH RI WKH V\VWHP EHLQJ LPSRUWDQFHWKDW WKH DGDSWLYHIHDWXUHVRI WKH V\VWHPFDQEH HDVLO\ HYDOXDWHG7KHNH\FKDOOHQJHLVWREHDEOHWRDGHTXDWHO\FRPELQH GLVWLQJXLVKHG IURP WKH JHQHUDO XVDELOLW\ RI WKH GHVLJQHG WRRO WKHGDWDGULYHQDSSURDFKWRDVVHVVLQJUHWULHYDOIURP,5HYDOXDWLRQ ,VVXHV DULVH LQ WKH VHOHFWLRQ RI DSSOLFDEOH FULWHULD IRU WKH ZLWKWKHPRUHXVHUIRFXVHGDSSURDFKWRHYDOXDWLRQIURP$+7KH HYDOXDWLRQ RI DGDSWLYLW\ 0DQ\ PHWULFV FDQ EH XVHG WR PHDVXUH K\EULGIUDPHZRUNIRU$,56 HYDOXDWLRQ SURSRVHG E\WKLVSDSHULV SHUIRUPDQFH IRU H[DPSOH NQRZOHGJH JDLQ $(+6  DPRXQW RI DVIROORZV UHTXHVWHGPDWHULDOVGXUDWLRQRILQWHUDFWLRQQXPEHURIQDYLJDWLRQ VWHSV WDVN VXFFHVV XVDELOLW\ HJ HIIHFWLYHQHVV HIILFLHQF\ DQG  (YDOXDWLQJ 4XHU\ )RUPDWLRQ 7UDGLWLRQDO ,5 PHWKRGV DUH XVHUVDWLVIDFWLRQ 7KHHYDOXDWLRQRIDGDSWLYHV\VWHPVLVQRWHDV\ GULYHQ E\ NH\ZRUGGULYHQ TXHULHV +RZHYHU ZLWK WKH DQG VHYHUDO UHVHDUFKHUV KDYH SRLQWHG RXW SRWHQWLDO SLWIDOOV ZKHQ LQWURGXFWLRQRITXHU\WHUPSHUVRQDOL]DWLRQDQGH[SDQVLRQWKH HYDOXDWLQJDGDSWLYHV\VWHPV([DPSOHVRISLWIDOOV>@LQFOXGH HIIHFWLYHQHVV RI WKHVH FRPSRQHQWV PXVW EH HYDOXDWHG 7KLV FDQEHDFKLHYHGE\DVVHVVLQJWKHFRUUHFWQHVVRIWKHLQIHUHQFH  'LIILFXOW\ LQ DWWULEXWLQJ FDXVH LV WKH DGDSWDWLRQ FDXVLQJ WKH RIDXVHUV¶LQWHQWLRQDQGLQIRUPDWLRQQHHG PHDVXUHG HIIHFWRUDQRWKHUDVSHFWRIV\VWHPIXQFWLRQDOLW\RU  (YDOXDWLQJ 5HWULHYDO (IIHFWLYHQHVV 7KLV LV DQ DVVHVVPHQW GHVLJQ HJV\VWHPXVDELOLW\  RI WKH $,56 V\VWHP LQ WHUPV RI WUDGLWLRQDO UHWULHYDO WDVNV  6WDWLVWLFDOO\LQVLJQLILFDQWUHVXOWV$GDSWLYLW\LVW\SLFDOO\XVHG 7KHUH DUH VRPH DGGLWLRQDO FRQVWUDLQWV KRZHYHU DV V\VWHPV ZKHQ LQGLYLGXDO XVHUV GLIIHU +RZHYHU GLIIHUHQFHV LQ ZKLFK XVH DGDSWLYH SUHVHQWDWLRQ PHWKRGV DUH UHTXLUHG WR DSSURDFK DQG SUHIHUHQFHV DUH OLNHO\ WR OHDG WR D ODUJH UHWULHYH RUGHUHG VHTXHQFHVRI FRQWHQW ZKLFKFUHDWHV D PRUH YDULDQFH LQ SHUIRUPDQFH UHVXOWV ZKLFK PDNHV LW PRUH FRPSOH[ GHSHQGHQF\ WKDQ D OLVW RI LQGHSHQGHQW FDQGLGDWH GLIILFXOWWRSURGXFHVWDWLVWLFDOO\FRPSDUDEOHUHVXOWV,QRUGHU GRFXPHQWV WR SURGXFH VLJQLILFDQW UHVXOWV ODUJH YROXPHV RI TXHULHV DQG  8VHU&HQWULF (YDOXDWLRQ $ XVHUFHQWULF HYDOXDWLRQ RI WKH XVHUVDUHUHTXLUHG7KHUHDUHIHZJHQHUDOJXLGHOLQHVIRUWKH V\VWHP LV QHFHVVDU\ WR DVVHVV WKH DVVXPSWLRQV PDGH DQG VHOHFWLRQRIWKHVHPHDVXUHPHQWV DGDSWLYLW\SHUIRUPHG.H\PHWULFVLQXVHUFHQWULFDSSURDFKHV  'LIILFXOW\ LQ GHILQLQJ WKH HIIHFWLYHQHVV RI DGDSWDWLRQ ,W FDQ WR HYDOXDWLRQ LQFOXGH VDWLVIDFWLRQ HIIHFWLYHQHVV DQG EH GLIILFXOW WR GHILQH ZKDW FRQVWLWXWHV D XVHIXO RU KHOSIXO HIILFLHQF\ DGDSWDWLRQ D (YDOXDWLQJ $GDSWLYLW\ (IIHFWLYHQHVV DQG (IILFLHQF\  ,QVXIILFLHQWUHVRXUFHV7RIXOO\HYDOXDWHDQDGDSWLYHV\VWHPLW 7KH HIIHFWLYHQHVV RI DGDSWLYLW\ SHUIRUPHG E\ WKH V\VWHP LV RIWHQ QHFHVVDU\ WR KDYH D ODUJH QXPEHU RI LQGLYLGXDOV FDQ EH PHDVXUHG XVLQJ GRPDLQ DSSURSULDWH SHUIRUPDQFH LQWHUDFWLQJ ZLWK WKH V\VWHP 7KLV LV LQ SDUW GXH WR WKH LQGLFDWRUV IRU H[DPSOH NQRZOHGJH JDLQ DQG NQRZOHGJH H[SHFWHGYDULDQFHEHWZHHQSDUWLFLSDQWVPHQWLRQHGDERYH UHWHQWLRQ LQ HGXFDWLRQDO VFHQDULRV DQG LQIRUPDWLRQ QHHG  7RR PXFK HPSKDVLV RQ VXPPDWLYH UDWKHU WKDQ IRUPDWLYH IXOILOOPHQW LQ ,5 'LIIHUHQW PHWULFV FDQ EH DVVHVVHG HYDOXDWLRQ(YDOXDWLRQVRIWHQPHDVXUHRQO\KRZJRRGRUEDG GHSHQGLQJ RQ WKH QDWXUH RI WKH $,56 DSSOLFDWLRQ DQG LWV D V\VWHP LV UDWKHU WKDQ SURYLGLQJ LQIRUPDWLRQ RQ ZKHUH WKH XVH7KLV LVQHFHVVDULO\D XVHUGULYHQHYDOXDWLRQEHFDXVH SUREOHPVDUHDQGKRZDV\VWHPFDQEHLPSURYHG WKH EHKDYLRU RI WKH V\VWHP GHSHQGV RQ WKH SURSHUWLHV RI WKH XVHU (IILFLHQF\ FDQ EH PHDVXUHG E\ H[DPLQLQJ XVHU 7KHVHOHFWLRQRIWKHPHWULFVWREHXVHGLQWKHHYDOXDWLRQRI$+6LV SHUIRUPDQFH ZKHQ FRQGXFWLQJ D VHW RI GHILQHG WDVNV FUXFLDO 7KHUH DUH FXUUHQWO\ QR DJUHHG HYDOXDWLRQ PHWKRGRORJ\ 0HWULFVPHDVXUHGFDQLQFOXGHWLPHWDNHQ7LPHWDNHQFDQ VWDQGDUGV WKXV PDNLQJ $+ HYDOXDWLRQ D GLIILFXOW FRPSOH[ DQG LQFOXGH WZR W\SHV RI PHDVXUHPHQWV FODVVLFDO ,5 WLPHFRQVXPLQJWDVN HYDOXDWLRQVVXFKDVWLPHWRJHWWKHILUVWUHOHYDQWGRFXPHQW  (9$/8$7,210(7+2'2/2*< DQGQXPEHURIGRFXPHQWVUHWULHYHGLQH[DFWO\PQ>@ QXPEHURITXHULHVQHHGHGHWF 7KHUHDUHPDQ\PHWKRGVRIFRPELQLQJWKHWHFKQLTXHVXVHGLQ$+ E 8VHU 6DWLVIDFWLRQ 8VHU VDWLVIDFWLRQ FDQ EH H[DPLQHG E\ DQG ,5 +RZHYHU LQ RUGHU WR GHILQH D FRPPRQ HYDOXDWLRQ HOLFLWLQJ GLUHFW XVHU IHHGEDFN :KLOH WKHUH DUH PDQ\ PHDQV RI HOLFLWLQJ VXFK LQIRUPDWLRQ DPRQJ WKH PRVW >@ &+|OVFKHU *6WUXEH:HE6HDUFK%HKDYLRURI,QWHUQHW ZLGHO\ XVHG DUH TXDOLWDWLYH TXHVWLRQQDLUHV VXFK DV WKH ([SHUWVDQG1HZELHV,Q3URFHHGLQJVRIWKHWK,QWHUQDWLRQDO 6\VWHP8VDELOLW\6FRUH 686 >@ &RQIHUHQFHRQWKH:RUOG:LGH:HE:::$PVWHUGDP7KH  6\VWHPHIILFLHQF\7KHFUHDWLRQRIFRPSOH[$,56UHTXLUHVD 1HWKHUODQGVSS ZLGH YDULHW\ RI FRQWHQW DQG PHWDGDWD $V VXFK WKH RYHUDOO >@ 3%UXVLORYVN\(6FKZDU] *:HEHU$7RROIRU'HYHORSLQJ FRVWRISURYLVLRQLQJDSDUWLFXODU$,56DQGWKHSHUIRUPDQFH $GDSWLYH(OHFWURQLF7H[WERRNVRQ:::,QWKH3URFHHGLQJVRI RI WKDW V\VWHP LQ WHUPV RI UHVRXUFH UHTXLUHPHQWV DQG :HE1HW :RUOGFRQIHUHQFHRIWKHZHEVRFLHW\SS UHVSRQVLYHQHVV PXVW EH DVVHVVHG 7KLV LV SDUWLFXODUO\  LPSRUWDQWDVV\VWHPVDSSURDFKPDVVZHEVFDOHXVH >@ &*HQD 6:HLEHO]DKO8VDELOLW\(QJLQHHULQJIRUWKH $GDSWLYH:HEYRO/1&6SS  6800$5< >@ 0'H-RQJ 36FKHOOHQV5HDGHU)RFXVHG7H[W(YDOXDWLRQ 7KLV SDSHU KDV SUHVHQWHG DQ RYHUYLHZ RI WKH HYDOXDWLRQ $QRYHUYLHZRIJRDOVDQGPHWKRGVYRO  SS DSSURDFKHVFRPPRQO\XVHGLQWKH,5DQG$+FRPPXQLWLHVZLWK  WKHREMHFWLYHRIILQGLQJDPHDQVRIFRPELQLQJWKHVHDSSURDFKHVWR >@ -1LHOVHQ8VDELOLW\(QJLQHHULQJ%RVWRQ0$$FDGHPLF3UHVV DVVHVV WKH QH[W JHQHUDWLRQ RI SHUVRQDOL]DWLRQ LQ ,5 7KHVH  DSSURDFKHV KDYH WUDGLWLRQDOO\ YDULHG LQ IRFXV WKH PDMRULW\ RI ,5 HYDOXDWLRQ LV IRFXVHG RQ WKH QXPHULFDO SHUIRUPDQFH DQG >@ &*HQD0HWKRGVDQGWHFKQLTXHVIRUWKHHYDOXDWLRQRIXVHU HIIHFWLYHQHVVRIWKHUHWULHYDORIGRFXPHQWVZKLOH$+V\VWHPVDUH DGDSWLYHV\VWHPV7KHNQRZOHGJHHQJLQHHULQJUHYLHZYRO SS8QLWHG.LQJGRP&DPEULGJH8QLYHUVLW\3UHVV HYDOXDWHGZLWKWKHXVHUVDVDQLQWHJUDOSDUW >@ &.DUDJLDQQLGLV '6DPSVRQ/D\HUHGHYDOXDWLRQRIDGDSWLYH 7KHOD\HUHGHYDOXDWLRQPRGHOSURSRVHGLQWKLVSDSHULVDQDWWHPSW DSSOLFDWLRQVDQGVHUYLFHV,QWHUQDWLRQDO&RQIHUHQFHRQDGDSWLYH WR GUDZ IURP WKH $+ FRPPXQLW\¶V H[SHULHQFH ZLWK WKH FRPSOH[ K\SHUPHGLDDQGDGDSWLYHDSSOLFDWLRQVDQGVHUYLFHV GHSHQGHQFLHVRIGLIIHUHQWLQWHOOLJHQWFRPSRQHQWVLQ$+67KLVLV >@ 3%UXVLORYVN\&.DUDJLDQQLGLV '6DPSVRQ7KHEHQHILWVRI FRPELQHGZLWKWUDGLWLRQDO,5 HYDOXDWLRQ PHWKRGVWRJDLQDEURDG OD\HUHGHYDOXDWLRQRIDGDSWLYHDSSOLFDWLRQVDQGVHUYLFHV,QWKH VSHFWUXPDVVHVVPHQWRIWKHHQWLUHV\VWHP SURFHHGLQJVRIWKHILUVWZRUNVKRSRQHPSLULFDOHYDOXDWLRQRI DGDSWLYHV\VWHPV806RQWKRIHQ*HUPDQ\SS 8VHUGULYHQHYDOXDWLRQRI,5LVQRWQHZLQLWVHOIEXWWKHHIIHFWVRI SHUVRQDOL]DWLRQ RQ FUHDWLQJ UHSURGXFLEOH ODUJH VFDOH H[SHULPHQWV >@ 6:HLEHO]DKO *:HEHU$GYDQWDJHVRSSRUWXQLWLHVDQG FDQ LQ WKH RSLQLRQ RI WKH DXWKRUV EHVW EH DGGUHVVHG E\ OLPLWVRIHPSLULFDOHYDOXDWLRQV(YDOXDWLQJDGDSWLYHV\VWHPV LQFRUSRUDWLQJHOHPHQWVRI$+HYDOXDWLRQWHFKQLTXHV YRO   >@ '&KLQ(PSLULFDOHYDOXDWLRQRIXVHUPRGHOVDQGXVHUDGDSWHG  $&.12:/('*0(176 V\VWHPVSS 7KLV UHVHDUFK LV EDVHG XSRQ ZRUNV VXSSRUWHG E\ 6FLHQFH >@ (+HUGHU8WLOLW\EDVHGHYDOXDWLRQRIDGDSWLYHV\VWHPV,QWKH )RXQGDWLRQ ,UHODQG *UDQW 1XPEHU &(,  DV SDUW RI WKH SURFHHGLQJVRIWKH6HFRQG:RUNVKRSRQ(PSLULFDO(YDOXDWLRQ &HQWUH IRU 1H[W *HQHUDWLRQ /RFDOL]DWLRQ ZZZFQJOLH   7KH RI$GDSWLYH6\VWHPVDWWKHWK,QWHUQDWLRQDO&RQIHUHQFHRQ DXWKRUV DUH JUDWHIXO IRU WKH VXJJHVWLRQV RI WKH UHYLHZHUV IRU WKLV 8VHU0RGHOLQJ803LWWVEXUJ86$SS SDSHU >@ *0DJRXODV6&KHQ .3DSDQLNRODRX,QWHJUDWLQJOD\HUHG DQGKHXULVWLFHYDOXDWLRQIRUDGDSWLYHOHDUQLQJHQYLURQPHQWV,Q  5()(5(1&(6 WKHSURFHHGLQJVRIWKH6HFRQG:RUNVKRSRQ(PSLULFDO  (YDOXDWLRQRI$GDSWLYH6\VWHPVDWWKHWK,QWHUQDWLRQDO >@ *-)-RQHV 9:DGH,QWHJUDWHGFRQWHQWSUHVHQWDWLRQIRU &RQIHUHQFHRQ8VHU0RGHOLQJ803LWWVEXUJ86$SS PXOWLOLQJXDODQGPXOWLPHGLDLQIRUPDWLRQDFFHVV,Q)&*H\1  .DQGR&</LQDQG&3HWHUV HGV 1HZ'LUHFWLRQVLQ 0XOWLOLQJXDO,QIRUPDWLRQ$FFHVVYROSS± >@ /)X*6DOYHQG\ /7XUOH\(IIHFWLYHQHVVRIXVHUWHVWLQJ DQGKHXULVWLFHYDOXDWLRQDVDIXQFWLRQRISHUIRUPDQFH >@ %6WHLFKHQ6/DZOHVV$2¶&RQQRU 9:DGH'\QDPLF FODVVLILFDWLRQ%HKDYLRXUDQGLQIRUPDWLRQWHFKQRORJ\YRO +\SHUWH[W*HQHUDWLRQIRU5HXVLQJ2SHQ&RUSXV&RQWHQW,QWKH SS 3URFHHGLQJVRIWKHWK$&0&RQIHUHQFHRQ+\SHUWH[WDQG +\SHUPHGLD+\SHUWH[W7RULQR,WDO\WK-XQH±VW-XO\ >@ )'HO0LVVLHU )5LFFL8QGHUVWDQGLQJUHFRPPHQGHU  V\VWHPV([SHULPHQWDOHYDOXDWLRQFKDOOHQJHVSS >@ &:&OHYHUGRQ-0LOOV 0.HHQ)DFWRUVGHWHUPLQLQJWKH >@ 7/DYLH-0H\HU.%HXJOHU -&RXJKOLQ7KHHYDOXDWLRQRI SHUIRUPDQFHRILQGH[LQJV\VWHPV9RO'HVLJQ$6/,% LQYHKLFOHDGDSWLYHV\VWHPV8VHU0RGHOLQJ:RUNRQWKH($6 &UDQILHOG3URMHFW7HFKQLFDO5HSRUW SS >@ *6DOWRQ$XWRPDWLF7H[W3URFHVVLQJ7KH7UDQVIRUPDWLRQ >@ 17LQWDUHYDQG-0DVWKRII(YDOXDWLQJ5HFRPPHQGHU $QDO\VLVDQG5HWULHYDORI,QIRUPDWLRQE\&RPSXWHU$GGLVRQ ([SODQDWLRQV3UREOHPV([SHULHQFHGDQG/HVVRQV/HDUQHGIRU :HVOH\ WKH80$3S >@ .YDQ5LMVEHUJHQ,QIRUPDWLRQ5HWULHYDO/RQGRQ(QJODQG >@(&UHVWDQ&GH/RXS\%URZVLQJ+HOSIRUD)DVWHU5HWULHYDO %XWWHUZRUWKV &R/WG &2/,1*SS*HQHYH6XLVVH >@-%URRNH686DTXLFNDQGGLUW\XVDELOLW\VFDOH,Q >@ 6&KDNUDEDUWL0YDQGHQ%HUJ %'RP)RFXVHG 8VDELOLW\(YDOXDWLRQLQ,QGXVWU\3:-RUGDQ%7KRPDV &UDZOLQJ$1HZ$SSURDFKWR7RSLF6SHFLILF:HE5HVRXUFH %$:HHUGPHHVWHU $/0F&OHOODQG HGV  'LVFRYHU\,Q7KH,QWHUQDWLRQDO-RXUQDORI&RPSXWHUDQG /RQGRQ7D\ORUDQG)UDQFLV 7HOHFRPPXQLFDWLRQV1HWZRUNLQJ9RO  (OVHYLHUSS  0D\