This paper investigates the notion of visual load, an estimate for a lexical item's e cacy in activating mental images associated with the concept it refers to. We elaborate on the centrality of this notion which is deeply and variously connected to lexical processing. A computational model of the visual load is introduced that builds on few low level features and on the dependency structure of sentences. The system implementing the proposed model has been experimentally assessed and shown to reasonably approximate human response.
Ordinary experience suggests that lexical competence,
i.e. the ability to use words, includes both the
ability to relate words to the external world as accessed
through perception (referential tasks) and the ability to
relate words to other words in inferential tasks of
several kinds
As it is widely acknowledged, one main visual correlate of language is imageability, that is the property of a particular word or sentence to produce an experience of imagery: in the following, we focus on visual imagery (thus disregarding acoustic, olfactory and tactile imagery), which we denote as visual load. The visual load is related to the easiness of producing visual imagery when an external linguistic stimulus is processed.
Intuitively, words like `dog' or `apple' refer to concrete
entities and are associated with a high visual load,
implying that these terms immediately generate a mental
image. Conversely, words like `algebra' or `idempotence'
are hardly accompanied by the production of vivid
images. Although the construct of visual load is closely
related to that of concreteness, concreteness and visual
load can clearly dissociate, in that i) some words have
been rated high in visual load but low in concreteness,
such as some concrete nouns that have been rated low
in visual load
The notion of visual load is relevant to many disciplines, in that it contributes to shed light on a wide variety of cognitive and linguistic tasks and helps explaining a plethora of phenomena observed in both impaired and normal subjects. In the next Section we survey a multidisciplinary literature showing how mental imagery affects memory, learning and comprehension; we consider how imagery is characterized at the neural level; and we show how visual information is exploited in state-of-theart Natural Language Processing research. In the subsequent Section we illustrate the proposed computational model for providing concepts with their visual load characterization. We then describe the experiments designed to assess the model through an implemented system, report and discuss the obtained results. Conclusion will summarize the work done and provide an outlook on future work.
As regards linguistic competence, it is generally
accepted that visual load facilitates cognitive
performance
L’ animale che mangia banane su un albero `e la scimmia
The animal that eats bananas on a tree is the monkey
FFigiugurere1:1:TThehede(psiemndpelinceyd)trdeeepceonrdreesnpcoyntdrienegctoorraessptiomn-duilnugs. to the sentence `The animal that eats bananas on a
tree is the Monkey'.
of information processing—, including a deep
representamtioorne, wquhiicckhlyis aandsemacacnutricatneleytwworhkenstomreadkining l ojundgg-tmeremnts
maebmouortyvitshuaatllycolonatadiends saenhteienrcaersch(Kiciarlanre&prTesuecnhttaetniohnagoefn,
im20a0g5e).descNrieputrioopnssy;cthhoelosgpiactailalrreespearerscehnthaatisonshionwtenndtehdat
fomr acnoyllecatpihnagsiimcapgaeticeonmtspopneernfotsrmalobnegttweirthwtihtheirlisnpgautiisatlic
feiatetmursest;htahtemviosruealearesiplryeseelinctiattvioisnuathlaimtbaugeilrdys (oCnoaltnheoacr-t,
cu1p98a0n)c,yaaltrhroayu,ghstothriengopipnofosritmeaptaiotntersnuchhasasalsshoabpee,ensidzeo,cetucm..ented
Visual imageability of concepts evoked by words and sentences is commonlyMkonodwenl to a ect brain activity.
TWhehmileodveisluinogsepmhaansetihcapsrboceeesnsicnhgarraecgtieornisz,edsubcyh tahselenfetedinoffedrieofirntienmgptohrealngoytriuons aonfdvfiussuiafol rlmoagdyrinusareuvneaifloerdmgraenatder coinmvpoluvteamtieonntaldlyuritnragctthabelceommpanrenheern.sioSnucohf hciognhclyepitm,aignefaacbtl,e iwsourdsesdanbdy saenndtenlacregse(Binoolkitheeriamtuerreetwaitl.h, 1d9i9↵8e;reMntelmleeta,niTngzos,urtihou,s Dgeivniins,g &raisMe atzooydeir↵,er1e9n9t8)le,veoltsheorf asemmbai-ngutiicty.brWaien dreefignioenvsisu(ia.el.l,oasdupaesritohre aconndcemptidrdelperetseemntpinorgal acdoirrteecxt)inadreicasetolerct(iavenluymaecrtiicvavtaeldueb)yoflotwhe-imeacgaecayblfeorseanletxeinccaelsit(eMmeltloetaecttiavla.t,e19m9e8n;tJaulsitm, aNgeewsmasasnoc,iKateeldlert,oMthceEcolennceeyp,t&reCfearrrpeedntteor,b2y00t4h)e. lFeuxirctahlerimteomr.e, Caognrosweqiunegnntulym,wbeerexopf escttudthieastsuvgisgueasltslotahdatawlsoordrsepernecsoedntinsgadni inerdeinretcvtimseuaaslupreroopfetrhtieesp(rsoubcahbialistycoolfora,ctsihvaaptieo,nmooftaiobnr,aeintc.a)reaare dpeproucteesdsetdo itnhecovritsiucaall parreoacsestshinagt.overlap with some of the aWreaesctohnajtecatrueraectthivaattveidsudaulrlionagdviissusiatlupaetrecdepattitohneoifntthero-se sepcrtoiopnerotfielsex(iKcaelmamnderseerm,2a0n1t0ic).spaces, mostly associated to tIhnevseesmtigaanttiincgletvheel.vTishuaatl fies,attuhreevsiasussaolclioaatdedistoprliimngauri-siltyicasinsopcuiattceadntobea ucsoenfcuelptto, abltuhilodugshemleaxnitciacl rpehseonuorcmeesndaeliskiegnteerdmtsoadveaaillawbiiltihtyN(aimtuprlaylinLganthgautatgheePmroocsetsfsrienqgu(eNntLlyP) upserdobtleermmss, saurceheaassiienrdtiovidreucaotginngizevetrhbasnsutbhocasetesgeoernizaletsison offtreanm(eTsv(eBresrkgysm&aK&ahGneomebaenl,, 12907131))), ceannricahlsionga↵ thecet tirta. ditiBonasaeldeoxntrtahcetiwonorkofbydiKstermibmuteiroenra(l2s0e1m0)awnteicesxpfrloorme ttheext hwypitohthaesis
multthimatodaalli mapitperdoancuhm,ibnetregorfatpirnigmtiteixvteuaellefmeaetnutrses can witbhevuissueadl tooncehsa(rBarcutenrii,zTeraannd, &evaBluaraotneit,h2e01v4is)u. aFlilnoaaldly, asvsiosuciaalteadtttroibcuotnesceaprtes.aNtatmheelbya,sKeeomfmtheererd’esvSeilmopumlaetniotnof Faranmnoetwaoterkd aclolropwosratoagnrdasrpesionuforcrmesatthioant caabnoubteauwseiddetovae-xriteetnydoftecxotn-cbeapstesdadnidstprirboupteirotnieasl usesemdatnoticdsenboytegorboujencdtsin,g evweonrtds manedanspinagtsiaolnrevlaistuioanlsf.eaTthurreees,maasiwnevlils(uSaillbseemrear,ntFiecrcoramrpi,o&neLntaspahtaav,e20b1e3en). individuated that, in our opinion, are also suitable to be used as di↵ erent dimensions along which to characterMizeotdheelconcept of visual load.
TAhletyhoaurge:h cmoulocrh pwroorpkerhtiaess,bseheanpeinvpersotpeedrtiinesd,ianerdenmt oa-rtieoans pforropinevrteisetsi.gaTtinhge ipmeracgeepatbi oilnityofinthgeesneerparlopaenrdtievsisiusal eximpeacgteerdy tion opcacrutricuinlaar,imatmtehdeiabteestwaoyf,osuurchkntohwatle“ddguern-o inagttoemurptorhdainsabryeeonbscearrvraietdionouotf ttohefowr moraldll,y tchheaseratchterreieze attributes of objects are tightly bound together in visual load, and no computational model has beeunndi-evised to compute how visually loaded are sentences and lexicalized concepts therein. We propose a model that relies on a simple hypothesis additively combining few low-level features, re ned by exploiting syntactic information.
The notion of visual load, in fact, is used by and large in literature with di erent meanings, thus giving rise to di erent levels of ambiguity. We de ne visual load as the concept representing a direct indicator (a numeric value) of the e cacy for a lexical item to activate mental images associated to the concept referred to by the lexical item.
We expect that visual load also represents an indirect measure of the probability of activation of brain areas deputed to the visual processing.
We conjecture that the visual load is primarily
associated to concepts, although lexical phenomena like
terms availability (implying that the most frequently
used terms are easier to recognize than those seen less
often
Based on the work by
They are: color properties, shape properties, and motion properties. The perception of these properties is expected to occur in a immediate way, such that \during our ordinary observation of the world, these three attributes of objects are tightly bound together in uni
ed conscious images"
In this setting, we represent each concept/property as a boolean-valued vector of four elements, each encoding the following information: lemma, morphological information on POS (part of speech), and then whether the considered concept/property conveys information about color, shape, motion and size.1 For example, this piece of information
1We adopt here a simpli cation, since we are assuming that the pair hlemma; POSi is su cient to identify a concept/property, and that in general we can access items by disregarding the word sense disambiguation problem, which is known as an open problem in the eld of NLP.
thTeurfionlloUwniinvgerisnitfyormPaartsieorn:(TleUmPm)ain, mthoerpdheopleongdiceanlcyinffoorr-- load of the concepts denoted by the lexical ite finger,Noun,1,1,0,1 (1) mmataiton(LoensmPoO,S20(0p7a)r.t oDfesppeenedcehn)c,ya nfodr mthaelinsmwshertehperrestehnet sentence, that is VL(s) = Pc2 s VL(c). can be used to indicate that the concept finger (associ- ccooshylneonasritd,daesc(rhetei.acgdp.er,ce,tolhamnetcoieovtpnieotrs/nbbp‘yraflonycpd’oenirsnntiyzetechc.t1eoinnsFgveoneayrtesdenoxicnmaefmoiTnrpmahlneeat,etiatwohgnoliesradflpb,iioteehucseet) depTehnedceanlccyulsattriuocntuorfethoef tVhLe sincopruet aslesnoteancccoesu.n ated to a Noun, and di↵ ering, e.g., from that associated ofa nindfoarmdoamtioinnated word, the dependent (e.g., the noun tactic structure of sentences is computed th tsoizea, bVuetrbn)otcoanbvoeuytsminoftoiromn.atIinonthaebfoolulotwcionlogrt,hsehsaepaerearned- ‘tehaegslee’t wino twhoerdsasmisfienusgseuenrat,leNlnyocureen)p.,1r,Te1sh,e0en,1tceodnbnyecutisoinngbleatbwee(lee1nd) Tmuartin(LUensmivoe,rs2it0y07P)a.rsDerep(eTnUdPen)ciyn ftohrmeadleipsmens z|The big carnivore withdeyfienl}ilt|oiownftssc{eheedaozrtnennWrtcdoeeveedfnipbshscttultaeoa.iasvmcalekdsufteelsthisathcetr(ernuiiiplvbrlbueeiiusnsssuitgliaratsasalasttfoecheddoca{eiincta.zhutctt.aeiereo.rrped}gtneset|iaat;gtfroeTaty·{}renaerbd)asyscmcoheocxamictnaorputaneoaccdslteleiyptdnoMtga.btnoiyhtnTrepsofhritghmoeaiomvtpalheeoslldnaeeg-micamftscaleoiatfbddTmazrieenieneer,hcaradnce,PfteeiboadcnbcVtOuemrytodutemesveSotuarredaepbeayisnarlewte)eNbaosinaddeottcopdtgi(nouohtapeeasnonnbenseslvyess,co(ivueynueaoiiytm.sndntfhgssuidee.matcca,iranaltdussoptshfeufitcaeoee↵naibtrsroauttejtsmneterherwcusni.eaanothrctttI)mgerieein:ses,ortefeethnterohhe.esr·bigealoelmelao.cbru,cvfstostooosahsfntrlluaonollcroeotacftaetmcwirptracitweeenitoittendoioelnh,fiongrodfadirrnuontoo,ttgrhrofaocseaFentaaehrtsseilpmdhsaegld(oppeDwuaedaicrietnrsegrnoiiisesatepctiaaovhtttedcr1nceeehinheeo)ifnddfe-----.,enaatruy‘tsadhrehyeniearnesadngestdcnleaeatoc(’etettdwdii.ancgooet.mder,twedhgitlnoheearasetdsti(oaevsednme.igrsseb.w,buso‘yssfleruundycbat,’ojleenlintcynnhtce)eretc:eh)tp.detirehnTespegesehencanentodetldcleneeoodnccmnettbniiTo(yneencah.utgneois.otfi,e stimsuilguns mstent of features scores has been conducted by thhleemmcopane,cnPedOpetnS.ti(s) (or governor(s)) are associated to some visual dency relations of a sentence forms a tree, roo tested in order to set features contribution to the {visua.l...s.etnhtaenncitess edleemscerinbtisntgaakecnoninceipsotl;aatniodnm(tahnautailsle,ym‘bamnlancaok,t’PaaOtneddS, cobl,ecsahuase, wmeota,sssuimze that some sort of reinforc auDthio↵resreonnt awpeiugrhetliyngintsrcohsepmeecstivw~e =bas{i↵s., , } have behhellneemmmmseifaasetW,,aeoPPtxefuOOphsreteSSaic.vmiiteeFudtolhirteo(enixlelbavumuositkplretlaeamt,ededainchpttaehiolrrneaimaasferatyeglirbek)syecieo‘nwxmtairtpahmocslobteeirldnmaegcbvmkiyitvsaisftd,rirPmoipwmOpealsSyae’, coTml,haeisnhdave,eprembndo(tes,neecsyitzhsetrupcatruseretriseereilleluvsatnrtatiendoiunr load associated to a concept c, that results from com- t h‘setrvipiseusa’)l, feaantdurietss avsissuoaclialtoeaddtois eeaxcphecctoendcleetpomt.smtTiallh, PegrOaoswS- , cofle,ctshmaa,y mapotp,lysiinz cases where both a word a puting siigfnwmeenadtdofafecaotourrdeinsastceodretserhma,s lbikeeeninco‘wndit.uh.cty.ee.dl.lobwy atnhde pendent(s) (or governor(s)) are associated to so VL(c, w~ ) = X i = ↵ ( col + sha)+ mot + siz. (2) aubwDtleahicao↵kdredsrseetodrnnitpaaewgsope’.viugerYhrenteliyotn,rginttehtsrrcemohVsep(mLleikecwsteoiv‘wu~feuldr=baw–s{riiet↵shc.,uyre,slilvo}ewlhya–anvgderboblweaecinkf ifseaetxupreec.teFdortoexeavmokpelem,eanptahlraimseagliekse i‘nwaithmobrlae TUP parseri tessttreipdeisn’).orWdeer tthoensetinfteraotduurceesdcoanptrairbaumtieotnerto⇠ tthoecvonisturaoll than its elements taken in isolation (that is, ‘ For the experimentation we set ↵ to 1.35, to 1.1 and lotahde caossnotcriibautetdiontoofa tchoenacfeoprtemc,enthtiaotnerdesfuelattsufrreosminccoamse- ‘stripes’), and its visual load is expected to to .9. ptuhteincgorresponding terms are linked in the parse tree by if we add a coordinated term, like in ‘with y To the ends of combining the contribution of concepts a modifier/aXrgument relation (denoted as mod and arg black stripes’. Yet, the VL would –recursivel itmnhaeainps.reiInnntceiopntlcheeeorsDfwtceoooprmdtehpsn,oedsweoitevnieocarnysasalusllimttVryeuL2cttshtocuaottrrheetehfeovrivsiuss,aulawlleolaoadaddadpoot-f FViVonrLLE((tcchqi,e)uw~a=e)txi=(opne⇠r3iiVm).Le(nicti=)at↵io(inf c9wolec+jsest.sth.↵am)t+ood1(.c3im5,c,otj )+_ toar1sgi.z1(.cai(,n2cdj)) tswhtereiapcdeodsn’e)td.riabWugetoivotehnrenonofritnthteerromadfu(olcrikeedem‘eafunprtiaowrnaietmhdeyfteeelarltou⇠wr a sentence can be computed by starting from the visual VL(ci) otherwise. the corresponding terms are linked in the par a modifier/argument relation (denoted as mo in Equation 3).
Non-Visual Target—Non-Visual Definition to .9. tha1tWtheeadpoapirt hhleemremaa,sPimOpSliifiicsatsiuon,ciseinntcetoweidaenretifayssaumcoinng- inInTaothsteehneetxeenpncedersi msofetnoctoatmthibeoinnoiv⇠negwraatlhslesVecLtontsotcro1irb.e2u.tfoiornso,fwceoNnacdeaNp(p3tVst) cept/property, and that in general we can accesws eiitgehmtsinbgy schtehmeeprw~in,ciisplteheonf ccoommppoustiteidonaasliftoyl2lotwost:he visual loadVdoiitadPwssorireseaaofr2ccocesfTotryeguummshannasritcbisldanctliyciilngpnoateranuig(icnVwnsaotecolihofslidry,ppethdhlaecueswnenootdsBmmporthdarsbefootaoaiebsnlrnaoeslemienndnt&ms:hgp(esaPhtAidtonoarbiafrtsgtitshairheeetameteessm,,tbmpfiht1iiae,hgee9lruae2d9tmnn0as5o1ti)eawnfi.4aongeN)nnd.icoanpooftgufrmaortonbbahfpilleneeaiLxmpewAsapeepa,nrglnyhewiggtnrsuteihasehuaniis.ocegcnenhyeesgeretcadmheiaTVspasVafirhe1|zTittLgeTdLnWn/egh(ue(tp.thTahfidervererIae,,onnbidnsawwpiic~~iusgdptneeeoti))aoargocitlmptcia==nhdryattrl,eeuhnonhrdshliealaviceebdwnPomrwraderiieonnoeazmVcrcrisdta2wdodaosLhtdsoni,mah(stas,PcVehTiteemdpiwOnaeLt)yiufidpnaset.de(Snetlir}caiiilseatdg|gofi)ttidSdsiaoeewimacinsstntlsays{uutbodaaeetlzossumyTrnimtuossatdnetbsslleati(tb,ivcmctwgrsalihoeesautreeilcmatannlkipt=octitcnpimaaseotgottnidhnrorhnwpifeesdpa((repelil,54ocendlvrecTascm))toaeioemssinbrissnu,e)tlstes,iateghfmahiilyinsetss{ev1nFtl,esut.zeoi9uaovtmunwc.iana(2prmim.rhngsescdh}grttueo.|esibgoitggnbhcaenaoaobeehTy.gens-fcfl}{,rrtlht.ehTeomnecfhsocIVewenissLodqT.triu(shthdcEaaieeisoila)dinetc(=sxy,hn.tp.iot(mote.hurfre⇠VuniiiarnpmlsVLeilteeea(olLwinalcnpl(tiseidlc)agtreiettee)haidntode4hcna8jeaebiod)⇠tcfs.toatehw9eutinaveatactrsessejwie3slnt)sy0siecs.tetemas..nsort,iedlmonvfueo1opttrh.dade2(eiso.irc.vsi(,esTcrycujahon)llnet-l_ is actually an open problem incothme pfieuldtaoftiNoantuarlalmLaondguealgedoefpVenaLddeneficyn)itsiotnrucdtuarnedofathtaercgoentsiTde(rsetd s=enhtden,Tceis) ProcEesxsinpge(rViimdhuenBthaaltai&onAbirami, 2014). tactic
2This principle states that the meaning of an expwreersseiohnomogeneous within conditdieofinnisti.on d Materials iaasrnea dsfyunnMtcatcieottnichaoloflytdhcseommbeainneidn:gstoofgietts tphaertms eaanndinogf tohf eawseanyTttehhneceye same zTsehte boifg sctairmniuvolireuwseitdh yfoelrlowthaend black stripes is
}| | {z human experiFigure 2: The pipeline to compute the VL Fsocrtoyr-fieveahcecawloethcryodmvibonilnugenwtetoerordss t(t2oh3feofremmpaprlhoersapsaeons,dsteh2e2dnmwcaeolceomsm),bpinue mtpaherntatsieows ansaglivemn ion dineplu.t to the systesmtimuimluspsltementing the 19 52 yeartso ofofrmagecla(umseesa, nan±dssdo o=n (2P5a.r7te±e, 51.919)5),. were proposed coTmhpeuvtiastuiaolnlaolamd oadsesol.ciTatheedsytostsetmcowmaspounseednttso, recruited for the experiment. One of them was excluded compute the visual load score associated to (lexicalized) because she was outlier with respect to the group. None concepts according to Eq. 4 and 5, implementing the vireferred to as the visual features associated withoft hthee subje`cetsahgalde'a ihnistotrhyeof spasymcheiatrsiecnortenneucreo-). sTuahl eloacdomnondeel cintiEoqn.2,bweitthwteheensystem’s parameters logical disorders. All participants gave their written in- set to the aforementioned values. given concept. formed consetnht ebseeforet wtaokinwgoparrdt sto itsheuesxupearilml yentralepresented by using labeled We have then built a dictionary by extracting it fprroomcedure, wdhiicrhewcatseadpperdovgedesby(teh.ege.t,hiscualbcjoemcmti)t-: thDeactaolalneacltyisoisn of all depentee of the University of Turin, in accordance with the a set of stimuli (illustrated hereafter) composed of sDiemcla-ration dofeHneclsyinkrie( lBaMtiJo1n99s1;o3f02a: 1s1e9n4t)e.nPcare- forTmhe spaartitcirpeanet,s’rpoeorfotremdanicne itnhthee “naming by defple sentences describing a concept; next, we have mtiacinpa-nts wermeaallinna¨ıvveetrobth(eseexepertimheentapl aprrosceedutrreee iinliltuions”trtaastkewdasi nevaFluaitgedurbey r1ec)o.rding, for each reand to the aims of the study. sponse, the reaction time RT, in milliseconds, and the ually annotated the visual features associated with eaTchhe set of Tstihmeuli dwaespdeevnisdedebnyctyhe msturltuidcistcuiprlineariys realcecuvraacnytACi n,asotuher paerpcepntraogeaocfht,he correct answers. concept. The automatic annotation of visual propertteaimesof philobsoepchaerus,sneeuwroepsyachsosluogmistes atnhdacotmaputreerinfTohrecne, mforeneatch esubejecctt, mboathyRaTpa-nd AC were comscientists in the frame of a broader project aimed at in- bined in the Inverse E ciency Score (IES), by using associated with concepts is deferred to future workv:estiigtating bpotlhy thine rcolaesoefsviswuahl eloraed ibn ocotnhceaptswino- rd tahenfdormitusla dIEeSp=e(nRdT e·AnCt)(/s1)00(.oIErS is a metrics comcan be addressed either through a classical Informavtoilovend in infegroenvtiearlnanodrr(esfe)r)entaiarletaaskss.sociated withmovnilysuusaedl tfoeaagtgurergaetse.reFacotiornetixm-e and accuracy and to summarize them. The mean IES value was used as Extraction approach building on statistics, or in a mEoxpreerimenatamldpelseig,naanpdhprraosceedusruechPaartsici`pwanittsh bdleapecnkdensttvrairpiaebsle' ainsd eenxtepreedcintead2 ⇥ 2 repeated meawere asked to perform an inferential task “Naming by sures ANOVA with ‘target’ (two levels: ‘visual’ and ‘notsemantically-principled way. definition”. Dt ouriengvtohkeetasmkaensetnatelnciemwaasgperosnoiunncaed movriseuavl’)ivaindd ‘wdeafinyititonh’a(ntwoitlesveelsl:- ‘visual’ and ‘not
Di erent weighting schemes w~ = f ; ; g have bauenludes ntghievesnubinjeemctthseewnheetraesdpinthsaotnrkueesctnaednditnotoliissotvoeenlrattlyotitnohanemset(,itmahs- at vwiiessru,eal`p’b)eralfsaorwcmikteh'dinba-ysnuubdsjienc`gtssttfhraeciptDouersns.c'Pa)no,sttehsto.c comparisons tested in order to determine the features' contributioancctuorately anmd oasrefaostvaesrpoistssiblev, itshue atalrgleotawdordicsore-xpected to further grow if The scores obtained by the participants in the vithe visual load associated with a concept c, that resrueslptosnding twoetheaddedfinaitiocno, ousrindgina amtiecrdophtoenremco,n-as isnua`l wloiatdhquyesetliolnonwaireanwedre banlaalcyzked by using paired from computing sneenctteedd tthoroausrgethsrptiohpneseeEsb-'P.orxi.mMeAousodrfiettwooravyree,striw,mhuitclhihwewearVesLpalrseo-wouTl-dtest{s,retwcoutrasilievd.elTyw{o gcormopwarisiofns were performed for visual and not-visual targets, and for visual and notused to recorwdedaatadodneadccuaragcyoavnedrrneaocrtiotnetrimmes(.like `fvuisrualwdeifitnhitiyoensl.low and black
X Furthermore, at the end of the experimental session, VL(c; w~ ) = i = ( col + sha) + mot + siz :th(e2su)bjects wsterreiapdemsin')is.terWed aequtehsteionnnianiret:rtohdeyuhcaded a Tphaercaommpuettaetironal mtoodeclroensutltrsowlere analyzed by usi to rate on a t1he7 LciokenrttsrciableuthteioinntenosiftytohfetheavfiosuraelmeipnnegrtfopiroaminreeedddfTo-rfteevsiatssut,aultwraoendstaniilonetd-.vcisTauwasleotacorgmeptsarainsodnsfowrevrieload they perceived as related to each target and to each For the experimentation we set to 1:35, to 1:1 daenfindition. the corresponding terms are linskuaeldanidnnott-hviesuapl aderfisneititonrse.e by to :9: these assignments re ect the fact that color saubnTjhdeectfsacfatoctraoiarlsm,diensoigwdnhiiocfhethtrhe/esatvurisdguyauilnmlcolaueddneodtf btwroeothlwatiattrhigionetn- ( daCneodnrrvoeiltsauetiadolnlasoasbdemtqwuoeeesdntioaInEnnSda,ircaeo.mrWgpeuatalstoioenxpallormedo dtheel shape information is considered more important, inantdhedefinitioinnwaEs qmuanaiptuiolanted3.)T.he resulting four ex- existence of correlations between IES, the visual load perimental conditions were as follows: questionnaire and the computational model output by computation of VL. using linear regressions. For both the IES values and
To the ends of combining the contribution of conceVpVptresVyiwsuitahl TVgarerLag(tetcw—iin)Vgis=suflayl(inDgefiovnVeitrLiothn(ec(eim.)go.u,n‘ Ttahiifnesb9iisrdtchojef s.ttm.heeamqnuooefsdttiho(encni3a0;ircseujsb)cjoe_rcetssa,’wrreegspc(aoclncisue;lsac.tjeIdn) faorfiresatchmiotdemel, twhee
in a sentence s to the overall VL score for s, we adopt.e. d.eagle’); VL(ci) otherwiseu.sed the visual-load questionnaire scores as independent
theTfhoellocwominpguatdatdiiotniveofscthheemVa:LVsLc(osr)e =alPsoc2asccVoLu(nct)s.VfNhooVtrteVstisoufaItlhnTeaftorguherte—eleeNmxoenpn-tVes riosifumathleeDanenfitcnaieittniotisonins(e...g..,fiwTreha)e;s setvuhatseeridtaIboEthlSeea1tsco:o2dmpe.rppeuedntiacdtteinottnhaevlapdraiaarttbailceai)ps;ainin(ntd3se’a)ppeseenrcdfooenrndmtavmnaorciedaeb(lwl,eiwtthoe
the dependency structure of the input sentences. NTVhVe Non-Visual Target—Visual Definition (e.g., The predict the participants’ visual load evaluation (with the
nose of Pinocchio stretched when he said a . . . lie); questionnaire scores as independent variable).
syntactic structure of sentences is computed by the The stimuli in the dataset are pairs consisting of
Turin University Parser (TUP) in the dependency for- a de nition d and a target T (st = hd; T i), such as
mat
VL(d; w~ ) = VL(T; w~ ) =
Pc2d VL(c)
VL(T ): (4) (5)
The whole pipeline from the input parsing to computation of the VL for the considered stimulus has been implemented as a computer program; its main steps include the parsing of the stimulus, the extraction of the (lexicalized) concepts by exploiting the output of the morphological analysis, and the tree traversal of the dependency structure resulting from the parsing step. The morphological analyzer has been preliminarily fed with the whole set of stimuli, and its output has been annotated with the visual features and stored into a dictionary. At run time, the dictionary is accessed based on morphological information, then used to retrieve the values of the features associated with the concepts in the stimulus. The output obtained by the proposed model has been compared with the results obtained in a behavioral experimentation as described below.
Thirty healthy volunteers, native Italian speakers, (16
females and 14 males), 19 52 years of age (mean
sd = 25:7 5:1), were recruited for the experiment.
None of the subjects had a history of psychiatric or
neurological disorders. All participants gave their written
informed consent before participating in the
experimental procedure, which was approved by the ethical
committee of the University of Turin, in accordance with
the Declaration of Helsinki
Experimental design and procedure Participants were asked to perform an inferential task \Naming from de nition". During the task a sentence was pronounced and the subjects were instructed to listen to the stimulus given in the headphones and to overtly name, as accurately and as fast as possible, the target word corresponding to the de nition, using a microphone connected to a response box. Auditory stimuli were presented through the E-Prime software, which was also used to record data on accuracy and reaction times. Furthermore, at the end of the experimental session, the subjects were administered a questionnaire: they had to rate on a 1 7 Likert scale the intensity of the visual load they perceived as related to each target and to each de nition.
The factorial design of the study included two withinsubjects factors, in which the visual load of both target and de nition was manipulated. The resulting four experimental conditions were as follows: VV Visual Target|Visual De nition (e.g., `The bird of prey with great wings ying over the mountains is the : : : eagle'); VNV Visual Target|Non-Visual De nition (e.g., The hottest of the four elements of the ancients is : : : re); NVV Non-Visual Target|Visual De nition (e.g., The nose of Pinocchio stretched when he told a : : : lie); NVNV Non-Visual Target|Non-Visual De nition (e.g., The quality of people that easily solve di cult problems is said : : : intelligence).
For each condition, there were 48 sentences, 192 sentences overall. Each trial lasted about 30 minutes. The number of words (nouns and adjectives), their balancing across stimuli, and the (syntactic dependency) structure of the considered sentences were uniform within conditions, so that the most relevant variables were controlled. The same set of stimuli used for the human experiment was given in input to the system implementing the computational model.
The participants' performance in the \Naming from
definition" task was evaluated by recording, for each
response, the reaction time RT, in milliseconds, and the
accuracy AC, computed as the percentage of correct
answers. The answers were considered correct if the target
word was plausibly matched with the de nition. Then,
for each subject, both RT and AC were combined in
the Inverse E ciency Score (IES), by using the formula
IES = (RT=AC ) 100. IES is a metrics commonly used
to aggregate reaction time and accuracy, and to
summarize them
The scores obtained by the participants in the visual load questionnaire were analyzed by using unpaired Ttests, two tailed. Two comparisons were performed for visual and non-visual targets, and for visual and nonvisual de nitions. The computational model results were analyzed by using unpaired T-tests, two tailed. Two comparisons were performed for visual and non-visual targets and for visual and non-visual de nitions. Correlations between IES, computational model and visual load questionnaire. We also explored the existence of correlations between IES, the visual load questionnaire and the computational model output by using linear regressions. For both the IES values and the questionnaire scores, we computed for each item the mean of the 30 subjects' responses. In a rst model, we used the visual load questionnaire scores as independent variable to predict the participants' performance (with
IESas the dependent variable); in a second model, we used the computational data as independent variable to predict the participants' visual load evaluation (with the questionnaire scores as the independent variable). In order to verify the consistency of the correlation e ects, we also performed linear regressions where we controlled for three covariate variables: the number of words, their balancing across stimuli and the syntactic dependency structure.
The ANOVA showed a signi cant e ect of the withinsubject factors \target" (F1;29 = 14:4; p < 0:001), suggesting that the IES values were signi cantly lower in the visual than in the non-visual targets, and \de nition" (F1;29 = 32:78; p < 0:001), suggesting that the IES values were signi cantly lower in the visual than in the non-visual de nitions. This means that, for both the target and the de nition, the participants' performance was signi cantly faster and more accurate in the visual than in the non-visual condition. We also found a signi cant interaction \target*de nition" (F1;29 = 7:54; p = 0:01). Based on the Duncan post hoc comparison, we veri ed that this interaction was explained by the e ect of the visual de nitions of the visual targets (VV condition), in which the participants' performance was signi cantly faster and more accurate than in all the other conditions (VNV; NVV; NVNV), as shown in Figure 3.
By comparing the questionnaire scores for visual (mean sd = 5:69 0:55) and non-visual (mean sd = 4:73 0:71) de nitions we found a signi cant di erence (p < 0:001; unpaired T-test, two tailed). By comparing the questionnaire scores for visual (mean sd = 6:32 0:4) and non-visual (mean sd = 4:23 0:9) targets we found a signi cant di erence (p < 0:001). This suggest that our arbitrary categorization of each sentences within the four conditions was supported by the general agreement of the subjects. By comparing the computational model scores for visual (mean sd = 4:0 2:4) and non-visual (mean sd = 2:9 2:0) de nitions we found a signi cant di erence (p < 0:001; unpaired T-test, two tailed). By comparing the computational model scores for visual (mean sd = 2:53 1:29) and non-visual (mean sd = 0:26 0:64) targets we found a signi cant di erence (p < 0:001). This suggest that we were able to computationally model the visualload of both targets and descriptions, describing it as a linear combination of di erent low-level features: color, shape, motion and dimension.
Results correlations. By using the visual load questionnaire scores as independent variable we were able to signi cantly (R2 = 0:4; p < 0:001) predict the participants' performance (that is, their IES values), illustrated in Figure 4. This means that the higher the participants' visual score for a de nition, the better the participants' performance in giving the correct response (or, alternatively, the lower the IES value).
By using the computational data as independent variable we were able to signi cantly (R2 = 0:44; p < 0:001) predict the participants' visual load evaluation (their questionnaire scores), as shown in Figure 5. This means that a correlation exists between the computational prediction about the visual load of the de nitions and the participants visual load evaluation: the higher is the computational model result, the higher is the participants' visual score in the questionnaire. We also found that these e ects were still signi cant in the regression models where the number of words, their balancing across stimuli and the syntactic dependency structure was controlled for.
In the next future we plan to extend the representation
of the conceptual information by grounding the
conceptual representation on a hybrid representation composed
of conceptual spaces and ontologies
This work has been partly supported by the Project The Role of the Visual Imagery in Lexical Processing, grant TO-call03-2012-0046, funded by Universita degli Studi di Torino and Compagnia di San Paolo.