The Use of PsychToolbox-3 for Tracking Bilingual Sentence Processing and Comprehension Judit Navracsics1[0000-0002-8147-3023] and Liubov Darzhinova2[0000-0003-2284-3140] 1Faculty of Modern Philology and Social Sciences, University of Pannonia, 1 Wartha Vince St, Veszprém 8200, Hungary 2Centre for Research on Linguistics and Language Studies, The Education University of Hong Kong, 10 Lo Ping Rd, Tai Po, New Territories, Hong Kong S.A.R. China navracsicsjudit@almos.uni-pannon.hu, liubov@s.eduhk.hk Abstract. The present article observes the continuous pursuit of experimental software in psycholinguistic research and the need to elaborate more on the prac- tical use of isolated software packages. On the example of a lab-based pro- gramme MATLAB with its experiment control library PsychToolbox-3, the arti- cle offers the results of the systematic review of literature conducted in the Sco- pus database. The results of the review suggest that there is a lack of information about PsychToolbox-3 in MATLAB in the literature. The article further provides a comprehensive overview of the two psycholinguistic experimental studies of (i) Hungarian-English bilinguals, and (ii) Russian-English bilinguals, undertaken using PsychToolbox-3 in MATLAB. The findings reveal that similar linguistic typology maintains a positive influence on sentence comprehension, and mor- phosyntactic information available at a word level may alleviate the overall pro- cessing load. The authors also conclude that language proficiency plays a signif- icant role in syntactic processing. Overall, the article illustrates that psycholin- guistic experiments built on MATLAB can be administered in future studies in- volving bilingual populations, whereas this instrumental groundwork can emerge as one of the technology-mediated language assessment tools. Keywords: PsychToolbox, MATLAB, Bilinguals, Experiment, Sentence Pro- cessing, Sentence Comprehension, Psycholinguistics. 1 Introduction In the field of psycholinguistics, there has always been a search for experimental soft- ware, which performs the tasks reaching a high level of precision in both spatial and temporal stimuli demonstration alongside a cost-effective collection of data from par- ticipants. In this context, one of the major steps that psycholinguists are required to take at the phase of planning and designing an experiment concerns choosing appropriate and relatively accessible apparatus and techniques of stimuli demonstration. For at least a decade, technological advances have been dictating the psycholinguists to favour for computer programmes out of all the interface spectrum to relatively easy and efficiently interact with research subjects and as a result, retrieve data for further analysis. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 There is plenty of software options to create and run psycholinguistic experi- ments [1]. The three groups of such software are pointed out: (i) allowing for the so- called “creative freedom”; (ii) giving “the advantage of the learning curve” at the cost of creative freedom; (iii) combining both the creative freedom and relatively user- friendly interface. The first group is represented by C, Java, MATLAB, Presentation, Python, R; the second – by E-Prime, Paradigm, SuperLab, and the third – by OpenSes- ame and PsychoPy. The one that is discussed in this paper is MATLAB, the research software tool from the first group that can fully adapt to researchers’ needs. It namely gives the ability to control for each step of an experiment, and any sort of stimuli can be generated includ- ing but not limited to text, pictures, and sounds [2]. Built on C and Java, MATLAB with its experiment control libraries allows researchers to write scripts for experiments directly via the software interface and run these scripts afterward. Kleiner et al. note that PsychToolbox is the most appropriate for psycholinguistic research out of all avail- able MATLAB experiment control libraries [3]. PsychToolbox supports functions permitting to link MATLAB interface up with the connected hardware so that investigators can account for the accuracy of both visual and auditory stimuli and at the end receive highly precise data output. Within Psych- Toolbox, one may also find useful the function to conduct eye-tracking experiments within EyeLink Toolbox or the function to command the EEG experiments within NetStation. To the best of our knowledge, not much psycholinguistic literature reports on the application of PsychToolbox in MATLAB. Only a limited number of studies (addressed in Paragraph 2) that can be found in the Scopus database concentrates on the use of PsychToolbox control library in psycholinguistic experiments. This paper is going to fill in the gap by elaborating on the psycholinguistic experimental studies of Hungarian- English and Russian-English bilinguals carried out with the use of PsychToolbox-3 in MATLAB and their implications for research and education. 2 Literature Review The below presented literature review is based on the search conducted in the Scopus database on September 21st, 2020. The search results exhibited 9 peer-reviewed re- search articles of the past ten years found with the use of the following Boolean opera- tors: TITLE-ABS-KEY ( psychtoolbox ) AND ( LIMIT-TO ( SUBJAREA , "NEUR" ) OR LIMIT-TO ( SUBJAREA , "PSYC" ) OR LIMIT-TO ( SUBJAREA , "ARTS" ) ) AND ( LIMIT-TO ( ACCESSTYPE(OA) ) ) AND ( LIMIT-TO ( PUBYEAR , 2020 ) OR LIMIT-TO ( PUBYEAR , 2018 ) OR LIMIT-TO ( PUBYEAR , 2017 ) OR LIMIT-TO ( PUBYEAR , 2013 ) OR LIMIT-TO ( PUBYEAR , 2012 ) OR LIMIT- TO ( PUBYEAR , 2011 ) ) After skimming the abstracts of the retrieved items, five of them were excluded from the list for further analysis due to their little or no association with psycholinguistics: 3 […] AND ( EXCLUDE ( PREFNAMEAUID , "Hallett, D.#36189855100" ) ) AND ( EXCLUDE ( PREFNAMEAUID , "Wijntjes, M.W.A.#18038930300" ) ) AND ( EXCLUDE ( PREFNAMEAUID , "Hung, C.P.#57215695276" ) ) AND ( EXCLUDE ( PREFNAMEAUID , "van Boxtel, J.J.A.#55898776100" ) ) AND ( EXCLUDE ( PREFNAMEAUID , "Nuutinen, M.#34873178300" ) ) As a result, we report on four peer-reviewed Scopus-indexed research articles, with an oldest-first order. Wilson, Tresilian, and Schlaghecken introduced the Masked Priming Toolbox (MPT) that is designed to carry out masked or unmasked priming experimental tri- als [9]. Masked priming is a classic psycholinguistic experiment for probing cognitive processing stages, which are not receptive to conscious control. For instance, masked priming can be performed for examining the mechanisms transmitting information about and yield reactions to emotional facial expressions. The MPT utilizes an array of devices for response collection, such as computer input devices (e.g., tablets) or sensors with its frequency response function (e.g., force transducers). The stimuli of the toolbox are operationalized across temporal (such as timing) and spatial constraints (such as location, orientation, and size). The MPT incorporates a prime, a target, and/or a mask, though is not confined to those only but can be modified to demonstrate stimuli appro- priate for semantic or emotional priming. After running a trial in the MPT, raw results are brought to Excel and MATLAB documents for further processing by a researcher. Sogo presented a new function for PsychToolbox entitled as Sgttoolbox [8]. It is a toolbox operating in order to command a cross-platform SimpleGazeTracker, an eye- tracking application of Gaze Parser programme. In connection with Simple- GazeTracker, Sgttoolbox affords a new approach to coordinate eye tracking process with scripts written in PsychToolbox. After examining this function, Sogo found only three minor issues in using Sgttoolbox along with SimpleGazeTracker: (i) absence of an on-line fixation-detection feature; (ii) latency of immediate obtainment of the latest gaze position; (iii) head movements of tested subjects can be delimited by means of a headrest. However, the three reported issues can be forestalled either by using an en- hanced graphics card or settled after conducting an experiment by giving specific in- structions in a command line. The study also revealed that time precision of eye-move- ment sampling frequency improves when done through one computer, whereas greater performance is achieved when presenting on one computer and recording via another one. Niehorster, Andersson, and Nyström proposed a package named Titta for the inte- gration of Tobii eye-tracking hardware used in conjunction with PsychToolbox in MATLAB and other software [7]. The authors elucidate on the five stages of Titta’s implementation in psycholinguistic eye-tracking experiments. For example, the first stage corresponds with the setup of experimental subjects by showing the display, which asks the subject to adjust the position of their head, i.e. move it in a way that it precisely fits in the displayed blue circle. The next one requests to proceed with cali- bration and validation of the gaze positioning with each eye separately by presenting numerical values at the end of the stage. The subsequent stages are the actual real-time data streams, synchronization, and file saving. 4 Bridges et al. juxtaposed a number of existing experimental software psycholinguis- tics makes use of to introduce stimuli to subjects and receive output data with their responses and response latency times [4]. The authors tested an array of experimental software applications, among which PsychToolbox in MATLAB was too. Tested on different platforms such as Ubuntu, macOS, and Windows 10, PsychToolbox showed high-quality results in giving temporal data with great precision in a series of studies using audio, visual, and response rates. The study concludes that PsychToolbox in MATLAB is proved to demonstrate the same accuracy in measuring response latencies as other packages as E-Prime, Presentation, and PsychoPy. It is bound to note that the reviewed Scopus-indexed studies are largely limited to the description of the isolated functions of PsychToolbox and do not extend any illus- trations of actual psycholinguistic studies with the use of this MATLAB experiment control library. Nevertheless, the analyzed research articles draw attention to the mag- nitude of PsychToolbox in MATLAB over other existing software packages by illumi- nating its high-level of precision in the context of temporal and spatial measurements as well as operational connectivity with the hardware used for experimental studies with eye-tracking or EEG. Surprisingly, the contemporary peer-reviewed research us- ing PsychToolbox, based on our Scopus database search, remains underrepresented in the current-day psycholinguistic literature, and necessitates further deliberation. We see a reasonable explanation for this state of affairs. First, the use of an open- access and open-code PsychToolbox entails a purchased MATLAB license or an insti- tutional subscription. Second, it requires researchers to have at least novice program- ming language skills to operate within the software such as to make modifications in the scripts of their experiments. Third, to our knowledge, there is no available archive of PsychToolbox scripts to allow for obtaining a skeletal framework in establishing a new experimental study carried out with the use of PsychToolbox in MATLAB. 3 The present study Two MATLAB-based experiments that are reported in this paper examined written lan- guage processing at the sentence level in the two bilingual groups: (i) Hungarian-Eng- lish (Group 1), (ii) Russian-English (Group 2). The experiment with Group 1 was projected to observe more accurate and speedier syntactic processing of Hungarian sentences owing to early access to morphosyntactic features of the processed lexical items. It also projected to highlight that second lan- guage proficiency modulates accuracy and speed of semantic processing. The experiment with Group 2 had a slightly different focus and expected to indicate more rapid semantic processing of both Russian and English sentences as opposed to syntactic processing. It was also expected to see syntactic processing of both Russian and English sentences as more precise compared to semantic processing. 5 3.1 Method Participants. For both experiments, we recruited healthy adults with normal or cor- rected-to-normal vision and reportedly with no neurological disorders. 97 Hungarian-speaking users of English were employed for the first experiment (70 females, 27 males: Mage = 31.21). The subjects were right-handed, 34 of them were early and 63 late bilinguals. Reliant on the literature, early bilinguals are those who commenced their acquisition of English by the age of 11 [5]. For the second experiment, 21 Russian-speaking users of English were recruited (19 females and 2 males: Mage = 21.8). These subjects were right-handed (n = 19) and left-handed (n = 2), and all were early bilinguals. The accepted subjects of the two groups were self-reportedly with C1 and B2 levels of English, according to the Common European Framework of Reference for Lan- guages. Both experiments included the dominant number of females. All the subjects belonged to the university setting, i.e. were either students or professors. They all re- sided in their country of origin at the time of participation in our experiments. Materials and design. The material for Experiment 1 entailed the sentence set of Hun- garian and English, the languages which diverge typologically and genetically. Hungarian is a Finno-Ugric language, whereas English is an Indo-European lan- guage and takes its place in the West Germanic group. The key difference between the two languages is related to morphology. In Hungarian, word meanings are changed by inserting multiple endings or suffixes, and that is why it is recognized as agglutinative, whereas English makes use of prepositions. Remarkably, Hungarian is typologically similar to the majority of Turkic languages, which are also agglutinative in nature and phonologically distinct with their phenomenon of vowel harmony. The stimulus material for Experiment 2 is built on the two languages of the Indo- European language family, Russian and English, which vary in different aspects. For example, unlike Russian, English has a fairly fixed word order. The other crucial dif- ference is that both languages use different alphabetical writing scripts, i.e. Russian uses Cyrillic, whereas English uses Latin. For Experiment 1, 240 sentences were generated, among which were sixty English and sixty Hungarian correct sentences as well as sixty semantically incongruent and sixty syntactically incorrect sentences in English and Hungarian: Table 1. Examples of stimuli for Experiment 1 Hungarian semanti- Hungarian syntacti- English semanti- English syntacti- cally anomalous sen- cally violated sentence cally anomalous cally violated sen- tence sentence tence A labda szögletes. Kati becsukta az I understand what He deep dug a Translation: The ball ablakig. you sneeze hole is square. Translation: Kati has closed to the window. The material for Experiment 2 was analogous to the first one. 240 sentences were pro- duced, among which were sixty English and sixty Russian correct sentences as controls 6 along with sixty semantically incongruent and sixty syntactically anomalous sentences in Russian and English: Table 2. Examples of stimuli for Experiment 2 Russian semantically Russian syntactically English semanti- English syntacti- anomalous sentence violated sentence cally anomalous cally violated sen- sentence tence Eto sto konfet el- Pered stojali nami de- see Table 1 see Table 1 ektrichestva. vushki. Translation: It is a Translation: Before hundred candies of stood us girls. electricity. Instrument. The customizable script was programmed in MATLAB [6] for testing sentence comprehension in the two bilingual groups via Psychtoolbox-3 [3]. Namely, the two scripts were designed to run the behavioral experiments by displaying either Russian and English or Hungarian and English sentences that are completely correct in terms of semantics and syntax, as well as those which have certain anomalies in syntax and incongruencies in semantics. The proportional fonts with relatively small letter sizes, 28 Segoe UI for the Russian- English experiment and Arial 14 for the Hungarian-English experiment, were set with the aim to minimize gaze saccades in the course of stimuli demonstration. The scripts were set to record the response latency times, or sentence processing times (as we refer to these) and calculate acceptability judgments in the form of perfor- mance measures for each of the following categories: syntactically correct and incorrect Russian / Hungarian and English sentences; semantically incongruent and congruent Russian / Hungarian and English sentences. Each cell in the scripts were responsible for stimuli presentation and had to contain the true order of the sentences so that MATLAB could display them to the observers in a randomized fashion. The results of each experimental trial were transferred to log files containing the data about each iteration: the displayed categories, the subject response, the true response, and the reaction time. The log file also saved the demographics that the subjects were requested to input at the onset of the experiment such as initials (e.g., M. K.), age (e.g., 23) and assigned ID (e.g., 14) for further processing. Procedure. Data collection in both MATLAB-based experiments was split into three stages. Each subject was tested on an individual basis and in one go, and the overall time of each experimental trial was around forty minutes. At the first stage, the subjects were informed about the study from the information data sheet and signed their consent to participate in the experiment. During the second stage, all the subjects were invited to proceed to a quiet room, where they were seated in a comfortable chair in front of a computer with a viewing distance of approximately 50 centimetres. The subjects were taught with eight exemplary sentences so that they are aware of their task in the forthcoming experiment. Each subject was trained to push the left (incorrect) or right (correct) arrow of the keyboard in order to give responses revealing their acceptability judgments per each sentence. 7 At the third stage, the subjects took part in the actual experiment. 240 stimulus sen- tences were evenly dispersed between eight sections and randomized for each trial so that the subjects could not become accustomed to merely one sentence type, but at the same time, each subject could view all sentence types. Each time the subjects pushed one of the keyboard arrows to indicate their response (← or →), they were moving to read the next sentence and give their sentence acceptability judgments again by pressing on a keyboard arrow. Each sentence was presented in black letters on a white backdrop for 5000 ms and adjusted to the center of the screen. Each sentence was displayed after the emergence of a red asterisk (*) being as a fixating point. As soon as 5000 ms were over at a given time (exposure time), the screen was showing a fixating point for 2000 ms prior to a new sentence (fixation time). The data containing response latency times and accepta- bility judgments, which were dependent variables of both experiments, were saved in the background for further analysis. Fig. 1. The summarization of the experimental procedure 4 Results 4.1 Response latency time data Based on the response latency time data from the Hungarian-English experiment, the reaction times in the processing of correct Hungarian sentences were the shortest among all (MPT = 2.14 s). Correct English sentences were processed faster than their semanti- cally and syntactically incorrect counterparts (MPT = 2.63 s). The longest response la- tencies were exhibited at reading English semantically incorrect sentences (MPT = 2.8 s). Reliant on the numerical data from the Russian-English experiment, we observe pro- cessing semantically correct Russian sentences processing time as the shortest among all (MPT = 1.44 s). The next shortest processing is shown for syntactically correct Rus- sian sentences (MPT = 1.54 s). Semantically incorrect Russian sentences were read a 8 little longer than their correct counterparts (MPT = 1.57 s), whereas processing syntac- tically incorrect Russian sentences exhibited the longest processing load (MPT = 1.59 s). Semantically incorrect English sentences were the fastest in processing (MPT = 2.03 s), followed by syntactically violated English sentences (MPT = 2.04 s) and semantically correct English (MPT = 2.06 s). The longest among all was the processing of syntacti- cally correct English sentences (MPT = 2.09 s). 4.2 Acceptability judgement data As of the results of accuracies in the Hungarian-English experiment, the best were which the subjects showed in the acceptability judgments about the sentences in their first language: (i) correct sentences (MA = 92.63%), (ii) syntactically incorrect (MA = 90.93%), (iii) semantically incorrect (MA = 87.35%). The next results were displayed in processing the correct English sentences (MA = 80.32%) and syntactically incorrect sentences (MA = 75.80%). Considering the results of accuracy in acceptability judgements retrieved in the Rus- sian-English experiment, the best were which the subjects exhibited in the judgments about the sentences in their native language in the next order: (i) semantically correct sentences (MA = 93.81%), (ii) semantically incorrect sentences (MA = 93.65%), (iii) syntactically correct sentences (MA = 91.9%), and (iv) syntactically incorrect sentences (MA = 89.67%). The next results were those of the English sentences as follows: (i) syntactically violated sentences (MA = 84.44%), (ii) semantically correct sentences (MA = 77.94%), (iii) semantically violated sentences (MA = 71.90%), and (iv) syntactically correct sentences (MA = 71.59%). 5 Discussion and conclusion Based on the findings of the Hungarian-English experiment, qualitative and quantita- tive differences were found in the course of processing of the Hungarian and English sentences. The subjects processed Hungarian sentences faster and could judge their ac- ceptability with better precision. We rationalize this by that the morphosyntactic con- straints inherent in the Hungarian language alleviate the processing load and there is no significant difference among reading times of correct and incorrect structures. Semantic processing was found to be slower than syntactic in both languages. Hungarian-English bilinguals could classify the target language correct structures more accurately than the incorrect ones. As of the results of response latency time data in the Russian-English experiment, we suggest that the semantic type of written language processing at the sentence level in both languages is roughly at the same level as the syntactic type. Based on the results of acceptability judgment data, we see the general trend that the semantic processing at the sentence level is more accurate than the syntactic type. When comparing the findings of both experiments, syntactic processing of English sentences is found to be more precise in the group of Hungarian-English bilinguals as 9 compared to processing of the same sentences in the group of Russian-English bilin- guals. This result may imply that linguistic typology has a positive effect on sentence comprehension. Also, we may generalize the results of both experiments and put for- ward that language proficiency level modulates syntactic processing: the higher the level of language proficiency, the faster and better understanding of the target language structures, and vice versa. To sum up, the current study serves as an attempt to improve present psycholinguis- tic experimental toolkits used to test bilingual individuals and fills the gap in the area of investigating language processing. The other potential implications of this study in- clude the usefulness of the described tool for testing bilingual populations and its ad- vocated inclusion to the pool of technology-mediated language assessment tools. As a follow-up study, our research team may conduct further studies of the bilingual groups speaking varied languages with tracking of their eye movements during reading to see the parsing patterns corresponding to each task. References 1. da Motta Sampaio, T. O.: The choice of software and hardware in psycholinguistics: review and opinion. Revista de Estudos da Linguagem, 25(3), 971–1010 (2017). 2. Borgo, M., Soranzo, A., Grassi, M.: MATLAB for Psychologists. Springer Science & Busi- ness Media (2012). 3. Kleiner, M., Brainard, D., Pelli, D., Ingling, A., Murray, R., Broussard, C.: What's new in PsychToolbox-3. Perception, 36(14), 1–16 (2007). 4. Bridges, D., Pitiot, A., MacAskill, M. R., Peirce, J. W.: The timing mega-study: comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414 (2020). 5. Kim, K. H., Relkin, N. R., Lee, K. M., Hirsch, J.: Distinct cortical areas associated with native and second languages. Nature, 388(6638), 171–174 (1997). 6. MATLAB.: version 2014b. Natick, Massachusetts: The MathWorks Inc. (2014). 7. Niehorster, D. C., Santini, T., Hessels, R. S., Hooge, I. T., Kasneci, E., Nyström, M.: The impact of slippage on the data quality of head-worn eye trackers. Behavior Research Meth- ods, 1–21 (2020). 8. Sogo, H.: Sgttoolbox: Utility for controlling SimpleGazeTracker from Psychtoolbox. Be- havior research methods, 49(4), 1323–1332 (2017). 9. Wilson, A. D., Tresilian, J., Schlaghecken, F.: The masked priming toolbox: an open-source MATLAB toolbox for masked priming researchers. Behavior research methods, 43(1), 210– 214 (2011).