The E-TAN ANALYST project: new horizons on E-BTT data analysis Antonietta Argiuoloa, Orazio Miglinoa and Michela Ponticorvoa a Natural and Artificial Cognition Laboratory, Department of Humanistic Studies, University of Naples, Via Porta di Massa, 1, Naples, Italy Abstract Psychological assessment through digital tool has transformed the way psychologist use and explain data. Among neglect assessment tool, an example is given by the E-BTT, a technology enhanced version of an ecological task, the Baking Tray Task. Thanks to an artificial vision module, it is possible to detect the spatial coordinates of the sixteen disks placed and their temporal order, so that the whole sequence can be recreated. In this paper, a focus on the new suggestions of data analysis for this data is provided. Then, a new tool will be proposed: an analyst able to automatically compute spatial and quality indexes. E-TAN ANALYST would be a boost to spatio-temporal sequences analysis, allowing ipsative and normative comparisons. Keywords 1 Baking Tray Task, E-TAN, E-BTT, data analysis, indexes 1. Introduction In the last decades, the increasing application of digital tools to psychological assessment has stimulated a shift of perspective for psychologists. Technological enhancement can be very useful in every step of cognitive evaluation, from automated scoring to online administration platform. Thus, it unveils new possibilities of data coding and analysis. As for the assessment of spatial neglect, some widely used tests witnessed the transition from paper- and-pencil to digital tools [1]. Scoring procedures originally had the scope to evaluate how the two halves of space had been explored: since unilateral neglect is characterized by a deficit in directing spatial attention, an asymmetry between detected stimuli on the left vs detected stimuli on the right could be a hint of neglect. Thanks to technological improvements applied to testing, like tangible interfaces or neural networks, new kinds of data and information are available, such as coordinates and the timestamp of each mark, as well as the trajectory that the patient’s hand has followed. To give an example of digital administration of a well-known task, it can be cited the developing of an open-source software for the administration of the cancellation task: CancellationTools [2]. Generally, cancellation tasks ask patients to mark target stimuli, amidst distractors; therefore they are usually addressed as a measure of multitarget visual search. Moreover, the location of omissions and the ratio between left and right cancelled items can be used as neglect indexes. Dalmaijer and colleagues [2] developed an open-source software which allowed to administer cancellation tasks and to analyse the collected data. In their package, the graphical user interface (GUI) was very simple to use and allowed to customize several parameters of a Landolt C cancellation task [3]. The data collection included time and coordinates for each mark; the software could use touchscreen or mouse input. The user can change the number of targets and distractors, the background colour and Proceedings of the Third Symposium on Psychology-Based Technologies (PSYCHOBIT2021), October 4–5, 2021, Naples, Italy EMAIL: antonietta.argiuolo@unina.it (A. 1); orazio.miglino@unina.it (A. 2); michela.ponticorvo@unina.it (A. 3) ORCID: 0000-0002-5284-5115 (A. 1); 0000-0002-7331-6175 (A. 2); 0000-0003-2451-9539 (A. 3) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) the visibility of the cancellation marks; moreover, each session and customized task can be saved and reused later. It is also possible to import tasks as image file. As regards the data analysis, the author included several quantitative measures from cancellation literature and completed them with others they have developed. These measures can be divided in three categories: biases in spatial attention (omissions, revisits, centre of cancellation, first marking), measure of search organization (standardized inter-cancellation distance, intersection rate, best R, standardized angle) and of general performance (timing, search speed, quality of search). For the explanations and calculation algorithms of each index, please refer to Dalmaijer and colleagues. What was important to underline was how the transition from analogical to digital allowed the collection of complex data and thus stimulated new ideas on how to use them. 2. Baking Tray Task and E-BTT Another example we would focus on in this paper is the Baking Tray Task (BTT) [4]. BTT is a neuropsychological task developed as a quick and simple test for the assessment of neglect. It consisted in placing sixteen 3.5 cm blocks on a 75x100 cm tray, as evenly as possible, as they were buns to be baked in an oven. Similarly to cancellation, the number of block in each half of space was counted: a configuration more unbalanced of 7/9 blocks were considered a sign of neglect. But unlike other classical tests like cancellation, the BTT referred to a typical everyday situation, such as baking buns, which made of it an ecological task. Ecological validity refers to result generalization to other situations: in other words, if the deficit or the behaviour observed in the “artificial” condition showed itself the same way in “natural” conditions. Indeed, it is common to notice dissociations between the performance on neuropsychological tests and the real functional recovery of patients with neglect [5, 6], from which derives the need to support the classical tests’ score with ecological tasks. BTT is easy to administer and proved to be sensitive in diagnosing neglect, especially when combined with other batteries [7]. It includes an empty space to be filled with “buns”, so there are no correct positions or configurations: the only rule is to spread them and not to move them once placed. As a consequence, the BTT seems to be unaffected by practice effects [4] or by demographics like age, education and gender [8]. In recent years, BTT was enhanced thanks to a technological improvement by Cerrato and colleagues [1, 9, 10]. Thanks to an augmented reality system and a platform connected with an online database, it was possible to improve the spatial data collection and data analysis. In its latest version, the prototype was called E-TOKEN and was a hardware-software platform for the evaluation of visuospatial abilities in peri-personal space. Although its original purpose was to administer BTT in an enhanced way, E- TAN potentially allows the design and administration of various types of tasks related to the assessment of visuospatial abilities. Therefore, the hardware part is composed by: • Modular 60x45 cm wooden frame. For practical reasons, a reduced area was preferred, considering that also Tham and Tégner used a smaller tray [4]. • Wooden squares for ArUco tags [11] that would be placed in the four corners of the frame. • Sixteen wooden 4 cm disks, each with its own ArUco tag; disks replaced the cubes in the final version to increase the accuracy of object detection by the software, which is more reliable with flat objects. Thus, the term “cookies” was used in place of “buns”, to fit this change. • Logitech C930e webcam, which was placed above the table, adjusted with a metallic arm in order to best focus the frame edges. ArUco marker tags [11] were adopted as a tool to detect tangible interfaces. They were realized by a research group of Cordoba University and were characterized by black squares out of a white binary matrix. The white little squares contained the tag id, while the black edge allows the external detection. The detection involving the ArUco markers will return the position (as coordinates) and the id of each recognized tag. An artificial vision module was then integrated [12] so that the platform software, E- TAN, could detect the position of the disks and the frame’s corners. Summing up, through E-TAN it is possible to administer a new version of BTT (from now on, E- BTT), using the same procedure of Tham and Tégner [4], asking the participants or the patients to spread the disks “as evenly as possible, as they were cookies to bake in an oven”. The only rule was to use one disk at time with one hand and to avoid moving already placed disks. The camera above the table detected the tags and coordinates were calculated by the software. Obviously, the frame middle was aligned with the sagittal plane of the user. Another innovation was represented by the online database which the software is synchronized to. This could allow the storage of potentially a great amount of data from different research teams. Once created an account, the user can download the .csv file containing all E-BTT and other tasks information connected to that account. The data collected are then: • Sixteen pairs of coordinates (in pixel); • The sequence, frame by frame, of each “step”, that is when the software correctly detected a disk; • The timestamp of each disk. The digital enhancement of the BTT allows now to carry out a more in-depth performance evaluation, not only counting the number of disks in one half or in the other, but also recreating the user’s temporal sequence. In other words, it is now possible to study how the user has processed the available space and which strategy (if any) has been used to complete the task. Figure 1: Some example figures of the apparatus. From top left, the Logitech camera, the screen from E-TAN platform, the disks with their ArUco tags, the available area (as the participant see it after completing the task). 2.1. Data analysis of the E-BTT data The number of studies analysing the new E-BTT data is still limited, as the final version is very recent [9]. In this paragraph, a short list of them is offered. As in the original BTT, a first, simple analysis Cerrato and colleagues performed was to count how many disks were placed on the left and how many disks on right part of the space [9], in a sample of right brain damage patients. Within the same study, they also applied the formula from Facchin [8] of the Laterality Bias. This bias gives the idea, in percentage, of the right-left asymmetry. A cut-off of - 12.5% (rightward bias) and +18.7% (leftward bias) was used to discriminate neglect patients. Moreover, the area available can be divided in quadrants, in order to study how the exploration started and ended. The quadrant analysis through Pearson’s Chi Square was carried out on the first and the last disk’s quadrant [12]. As for the beginning of disks placement, a preference for the upper part of the space was highlighted, while the sixteenth disk was usually placed in the bottom right quadrant. Similarly, Somma [13] and colleagues found out the trend to place the first disk in the upper left quadrant. Moreover, this study proposed to study pseudoneglect with the E-BTT. Pseudoneglect is the tendency to start spatial exploration slightly from the left [14] in several spatial task, like line bisection. This bias was found also in E-BTT [13], analysing the first disk’s X coordinate and the mean of all disks’ X coordinates. E-BTT gives the possibility to recreate the disks’ sequence, as said before. In one study was then performed a trajectory clustering analysis to find homogeneous clusters of strategies [15], in a sample of healthy undergraduate students. Three distinct groups emerged: a group in which subjects followed no clear plan; a second group which started from the right side and then went to the left; a third group of participants who tended to start from the left side. Using the sequence described by the disks’ positions, it was also possible to analyse the portion of the convex hull area delimitated by them [9]. In a sample of right stroke patients, this index proved to be sensitive to detect and discriminate patients from healthy subjects. More recently, the spatial sequences resulting from the E-BTT were analysed through Euclidean Distances, calculated in two different ways [16]: between two different configurations (BD) and within the same configuration (WD). While BD proved to be useful to detect different groups of patterns, WD discriminated between organized and disorganized configurations. 3. Future directions: E-TAN ANALYST As previously mentioned, there has been a gradual transition from pen and paper to digital in the administration and personalization of cancellation tasks. Again, this provided scholars with new and richer information, like for example, reaction times. Starting from the coordinates of each sign and considering the path that goes from the first to the last of these marks, it was therefore possible to develop some indexes [17, 18]. These studies focused on visual search and its organization in patient with right hemisphere damage. Some author related a disorganized visual search to neglect deficits, but the issue is still controversial [17]. The use of these visual search organization indexes could be an interesting inspiration for the E- BTT data. Despite the differences with the cancellation task, the result of the E-BTT task consists of a series of coordinates too. This would allow us to hypothesize that these indexes used in literature can also be applied to the case of E-BTT. So far, spatiality (in particular laterality) was the only dimension to be investigated, through the analysis of the right-left asymmetry or the pseudoneglect. This was of course due to the original goal of the BTT task, the assessment of unilateral neglect. However, extending this horizon also to healthy subjects with the aim of studying neurotypical spatial cognition, it would be interesting to pay attention to this new construct as well. Our goal was initially to look for visual search indexes in cancellation studies. These measures were brought together with the ones already used and some new calculations, inspired by the new information available. This work led to a long list of old and new indexes to be applied in future E-BTT research. On this list, a first classification could be made between spatial indexes and quality indexes. The spatial indexes summarized where the disks have been arranged and are in turn divided into laterality (if calculated on the X coordinates) or verticality (calculated on the Y coordinates). The quality indexes, on the other hand, inform on how the disks were placed and therefore the organization of the spatial sequence. A short sample of these indexes is summarized in Table 1. These measures would be very helpful in spreading a new light on spatial cognition studies. Not to mention that ensuring cut-offs that distinguishes between patients and healthy subjects would support clinician’s work. Table 1 Examples of indexes divided in types. Name Type What measures Authors References Quadrant Spatial 1x4 Chi Square on Cerrato et al., [12, 13] analysis2 disks placement 2020 frequencies Somma et al., 2020 2 The quadrant analysis cannot be strictly considered an index. Nonetheless, it was included as an important measure of disks placement tendency. First X Laterality First disk’s X Somma et al., [13] coordinate 2020 First Y Verticality3 First disk’s Y None coordinate Total Area Quality Proportion of space Cerrato et al., [9] occupied by the 2020 convex hull delimited by the disks 3.1. E-TAN ANALYST Starting from these indexes, it is possible to imagine a digital tool that uses the cloud database and calculate them. This paragraph, therefore, will present the proposal for the design and then implementation of an application for the analysis of spatial and temporal data of the E-BTT: E-TAN ANALYST. This need arises from the evidence that the calculation of the spatial and quality indexes mentioned above requires a considerable workload. It would certainly be a plus to have a tool capable of automatically calculating each index. Furthermore, in order to develop the potential of the database, that is to collect a large number of different sessions - potentially from different research groups - it will also be possible to develop a data dashboard able to show the distributions of the indexes as a function of the variables of interest. In figure 2 we show a general overview of the software system. Furthermore, E-TAN ANALYST means to be a data analysis app designed in order to have the elaborations of the individual sessions available, providing normative and ipsative assessment. Thanks to an authentication system, it will be possible to access one’s analysis profile, or to create an account. The analysis can be carried out both on local data, uploading them from time to time; or by synchronizing the account with the cloud database. All sessions recorded in E-TAN will then be available, obviously divided by Id and type of task. At the time of the analysis, the user will be asked what type of processing they prefer (plot, spatial indexes and quality indexes) and if they prefer to download or view it in the main dashboard. Here it will be possible to have an exhaustive summary of the measures for the specific session, as well as to make ipsative comparisons, with the past sessions of the same patient, or normative, comparing them with the aggregate data of the online database. The application will preferably be developed with STELT, a software platform created by SMARTED s.r.l. [19]. STELT (Smart Technologies to Enhance Learning and Teaching) responds to the need for providing support to psychologists, educators, trainers and teachers. It is a very versatile platform, able to integrate with other programming languages thanks to its set of libraries. STELT has already been used with various learning applications, thanks above all to the possibility of creating an environment for gamification of tasks and the possibility of integrating the implementation of augmented reality systems with tangible interfaces. Summing up, the prototype of E-TAN ANALYST could represent the possibility to establish a standardized procedure of data treatment and analysis for the E-BTT task. 3 Verticality indexes are specular to laterality ones. Note that there is no reference about them. This is because verticality has not been explored in these tasks, yet. Figure 2: E-TAN ANALYST architecture 4. Conclusion In this paper, new possibilities of data analysis for the spatio-temporal data from E-BTT task were explored. Some indexes were already used in classical BTT, mostly laterality ones. We proposed to complement them with verticality measure, calculated in the same way. Moreover, inspired by the visual search studies, other kinds of indexes were considered: quality indexes. Further validation and reflections about the use of these indexes will be needed, though. In particular, the structure we proposed in this paper should be verified through a Principal Component Analysis (PCA). Future studies will aim to that goal. Once the use of these indexes would be well defined, they will be implemented in a software, E- TAN ANALYST, able to compute those measure automatically. This application would have the goal of analysing the E-TAN database data (or uploading them manually), giving scholars and clinician a standardized procedure of data analysis for this particular task. Having a standardized data analysis protocol available would make it possible to fully exploit the potential of E-BTT. As it was designed, the E-BTT uses tangible interfaces to study the arrangement of objects in space. Moreover, thanks to the application of some indexes such as those of quality, it would also be possible to study the representation of the peri-personal space and the construction of cognitive maps. These processes are obviously active and embodied in action, so that it was suggested that place cells activation depends not just on movement itself, but on the active role the animal (or the robot) has in the space [20]. Indeed, place cells were implied in building cognitive maps and their activation was selective to target area only when the subject was free to explore. Therefore, the selection of salient stimuli made by the agent is fundamental in these processes. Summing up, the use of digital tools in psychological assessment and data analysis may have implication in our current understanding of spatial cognition. 5. Acknowledgements The first author’s PhD scholarship has been supported by Programma Operativo Complementare Ricerca e Innovazione 2014-2020, funded by Italian Ministry for University and Research. This research was also funded by Programma per il Finanziamento della Ricerca di Ateneo (FRA), ASIA project. For the remarkable work done at NAC laboratory, authors would like to thank Antonio Cerrato. 6. References [1] A. Cerrato, M. Ponticorvo, Enhancing neuropsychological testing with gamification and tangible interfaces: The baking tray task. In: J. M. Ferrández, J. R. Álvarez-Sánchez, F. de la Paz López, J. Toledo Moreo, H. Adeli (eds) Biomedical Applications Based on Natural and Artificial Computing. IWINAC 2017. Lecture Notes in Computer Science, vol 10338. Springer, Cham. doi:10.1007/978-3-319-59773-7_16. [2] E. S. Dalmaijer, S. Van der Stigchel, T. C. W. Nijboer, T. H. W. Cornelissen, M. Husain, CancellationTools: All-in-one software for administration and analysis of cancellation tasks, Behav Res Methods, 47 (2014): 1065–1075. doi: 10.3758/s13428-014-0522-7. [3] A. D. Parton, P. A. Malhotra, P. Nachev, D. Ames, J. Ball, J. Chataway, M. Husain, Space re- exploration in hemispatial neglect, NeuroReport, 17 (2006), 833–836. doi: 10.1097/01.wnr.0000220130.86349.a7. [4] K. Tham, R. Tegnér, The baking tray task: a test of spatial neglect, Neuropsychol Rehabil 6 (1996) 19–26. [5] P. Azouvi, The ecological assessment of unilateral neglect, Ann Phys Rehabil Med 60 (2017) 186– 190. doi:10.1016/j.rehab.2015.12.005. [6] Y. Takamura, M. Imanishi, M. Osaka, S. Ohmatsu, T. Tominaga, K. Yamanaka, et al., Intentional gaze shift to neglected space: a compensatory strategy during recovery after unilateral spatial neglect, Brain 139 (2016) 2970–2982. [7] P. Appelros, G. M. Karlsson, A. Thorwalls, K. Tham, I. Nydevik, Unilateral neglect: Further validation of the Baking Tray Task, J Rehabil Med 36 (2004) 258–261. doi: 10.1080/16501970410029852. [8] A. Facchin, N. Beschin, A. Pisano, C. Reverberi, Normative data for distal line bisection and baking tray task, Neurol Sci 37(2016) 1531–1536. doi: 10.1007/s10072-016-2626-6. [9] A. Cerrato, D. Pacella, F. Palumbo, D. Beauvais, M. Ponticorvo, O. Miglino, P. Bartolomeo, E- TAN, a technology-enhanced platform with tangible objects for the assessment of visual neglect: A multiple single-case study, Neuropsychol Rehabil (2020). doi: 10.1080/09602011.2020.1762671. [10] C. Gentile, A. Cerrato, M. Ponticorvo, Using technology and tangible interfaces in a visuospatial cognition task: The case of the Baking Tray Task, in: O. Miglino, M. Ponticorvo, Proceedings of the First Symposium on Psychology-Based Technologies, Naples, Italy, 25-26 September, 2019, CEUR Workshop Proceedings, 2524. [11] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, M. J. Marín-Jiménez, Automatic generation and detection of highly reliable fiducial markers under occlusion, Pattern Recogn 47 (2014) 2280–2292. doi: 10.1016/j.patcog.2014.01.005. [12] A. Cerrato, M. Ponticorvo, O. Gigliotta, P. Bartolomeo, O. Miglino, Btt-scan: uno strumento per la valutazione della negligenza spaziale unilaterale [Btt-scan: a tool for evaluating unilateral spatial neglect], Sistemi intelligenti 31 (2019) 253–270. [13] F. Somma, A. Argiuolo, A. Cerrato, M. Ponticorvo, L. Mandolesi, O. Miglino, et al., Valutazione dello pseudoneglect mediante strumenti tangibili e digitali [Pseudoneglect evaluation using tangible and digital tools], Sistemi Intelligenti 32 (2020) 533–549. [14] D. Bowers, K. M. Heilman, Pseudoneglect: Effects of hemispace on a tactile line bisection task, Neuropsychologia. 18 (1980), 491–498. [15] F. Palumbo, A. Cerrato, M. Ponticorvo, O. Gigliotta, P. Bartolomeo, O. Miglino, Clustering of Behavioral Spatial Trajectories in Neuropsychological Assessment Analisi dei gruppi di traiettorie spaziali nella valutazione neuropsicologica, in: Proceedings of the SIS 2019-Smart Statistics for Smart Applications, pp. 463–470. [16] A. Argiuolo, F. Somma, D. Marocco, O. Gigliotta, P. Bartolomeo, O. Miglino, M. Ponticorvo, Analysis of spatial behaviour by the means of Euclidean Distances, To appear. [17] V. W. Mark, A. J. Woods, K. K. Ball, D. L. Roth, M. Mennemeier, Disorganized search on cancellation is not a consequence of neglect, Neurology 63 (2004) 78–84. doi: 10.1212/01.WNL.0000131947.08670.D4. [18] M. Warren, J. M. Moore, L. K. Vogtle, Search performance of healthy adults on cancellation tests, Am J Occup Ther 62 (2008) 588–594. doi:10.5014/ajot.62.5.588. [19] O. Miglino, A. Di Ferdinando, M. Schembri, M. Caretti, A. Rega, C. Ricci, STELT (Smart Technologies to Enhance Learning and Teaching): Una piattaforma per realizzare ambienti di realtà aumentata per apprendere, insegnare e giocare [STELT (Smart Technologies to Enhance Learning and Teaching): A platform for creating augmented reality environments for learning, teaching and playing], Sistemi Intelligenti 25 (2013) 397-404. [20] O. Miglino, M. Ponticorvo, P. Bartolomeo, Place cognition and active perception: a study with evolved robots, Connection Science 21 (2009), 3-14. doi: 10.1080/09540090802364769.