Cognitive impairment is an increasingly prevalent issue, particularly with the ageing population. Brief cognitive screening tests play a key role in early detection, and their progressive digitisation raises new methodological challenges. One such test, the SAGE (Self-Administered Gerocognitive Exam), which assesses areas such as orientation, language, and memory, has proven useful for the early detection of cognitive impairment. This paper presents a pilot study designed to compare the traditional and digital formats of a cognitive test based on a multimodal protocol. The protocol integrates objective performance scores, subjective assessments of user experience (UX) and mental workload, and emotional response analysis conducted in two sequential phases: the ifrst using facial expression analysis, and the second focusing exclusively on stress via electrodermal activity (EDA) sensors. The second phase was conducted in care home environments, where ethical constraints prohibited the use of cameras, highlighting the protocol's adaptability to real-world sensitive contexts. This paper contributes to understanding how test format influences not only cognitive outcomes but also UX and perceived mental workload, providing a replicable protocol for evaluating the digital transition of cognitive assessment tools.
Cognitive impairment is a growing challenge for global public health, driven by an ageing population
and aggravated by factors such as unwanted loneliness among older adults [
Among the tools available for this purpose, the Self-Administered Gerocognitive Exam (SAGE) [
The digitisation of the SAGE test has been motivated by the need to increase its accessibility,
particularly for vulnerable populations such as older adults with mobility limitations, those living in rural areas,
or individuals experiencing social isolation [
This paper presents a pilot study aimed at comparing the traditional and digital formats of a cognitive
test, taking SAGE as a representative case. The digital format used in this study corresponds to the
version developed in [
CEUR
ceur-ws.org assessments of user experience and mental workload, and emotional response analysis conducted in two sequential phases that correspond to diferent settings. In the first phase, emotional data were gathered via facial expression analysis in a lab setting. In the second phase, conducted in care home environments, emotional responses were monitored through electrodermal activity (EDA) sensors, due to restrictions on video recordings in such settings.
This pilot study highlights the adaptability of a multimodal evaluation to ethically sensitive realworld contexts, where traditional data collection methods may be restricted, ofering insights for the development of digital cognitive testing approaches that balance methodological rigour with user acceptability.
The remainder of this paper is organised as follows: Section 2 provides background and reviews related work on the digitisation of cognitive tests. Section 3 presents the methodology, including the study’s objective, variables and experimental procedure. Section 4 reports the initial results. Finally, Section 5 discusses some key aspects to be considered in future extended formal evaluations.
Several tools have been developed considering the importance of early detection and prevention.
For this study, the Self-Administered Gerocognitive Examination (SAGE) test will be brought into
focus. Its reliability and validity as a tool for early detection of dementia and Alzheimer’s disease are
well-documented through several studies [
This section critically discusses the similarities and diferences between the current study and related
literature in early detection and prevention of Alzheimer’s and dementia. In terms of the validity of the
SAGE test, [
Other experiments have validated the applicability of the test in broader contexts, demonstrating
that the SAGE test is feasible, practical, reliable, and efective in various settings and among diferent
population groups [
Finally, Scharre et al. [
This section describes the methodological approach of the pilot study. Given its exploratory nature, the study adopts a multi-phase research design in which each phase addresses complementary aspects of the overall goal through specific data collection techniques and settings. This design allowed the protocol to adapt to evolving constraints in both controlled and real-world environments.
The main objective of this study is structured using the Goal–Question–Metric template [
The study is guided by the following research questions and related variables. The dependent variables measured are detailed in Table 1.
• RQ1. How diferent is the cognitive performance between the traditional and digital
formats of the cognitive test?
This question explores whether the format of the cognitive test influences participants’
performance, which is crucial for assessing the comparability of both formats in the early detection
of mild cognitive impairment (MCI). Cognitive performance is measured using the test scores
(range: 0–22) obtained in both the traditional and digital formats of the SAGE test.
• RQ2. How does the test format (traditional vs. digital) afect the perceived workload?
This question aims to explore whether participants perceive diferent levels of workload
depending on the test format. In the study, perceived workload was measured using the NASA
TLX questionnaire [
The study was structured in two phases to evaluate cognitive performance, mental workload, and
emotional responses elicited by the SAGE test in both its traditional (paper-based) and digital formats.
In both phases, the digital format of the SAGE developed in [
The implementation of each phase was subject to ethical approvals and institutional permissions. Participants in both phases were required to meet the following inclusion criteria: no prior formal diagnosis of cognitive impairment, and provision of informed consent.
In Phase 1, conducted in a laboratory setting, the SAGE test was administered in both formats under controlled conditions, with a main focus on the evaluation of the digital version. This version was designed based on usability principles for older adults and aimed at reducing cognitive load, minimising errors, and ensuring an accessible experience through large buttons and typography, clear instructions, linear navigation, and a minimalist design with emotionally positive colours (i.e., Blue, White and Yellow).
Participants were selected through convenience sampling, based on their availability, proximity and voluntary participation. To assess emotional responses during the administration of the digital test, participants’ facial expressions were recorded using a camera and OBS Studio, a free video recording software configured specifically to capture close-up video in MP4 format. The videos were used as input in the Kopernica tool, which applies facial emotion analysis algorithms and generates CSV files with the corresponding results for interest, frustration, valence, and arousal percentages.
Immediately after the test, participants responded to the NASA TLX Workload questionnaire to assess perceived workload. The order of test format presentation was randomized across participants, and a one-week gap was maintained between the sessions for each format.
In Phase 2, carried out in a home care setting, participants were older adults residing in a local nursing home, which allowed for the inclusion of a more diverse population and testing under uncontrolled conditions, in contrast to Phase 1. Each participant completed both the digital and traditional versions of the test, with at least a 7-day interval between each administration to avoid short-term memory efects. In addition to cognitive scores, physiological data were collected during both tests to evaluate user experience.
For physiological data collection, the EmbracePlus smartwatch, a wearable device that records realtime electrodermal activity data through integrated electrodes that must be in contact with the skin to capture accurate information, was used. It operates with a sampling frequency between 1 and 4 Hz, within a range of 0.01 to 100 microsiemens (μS). To initiate monitoring, each participant was previously assigned a unique credential through Empatica’s Care Lab Portal, which generates a QR code used to pair the watch with a mobile phone via Bluetooth. Once paired, the device was placed on the non-dominant wrist of the participant. Raw sensor data collected from the wearable device were stored in AVRO format, a compact binary format for eficient data storage and fast access, and subsequently converted to CSV using a Python script provided by Empatica for further analysis. The NASA TLX Workload questionnaire was also administered after each format.
All participants provided informed consent prior to participation, in accordance with the guidelines approved by the relevant ethics committees. In the home care setting, consent from legal guardians or family members, when applicable, was managed directly by the facility staf. To protect participants’ privacy, video recording was restricted in Phase 2, and only non-intrusive physiological data were collected using a wearable device. Data from the EmbracePlus smartwatch was managed by Empatica under a research license valid for the duration of the project. For video-based emotional analysis in Phase 1, only numerical output from the tool was stored, and no raw video footage was retained. In line with the agreement established with the care facility, the research team also provided both individual and group-level summaries of test results, along with tailored cognitive health recommendations.
Five older adults (three individuals aged between 60 and 80 and two individuals over 80) completed the SAGE test in both formats.
Table 2-Phase 1 shows the SAGE test scores for participants in the first phase, comparing both the digital and traditional formats and including the diference in performance per participant. The scores showed high consistency between the traditional and digital formats for most participants. For example, P2, P3, and P4 obtained identical scores in both formats (19, 11, and 17, respectively), while the remaining two (P1 and P5) showed a small diference of two points in favour of the traditional version. These results, under controlled conditions, suggest that cognitive performance is generally consistent across formats, although limited digital experience may slightly influence outcomes in favour of the traditional version, likely due to lower familiarity with mobile devices, which is related to age—as in the case of P1, who is 91 years old.
Regarding mental workload (see values in Table 3-Phase 1), the NASA TLX Workload data for the digital version showed significant variations among participants. The average workload ranged from -6.0 to 7.2, with P4 reporting the highest workload (7.2) and P2 the lowest (-6.0). Participant P4 indicated a notably demand with the digital format, despite obtaining identical scores in both formats (17). In contrast, P2 reported negative values in most dimensions, suggesting a comfortable and undemanding experience.
In terms of emotional responses, the overall analysis of valence, arousal, and frustration (processed through Kopernica) in the digital version revealed that most participants are located in the lower-right quadrant of the afective circumplex, corresponding to a ”Calm” emotional state, characterized by positive valence and low arousal (see Figure 2). This suggests that these participants felt generally comfortable and relaxed while completing the test. An exception was observed in the case of Participant P2, who exhibited slightly negative valence and higher arousal, placing between ”Sad” and ”Upset” zones. According to Figure 3-a, this participant also reported the highest frustration percentage (34%).
These results suggest the digital format is comparable in performance but poses challenges for some participants, reflected in higher frustration and workload. The prevailing calm indicates good acceptance, but cases like P2 highlight the need for more intuitive interfaces to reduce frustration.
Seven older adults (five women, two men; ages: 90, 76, 75, 80, 70, 90, 80, respectively) completed the SAGE test in both formats in a more realistic setting. Table 2-Phase 2 shows the SAGE test scores for participants in this phase. The group showed a general trend of improvement in the digital format, where average scores increased from 13 (traditional) to 16 (digital). Regarding mental workload (see values in Table 3-Phase 2), some diferences between formats were also observed. In the digital format, average workload ranged from -7.7 to 4.5, with U6 showing that the task was demanding despite improvement in performance (14 to 18). U4, with the lowest average (-7.7), reported negative values across all dimensions, indicating a very comfortable experience. In the traditional format, averages were more homogeneous, ranging from -3.7 to 1.0. However, the overall trend indicates that while the digital format improved performance, it may introduce additional demands for some.
The Stress levels graphs, (see Figure 3-b), revealed that stress was generally lower in the digital format, with lower average peaks compared to the traditional version. These results indicate that in this group of individuals, overall cognitive performance improved in the digital format and stress was reduced, but variations in workload were introduced.
Table 4 summarises stress-related measures for each participant and test format in Phase 2. The table includes the percentage of time spent in high stress levels (≥4), average stress level, standard deviation, and the number of detected stress episodes based on EDA signals. When comparing formats, most participants exhibited slightly higher stress levels and more frequent stress episodes in the traditional version than in the digital one. This pattern is particularly evident in the percentage of time spent at stress levels 4 and 5, which tended to be greater under the paper-based condition. These diferences may be partially explained by the improvements made to the digital prototype following usability feedback in Phase 1. Additionally, the fixed font size and layout of the paper-based test could have introduced visual strain for some participants.
This pilot study tested a multimodal protocol in two distinct settings to compare cognitive performance, workload, and emotional responses across digital and traditional formats of a cognitive test. Several aspects of the protocol proved efective across both phases. Paper-based instruments—used for the traditional cognitive test and workload measurement—were reliable and well accepted in both contexts. The informed consent process was also simple, with most participants signing voluntarily without raising concerns.
In both settings, certain test items required verbal clarification during administration, and a thinkaloud protocol could have enriched feedback beyond written responses. Most participants completed both test formats, with two exceptions in Phase 2 due to contextual constraints.
Scheduling was more consistent in Phase 1, with randomised order and a one-week interval between formats. In Phase 2, scheduling varied due to participants’ availability or health issues, extending in some cases to three weeks or resulting in partial data.
Distinct challenges emerged in each phase. In Phase 1, one of the main dificulties involved managing and analysing video recordings used for emotion detection via Kopernica. The videos had to be processed ofline, and in several cases, facial detection accuracy was compromised due to posture. Therefore, emotion recognition based on facial expressions is only available for the digital format. A few participants also expressed concern about receiving feedback on their cognitive performance, which may reflect a general sensitivity to test results in this demographic.
In Phase 2, ethical restrictions within the home care setting strongly influenced data collection. Video recording was not allowed at all; instead, physiological data were gathered through the EmbracePlus wearable as an acceptable alternative. Although this method limited emotional data granularity, restricting it mainly to EDA-based stress levels, it was well tolerated. Figure 3-b shows that stress in digital format starts high but decreases and stabilises, while on paper it gradually increases. Table 4 confirms that most participants experienced more episodes and longer durations of elevated stress on paper, possibly due to usability improvements in the digital version and visual discomfort with the traditional format. Participants often reported being unaware of the device. Thus, using nonobstructive acquisition methods could be a preferable alternative in settings where administrators/users are reluctant to camera-based methods.
Participant recruitment in Phase 2 was coordinated by the facility, which limited researchers’ ability to select or balance participant profiles. Still, the core testing instruments were applied consistently across both phases, with variation only in the method of emotion measurement. These contextual and methodological diferences were documented and considered in interpreting results.
Overall, this paper contributes a protocol for conducting a pilot involving two diferent types of settings, which aimed at gauging diferences in cognitive performance and workload influenced by the implementation format of the SAGE tests. The pilot reported initial results, providing some encouraging preliminary results on whether the digital version may pose or not a serious entry barrier before running a more formal experiment, It also illustrated how multimodal measurements concerned with emotional responses can be used to complement the performance and workload measurements.
This pilot study highlights the importance of planning for granular physiological data collection, balanced test scheduling, and adaptive digital design when working with older adults. Despite contextual constraints, the proposed multimodal evaluation protocol proved adaptable to both controlled and real-world settings. In this regard, the consistent use of the core instruments across both phases, despite diferences in emotional measurement, supports the protocol’s scalability. However, ethical aspects such as informed consent, data privacy, and participant vulnerability must be adapted to each setting. Addressing them early is key to ensuring ethical and contextual suitability at scale.
These observations support the feasibility of replicating this multi-phase approach in similar studies evaluating digital transitions in cognitive testing. The protocol ofers a comprehensive framework to examine the impact of test format in elderly populations.
Authors acknowledge the support of the Galician Ministry of Culture, Education, Professional Training and University (grants ED431G2023/04 and ED431C2022/19). Supported also by Interreg VI-A SpainPortugal Program (POCTEP) 2021-2027 with grant 0144_TRANSFIRESAUDE_1_E, CNS2024-154915 by MCIN/AEI, the Erasmus Mundus Joint Master Degree program SE4GD-619839, and the ERDF.
During the preparation of this work, the author(s) used Grammarly for Grammar and spelling checks.