Name and Face Recall Cognitive Failure: Presenting a Short Literature Review and System Design Karolina Trajkovska1 , Maheshya Weerasinghe1,3 , Klen Čopič Pucihar1,2,4 and Matjaž Kljun1,4 1 University of Primorska, Faculty of Mathematics, Natural Sciences and Information Technologies, Koper, Slovenia 2 Faculty of Information Studies, Novo Mesto, Slovenia 3 University of Glasgow, Scotland, United Kingdom 4 Stellenbosch University, Department of Information Science, Stellenbosch, South Africa Abstract Cognitive lapses, such as remembering names and faces, are common minor disruptions that interfere with intended actions in everyday life. Existing research in psychology and cognitive science has explored various strategies, such as mnemonic devices and memory augmentation systems, to address these lapses. However, to the best of our knowledge, digital training systems tailored to enhance name and face recall do not exist, despite the technology offering various possibilities mentioned in the literature. To address this gap, this paper presents a literature review exploring existing training systems in order to propose a digital training system designed to enhance semantic memory with regard to the retrieval of face-name associations. In order to do this, we are planning to built a system to investigate the feasibility of the proposed approach. Keywords memory augmentation, cognitive lapses, face-name mnemonics 1. Introduction Memorising and recalling information are integral components of our cognition. However, we often encounter everyday cognitive lapses that manifest as moments when the mind encounters temporary obstructions. They frequently involve the elusive sensation of having information “on the tip of one’s tongue” yet being unable to retrieve it [1]. While modern technology has surpassed human capabilities in numerous tasks, the quest to develop technology that enhances human cognition performance remains an enduring challenge [2]. Psychology and cognitive science offer a rich repertoire of techniques proven to enhance cognitive performance, including the ability to store and retrieve information from memory. One method is the use of mnemonic devices, which entails establishing a connection between the information we try to memorise and a stimulus that is more likely to facilitate its retrieval. The link mnemonic method, which we will delve into later, is one such a strategy. However, Human-Computer Interaction Slovenia 2023, January 26, 2024, Maribor, Slovenia Envelope-Open trajkovskakarolina@gmail.com (K. Trajkovska); maheshya.weerasinghe@famnit.upr.si (M. Weerasinghe); klen.copic@famnit.upr.si (K. Čopič Pucihar); matjaz.kljun@upr.si (M. Kljun) Orcid 0000-0003-2691-601X (M. Weerasinghe); 0000-0002-7784-1356 (K. Čopič Pucihar); 0000-0002-6988-3046 (M. Kljun) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings many more mnemonic techniques exist such as the peg and loci method, the keyword mnemonic method, the major system among the most known, all of which have found practical applica- tions. For instance, a recent study used the keyword method with immersive technologies for vocabulary acquisition [3]. Despite, in the course of an ordinary day, individuals grapple with various cognitive chal- lenges, often unaware of the above mentioned strategies that could ease their cognitive burdens. Clinch and Mascolo [2] have explored domains where technology could intervene to offer assistance, highlighting the potential for leveraging evolving technologies to address these chal- lenges effectively. According to them, the second most common semantic memory lapse is the inability to recall someone’s name. This lapse can lead to self-embarrassment and inconvenience in various social situations. To the best of our knowledge, very few studies have addressed the problem of forgetting names and faces through an efficient digital tool. We hypothesise that such a tool could substantially aid individuals in mastering mnemonic systems. In this survey and position paper, we explore the digital and traditional methods for addressing everyday cognitive failures, elucidate the identified gap and propose a digital system designed to enhance semantic memory with regard to the retrieval of face-name associations. 2. Addressing Everyday Cognitive Lapses The existing literature predominantly emphasises interventions for individuals with medical memory impairments [2, 4]. However, it is also valuable to investigate cognitive processes in healthy populations, as cognitive lapses are pervasive across population [2]. Cognitive failures, as introduced by Broadbent et al. in 1982 [5], manifest as disruptions in intended actions, whether they involve mental or physical tasks, indicating a general vulner- ability to lapses in cognitive control. The Cognitive Failures Questionnaire (CFQ) and other instruments have been developed to assess an individual’s predisposition to cognitive failures, providing vital insights into the subjective experience of such lapses. The CFQ itself delves into errors stemming from perception, memory, and misdirected actions. Additional instruments are presented in the systematic review [6], including the Cognitive Slippage Scale [7] and the Prospective and Retrospective Memory Questionnaire [8] focusing on memory failures. Confidence in memory has also been assessed using self-reported diary studies as in [9, 2]. Carrigan et al.’s systematic review on cognitive failures in the everyday lives of healthy population [6] furthermore examines the construct of cognitive failures as well as the factors impacting the frequency of cognitive lapses’ occurrence. Broadbent et al. gave 3 aspects of the construct of cognitive failures (memory, perception, and action). However, later studies have revealed more internally-consistent aspects of the CFQ as memory, distractibility, blunders, and memory for names [10]. Finally, the systematic review categorises the possible influencing factors in two main groups: stable and variable (details visible in Table 1). According to the authors, stable factors have a stronger correlation with cognitive failures compared to the variable factors. Chan’s study [4] illuminates various memory failures, covering semantic, episodic, and procedural memory (retrospective) as well as prospective memory (future plans and inten- tions) [9]. These studies have highlighted the vulnerability of each of these memory systems to Table 1 Factors influencing cognitive failures based on Carrigan et al.’s systematic review [6]. Category Factor Influence on Cognitive Failures (CFs) Stable Genetics Fundamental biological influence on the experience of CF. Sex Women reported more CFs than men. Neuroticism Inappropriate worries may affect reports of CF. Schizotypy CFs are not only associated with schizotypy but may also play a predictive role in the development of specific negative schizotypal symptoms. Dissociative Linked to CFs via difficulty integrating information and processes tendencies and a higher frequency of involuntary autobiographical memories. Trait anxiety Emotional regulation issues affect cognitive capacity. Circadian ’Morning larks’ are more likely to experience CFs in the evening typology hours, while ’night owls’ throughout the day. Wakefulness Daytime sleepiness and distractibility are highly correlated to CFs. Variable Mood Negative mood states exacerbated cognitive failures in daily life. Stress Higher CFQ-scores during periods of stress, especially in high-stress environments. Environment CFs of individuals with good control capacity were more likely to be increased when faced with distracting environmental factors (e.g. chaotic surroundings). Activity Challenging or boring tasks might increase CFs. Age Associated with specific CFs, such as those involving the demand for recall. However, not all studies showed high correlation of age and CFs. Hormonal Biological factor affecting CFs, particularly evident in women. state Time of day Although it is frequently disregarded in research, this variable may have an impact. breakdowns, frequently referred to as everyday instances of memory failure [4, 9]. Prospective memory lapses, documented in two diary investigations [9, 2], were the most prevalent type of memory failure. Common incidents included forgetting object locations or failing to complete tasks like sending emails or making purchases, as addressed in [4]. Semantic memory lapses were ranked as the second most frequent. In [2], the primary reported semantic memory lapse was “failure to recall someone’s name”, documented in a three-week study with 14 participants, resulting in 82 documented semantic failures. Terry’s earlier diary study [11], involving 50 participants over several weeks, recorded 30 incidents of forgotten names. These studies indicate that remembering names is a common challenge for a lot of people and as such it could be addressed with technology. Clinch’s study [2] advocates addressing common memory failures with technological inter- ventions, aligning with Chan’s recent work [4]. Chan explores two main approaches: internal and external memory aids. Internal aids involve mental strategies, like mnemonics, which demand time and effort. External aids encompass tangible tools which are user-friendly but not always accessible. 2.1. Internal Aid: Mnemonic Devices A mnemonic device can be defined as a strategy for organising and/or encoding information through the creation and use of cognitive cuing structures [12]. The sole purpose of a mnemonic device is to enhance recall performance. They gained popularity in the 1970s after the general acceptability of cognitive processes as a legitimate area for research [13]. According to the classification from 1981 [12], they can be classified into 2 main groups: Peg type and Chain type. The sub-categories of the Peg Type are Method of Loci and Peg-word Mnemonic, while the sub-categories of the Chain type are Story Mnemonic and Link Mnemonic. The Method of Loci involves memorising specific locations (loci) in a building (the memory palace). The loci are then associated with the items to be remembered as visual images. The recall is achieved by mentally traversing these locations of the palace [14]. The Peg-word technique uses concrete objects as anchors for remembered images, often employing schemes like rhyming or encoding numbers to remember both the peg words and their order (”one is a bun, two is a shoe, three is a tree”). Both the Method of Loci and Peg-word Mnemonic yield similar recall results and can be used for remembering key pieces of information of a talk, lists of items or tasks. They provide a cognitive cue structure permanently stored in memory for associating and later recalling information. The Link Mnemonic offers an alternative to Peg-type mnemonics for remembering lists of items. It involves forming visual associations between each pair of consecutive words in the list, creating a chain of interconnected mental images. The story mnemonic involves incorporating each word in the list into a self-created story as they are presented. Variations of these methods also exist. In [1], Lorayne, a professional mnemonist, introduces a successful application of the Link Mnemonic, also known as the Link Method. One of the scenarios includes the recall of faces and names (Figure 1). The method involves several encoding phases: 1. Attention: Focus on both the name and the face. 2. Feature selection: Choose a distinctive feature of the face. 3. Keyword association: Select keywords that phonetically resemble the name. 4. Integration by mental imagery: Connect the keywords to the chosen facial feature by visualising the keywords interacting with the feature in an eccentric manner. One initially recognises the face’s distinctive feature. Looking at the distinctive characteristic activates mental imagery, which brings up the person’s name keywords. This process is not straightforward and necessitates practice before proficiency is achieved. Internal aids like this one are readily available but demand time and effort for effective use [4]. 2.2. External Aid: Memory Augmentation Systems Memory augmentation systems are technologically supported set of tools designed to enhance and improve human memory. Several such systems are a result of extensive research and innovation in the domain of Human-Computer Interaction (HCI) [4]. The systems are used to support different parts of the memory and help in behaviour change, learning, failing memories, achieving selective recall and in several other domains [15]. Figure 1: Link Mnemonic for Miss Van Nuys using the eyes as a distinctive feature. The keywords that phonetically resemble the name are vans and eyes. The interaction between the facial feature and the keywords involves vans coming out of the eyes. Courtesy of [1]. Memory augmentation systems use a wide range of approaches to support memory. Mainly they can be divided into training systems and assistance systems. The training systems are further divided into process-based and strategy-based. They stem from research in psychology that has developed and studied internal memory aids. The assistance systems are divided into reminder systems, life-logging systems and just-in-time systems [4]. They provide memory assistance to the user and have their roots in the idea of the “memory prosthesis”. Recent advancements in storage, machine learning, and wearable tech have sparked innova- tive ideas for enhancing human memory with assistance systems. While promising improved recall and freed cognitive resources, they also introduce a vulnerability: heavy reliance on these technologies may lead to dysfunction when it is absent. In the worst case, it can irreversibly alter cognition, resulting in poorer performance without it, potentially manifesting as memory loss or distortions in remembered events, skills, or knowledge [16]. Conversely, training systems not only avoid the risk of cognitive degradation but can also reduce age-related cognitive decline and lower the risk of cognitive diseases like Alzheimer’s [17, 4]. Digital strategy-based training systems make mnemonic strategies accessible on the-go and train users to apply these strategies. The Method of Loci has attracted the attention of quite a few researchers for creating a strategy-based training system as the Physical Loci [18], vMPeg [19] and NeverMind [20]. To our knowledge, there are no existing training system that instruct users in applying mnemonic strategies specifically to name and face recall. While various “memory prostheses” like Google Glass [21], Vimes [22], Pal [23], and “Haven’t We Met Before” [24] assist with face-name recall, they do not try to enhance human memory. Additionally, systems using the Loci method train semantic memory for facts, but the Link method is more suitable for encoding names [1]. Given that forgetting names and faces is a common semantic failure [2], addressing this directly is a promising research direction, especially since the literature shows positive results. For example, Lorayne’s face-name mnemonic, assessed in [25], significantly outperformed a control group when explained to students. In a recent evaluation by [26], the simplified face-name mnemonic surpassed three other groups in a between-subject design. Implementing a digital training system that would enhance the verbal-only instructions by showing the actual imagery projected upon the face of the individual should therefore enhance memorisation of names. Unlike traditional verbal instructions, a technology system can simu- late the application of the method in real-life situations, providing a more effective learning experience [26]. Further, we believe that our method will provide better results since recent research shows that showing external images enhances short and long-term memory [3]. 3. Designing Digital Training System Recognising the well-established advantages associated with strategic training systems [4, 16], we have identified the following gap in the current body of literature: Currently, there are no strategy-based digital training systems designed to enhance semantic memory with regard to the retrieval of face-name associations. However, the design space of a technologically supported training system is vast. Based on the idea from Figure 1 as well as the future directions proposed in [26] we propose to build a desktop training system based on the face-mnemonic strategy. Our first step towards designing the training system, is to implement a system that will augment faces on conversational videos with an overlaying animation providing interaction of the keyword with the (prominent feature of the) face (Figure 2d). Our key inquiry involves assessing the efficacy of such augmented videos in improving recall and the feasibility of implementing them. If the outcomes will be positive, our focus will shift towards developing the training system, which will integrate the augmenting system method and will train the users how to apply the mnemonic strategy independently. To achieve this, we propose to build a desktop system to support learners in remembering names. The system will have 4 different modes to augment the faces in real world videos with digital content (not in real time). Each mode will incorporate a different set of instructions as seen in Table 2 for remembering the presented names based on the simplified face-name mnemonic method [26]. Table 2 Instructions provided within each mode. Expert Mode Simplified Instructions provided 1 Remember as many faces as possible. – Transform the name into an imaginable object and imagine it 2 strategy interacting with the face. The name is already transformed into an imaginable object. strategy, 3 Imagine the object interacting with the face. keywords The name is already transformed into an imaginable object strategy, 4 and shown on the video. Observe its interaction with the keywords, face. visualisation Figure 2: Illustration of the proposed concept and the 4 different modes of operation: (a) no strategy, (b) mnemonic strategy described, (c) strategy described + keywords already provided, and (d) strategy described + keywords already provided + visualisation shown. 4. Conclusion and Future Work To our knowledge, memory training systems focusing on the Link method for name and face recall do not exist. Existing solutions, including AI-driven systems and wearables, often foster user dependency without strengthening human memory. To address this gap we propose to build a digital system designed to explore the feasibility to enhance semantic memory with regard to the retrieval of face-name associations. With this system we plan to conduct a user study to evaluate the proposed mnemonic training effectiveness. Finally, our aim is to build a memory enhancement training system, which would train users to use mnemonics on their own. Acknowledgments This research was funded by the Slovenian Research Agency, grant number P1-0383, P5-0433, IO-0035, J5-50155 and J7-50096. This work has also been supported by the research program CogniCom (0013103) at the University of Primorska. References [1] H. Lorayne, How to develop a super power memory: Your absolute, quintessential, all you wanted to know complete guide, Frederick Fell Publishers, 2000. [2] S. Clinch, C. Mascolo, Learning from our mistakes: Identifying opportunities for technology intervention against everyday cognitive failure, IEEE Pervasive Computing 17 (2018) 22–33. [3] M. Weerasinghe, V. Biener, J. Grubert, A. Quigley, A. Toniolo, K. Č. Pucihar, M. Kljun, Vocab- ulary: Learning vocabulary in ar supported by keyword visualisations, IEEE Transactions on Visualization and Computer Graphics 28 (2022) 3748–3758. [4] W. T. S. Chan, Augmenting Human Prospective Memory through Cognition-Aware Tech- nologies, Ph.D. thesis, ResearchSpace@ Auckland, 2022. [5] D. E. Broadbent, P. F. Cooper, P. FitzGerald, K. R. Parkes, The cognitive failures question- naire (cfq) and its correlates, British journal of clinical psychology 21 (1982) 1–16. [6] N. Carrigan, E. Barkus, A systematic review of cognitive failures in daily life: Healthy populations, Neuroscience & Biobehavioral Reviews 63 (2016) 29–42. [7] T. Miers, M. Raulin, Cognitive slippage scale, Measures for clinical practice: A sourcebook (1987) 125–127. [8] R. H. L. Geoff Smith, Sergiola Del Sala, E. A. Maylor, Prospective and retro- spective memory in normal ageing and dementia: A questionnaire study, Mem- ory 8 (2000) 311–321. URL: https://doi.org/10.1080/09658210050117735. doi:10.1080/ 09658210050117735 . arXiv:https://doi.org/10.1080/09658210050117735 , pMID: 11045239. [9] L. Ramos, E. Van Den Hoven, L. Miller, Designing for the other’hereafter’ when older adults remember about forgetting, in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2016, pp. 721–732. [10] S. J. K. J. Craig Wallace, C. J. Stanny, The cognitive failures questionnaire revisited: Dimensions and correlates, The Journal of General Psychology 129 (2002) 238–256. URL: https://doi.org/10.1080/00221300209602098. doi:10.1080/ 00221300209602098 . arXiv:https://doi.org/10.1080/00221300209602098 , pMID: 12224809. [11] W. S. Terry, Everyday forgetting: Data from a diary study, Psychological reports 62 (1988) 299–303. [12] F. S. Bellezza, Mnemonic devices: Classification, characteristics, and criteria, Review of Educational Research 51 (1981) 247–275. [13] K. L. Higbee, Recent research on visual mnemonics: Historical roots and educational fruits, Review of Educational Research 49 (1979) 611–629. URL: http://www.jstor.org/ stable/1169987. [14] F. Yates, The art or memory, 1966. [15] N. Davies, A. Friday, S. Clinch, C. Sas, M. Langheinrich, G. Ward, A. Schmidt, Security and privacy implications of pervasive memory augmentation, IEEE Pervasive Computing 14 (2015) 44–53. doi:10.1109/MPRV.2015.13 . [16] S. Clinch, O. Alghamdi, M. Steeds, Technology-induced human memory degradation, in: Creative Speculation on the Negative Effects of HCI Research. Workshop at The ACM CHI Conference on Human Factors in Computing Systems (CHI 2019)., 2019. [17] S. Umanath, J. Toglia, M. A. McDaniel, Prospective memory, Cognitive training: An overview of features and applications (2016) 81–91. [18] S. T. Perrault, E. Lecolinet, Y. P. Bourse, S. Zhao, Y. Guiard, Physical loci: Leveraging spatial, object and semantic memory for command selection, in: Proceedings of the 33rd annual acm conference on human factors in computing systems, 2015, pp. 299–308. [19] Y. Ikei, H. Ota, Spatial electronic mnemonics for augmentation of human memory, in: 2008 IEEE Virtual Reality Conference, IEEE, 2008, pp. 217–224. [20] O. Rosello, M. Exposito, P. Maes, Nevermind: Using augmented reality for memorization, in: Adjunct Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology, UIST ’16 Adjunct, Association for Computing Machinery, New York, NY, USA, 2016, p. 215–216. URL: https://doi.org/10.1145/2984751.2984776. doi:10.1145/ 2984751.2984776 . [21] R. Győrödi, C. Győrödi, G. Borha, M. Burtic, L. Pal, J. Ferenczi, Acquaintance reminder using google glass, in: 2015 13th International Conference on Engineering of Modern Electric Systems (EMES), IEEE, 2015, pp. 1–4. [22] C. Bermejo, T. Braud, J. Yang, S. Mirjafari, B. Shi, Y. Xiao, P. Hui, Vimes: A wearable memory assistance system for automatic information retrieval, in: Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 3191–3200. [23] M. Khan, G. Fernandes, U. Sarawgi, P. Rampey, P. Maes, Pal: A wearable platform for real-time, personalized and context-aware health and cognition support, arXiv preprint arXiv:1905.01352 (2019). [24] M. Iwamura, K. Kunze, Y. Kato, Y. Utsumi, K. Kise, Haven’t we met before? a realistic memory assistance system to remind you of the person in front of you, in: Proceedings of the 5th augmented human international conference, 2014, pp. 1–4. [25] P. E. Morris, S. Jones, P. Hampson, An imagery mnemonic for the learning of people’s names, British Journal of Psychology 69 (1978) 335–336. [26] Y. Gopi, E. Wilding, C. Madan, The impact of mnemonic strategies on name recall (2022).