I See What You See! Towards Augmented Joint Visual Attention between Beginner and Instructor Surfers Tao Bi 1 1 University College London, 66-72 Gower Street, London, UK Abstract Wave reading and selection are vital, yet challenging, for beginner surfers during the process of learning to surf. Built on the author's auto-ethnographical diaries, this paper reflects the real- life challenge of noticing when and where to catch the best wave. More importantly, beginner surfers often struggle to follow the instructor surfers who are instructing them with references to a particular wave or specific sections of a wave. The author uses the joint visual attention theory to provide a tentative explanation for this observed phenomenon. This paper proposes a speculative design solution of using AR and gaze-tracking-based goggles to foster joint attention on waves between beginner and instructor surfer, allowing the former to follow the instructor's directions and the latter to understand where the beginner is looking. This paper primarily aims to facilitate discussions in the sports HCI community and hopes to provide a reference to surfing literature in which beginner surfers and their learning experience lack attention. Keywords 1 Surfing, Learning to Surf, Wave Reading and Selecting, Surf Coaching, Augmented Reality, Joint Visual Attention, Smart Goggles, Speculative Design, Auto-ethnography 1. Introduction Surfing has been one of the fastest-growing water sports [1] that offers spiritual and adventurous opportunities for being closely connected to the mother ocean. It offers not only fitness benefits but also mental wellness boost [2-6]. However, surfing remains to be one of the most challenging and slow- progress sports. Catching good waves consistently requires years of practice. Selecting the right wave and catching it at the right moment is critical to a successful surfing experience. Taking in-person surfing lessons from experienced surfers or professional surfing instructors is getting more popular to effectively gain knowledge about the ocean and waves [7]. However, problems and frustration are still frequently observed. For instance, beginner surfers often feel confused and struggle to follow an instructor surfer who is pointing to a particular wave section in the ocean. This not only causes beginner surfers’ frustration but also extends the learning curve for reading and catching waves. To further discuss such issues, the author reflects on auto-ethnographical diaries of his own surfing- learning experience as well as his observations of a local surf community over two years in Newquay, Cornwall, UK. In this paper, the author will first introduce the “Joint Visual Attention” theory originating from psychology and child development literature. This theory could provide explanations for how and why it is challenging for beginner surfers to develop shared and synchronized visual attention on target waves during their communication with instructors. In addition, the author will discuss several characteristics of the complex and dynamic ocean waves that potentially led to the challenge of accurately joining an instructor’s visual attention on a wave. The also paper proposes a speculative design solution - using AR goggles to help surfers achieve joint visual attention in real- time. The problem description and design speculations are supported by some explicative figures. NTSPORT’22: New Trends in HCI and Sports Workshop at MobileHCI’22, October 1, 2022 EMAIL: t.bi@ucl.ac.uk (A. 1) ORCID: 0000-0002-7406-8943 (A. 1) ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) As surfing has not been widely studied in the HCI community, this paper aims to gain attention and discussion from HCI researchers who are interested in sports and outdoor scenarios. Surfing literature has also been mainly focused on experienced surfing (e.g., surfing big waves [8, 9]) and on the understanding of its societal aspects [3, 6, 10, 11]. This paper hopes to contribute to a reference to literature on beginner surfers’ learning experiences [12]. Last but not least, the speculative design proposal of an AR porotype could also raise discussions and provide inspiration on how to reach joint visual attention in other outdoor sports (e.g., referring to a snow slope during a ski or snowboarding session, or referring to a mountain section during climbing or mountain biking). 2. What is joint visual attention? Joint visual attention means one person looks at where another is looking by following their pointing gaze or gesture. The pair aims to focus on the same object or event, to interact with each other. Joint attention can be achieved by using eye contact (looking at another’s eye gaze and face), gestures (pointing using index finger), or vocalisations (e.g., saying “look over there!”), giving matching reinforcement by verbal and behavioural confirmation [13]. Joint attention is bi-directional. It can also occur by one responding to another’s attention invitation, or by initiating joint attention. Joint visual attention has been researched in areas of children's development (e.g., infant-mother interaction, cognitive development of children with autism), education and collaborative learning. For instance, joint attention skills normally start to develop at the child development stage. Infants only after 8 months start to be able to look towards the region following another’s finger [14]. By the age of three, children are usually able to gain and maintain joint attention from adults and peers. Being able to establish joint attention is a crucial social and communicative behaviour [14]. It helps children to see from another’s point of view so that they can learn and develop cognitive skills during the development stage. It helps people to maintain effective social interactions with their peers, caregivers and educators, which is important for building social bonding with others [14]. Children with autism are found to have difficulty with joint attention, which makes it challenging for them to maintain social interactions while patting attention to an object (and vice versa). Joint attention theory also helps HCI researchers to develop interactive tools powered by mobile eye-tracking technology to assist co-located collaborative learning on a tangible interface [15]. Similarly, researchers in [15] studied joint attention in beginners’ collaborative programming process. However, such research mainly focuses on the joint attention on static objects and items, and they are the main indoor settings. It makes sense to say that it is easier to achieve joint attention on static objects in static indoor environments than on dynamically moving objects outdoors. However, there is little understanding of what makes it difficult to achieve joint visual attention to dynamic waves in the complicated ocean environment. 3. Why is hard to achieve joint attention to waves in the ocean? Many new surfers are grownups or kids who have already developed basic joint attention skills. In a normal day-to-day case, they should have little difficulty in following another's pointing, gaze or verbal instructions to reach joint attention. However, what makes it hard to achieve joint attention on a wave between a beginner surfer and an instructor in the ocean? This section discusses several potential factors below. First, the nature of waves and the ocean is dynamic and complex. On the internet, surfing is often presented with beautiful pictures of a calm ocean, clear weather, gentle wind, and nicely shaped waves. However, this is not always the case in reality. Conditions for surfing constantly change. For instance, the wind might pick up and drop down or change its direction within hours. Tide periodically comes in and goes out every day. Swell travels from and to different directions with varied sizes and power. Other factors such as air pressure, wind direction, sandbar, reef, beech shape, etc., could affect when and how waves form and break. In good conditions, waves can be smoothly lined up so that it is easy to predict and analyse. However, when in tougher conditions as illustrated in Figure 1, waves are messy and unpredictable. It is difficult to analyse and pick the best one from many waves. Figure 1: The complicated and dynamic wave condition In addition, there might be interindividual differences in one’s spatial perception in the ocean environment. For instance, some people tend to underestimate actual distance in open water [16], but the such perception of distance also involves high interindividual variance [16]. This variance in spatial perception could also apply to the sense of angle. The author personally finds it not easy to judge the direction and the angle of the travelling wave, which might be due to lacking a specific referencing point in open water. Nevertheless, this could be a matter of inexperience. Further research or literature is needed to investigate such issues. A beginner surfer’s cognitive load could also make it difficult to follow an instructor’s visual attention. Beginner surfers often feel overwhelmed due to overloading multiple types of complex information. For instance, when a beginner practices surfing in a whitewater area (whitewater refers to the broken wave that rolls toward the shore [17]), the broken waves could constantly push them back to shore. This requires intense physical effort as a surfer needs to use strength to keep balance. It is also cognitively demanding because surfers have to pay attention to incoming big broken waves to avoid any further impact. The process of reading, catching, and riding a wave is also very cognitively consuming for beginners. Within a short period, one has to find a target wave, paddle to gain speed, observe waves while paddling, and decide when it is the best moment to pop up from a surfboard. For beginner surfers, they might have to think about the motion of popping up: where should their feet land? Where should their hands and arms point to? What should they look at? How could they control the board to manoeuvre? During the process of all these thoughts, beginners often have to pay attention to the instructors’ guidance. Such high cognitive loads could also lead to emotional stress during wave catching and popup moments and worries about the incapability of catching a wave. The waves do not stay still to wait for two surfers to create joint attention. They emerge, evolve, and disappear very fast, unlike the indoor static environment in which object does not move at all. A beginner surfer has to pay attention to both waves and an instructor. They often struggle to shift their gaze between instructors and waves. Normally, to follow another person’s visual attention, the person has to create eye contact with the initiator. However, setting up eye contact also hinders the beginner surfer’s ability to pay attention to the wave itself. They might not be able to respond to an incoming wave fast enough if they have been paying attention to the instructor's eye gaze. Therefore, they might miss the best moment when a good wave is coming. As join-attention is bi-directional, it is also equally difficult for the instructor to make sense of how the beginner perceives the direction and target. The instructor might not understand where exactly the beginner is looking. Weather and wind also play a role in making it difficult to create joint visual attention on a wave. For instance, when the wind is very strong, a beginner surfer might struggle to hear the instructor’s verbal instructions. As discussed above, a person’s natural capability of joining another’s attention perhaps might not be the core cause. Instead, it is the complex and dynamic ocean environment that complicates the communication between beginner surfer and instructor surf. In the next section, I explain such issues with the support of surfing visualizations. 4. Illustration of Problem Although little surfing literature investigated the phenomenon of joint visual attention on waves, a recent paper does document an example scenario. A 7-year-old son had to rely on following his surfer dad’s guide (e.g., by following the finger-pointing gestures) for making sense of the ocean. Beginner surfers often lack awareness and attention to emerging signs in the ocean, and they could benefit from following an experienced surfer’s guide for paying attention to the ever-changing waves. Figure 2 describes a scenario in which a beginner surfer struggles to join in the instructor’s visual attention on a target wave. The surfer on the left is the beginner surfer, and the one on the right is the instructor surfer. Table 1 demonstrates the example conversations between the pair. For instance, the instructor points to one wave using his left hand and index finger and says, “Look there! Let’s catch that wave!” The beginner surfer tries to follow the instructor’s pointing direction but still feels confused, “Which part of the wave is he looking at?” Figure 2: Difficulty in creating joint visual attention in the ocean between a beginner surfer (left) and an instructor surfer (right) Table 1 Example conversations during creating joint visual attention Instructor surfer: “Look there! It is coming, it’s going to be good! Let’s catch it !” Beginner surfer: “I am so confused! Which one is he talking about!?” “Where are you looking at, instructor ?” “I cannot hear you, Instructor !” “Which part of the wave should I target?” Figure 3 and Figure 4 demonstrate two real-life examples of joint visual attention on waves between an instructor surfer and beginner surfers. In both figures, an instructor surfer (who sits on the left side and is annotated in green) is pointing at part A of the wave, whereas the beginner surfers (who sit on the right side and are annotated in red) try to follow his pointing gesture to join the attention on A. However, both beginner surfers are looking at other points like B and C. Figure 3: Example 1 - Joint visual attention on waves between instructor and beginner surfers. Figure 4: Example 2 - Joint visual attention on waves between instructor and beginner surfers. Figure 3 and Figure 4 consist of two photos by the professional photographer James Gilpin, a Cornwall-based photographer who loves shooting seascapes, surfing and wildlife (Instagram and Facebook Page: @acornerofkernow). The annotations were made by the author only for demonstration purposes for this paper. They do not represent any personal thoughts or experiences from the surfers included in the pictures. 5. Speculative Design This section presents a series of low-fidelity sketching to illustrate the speculative design and interaction process in five stages as overviewed in Figure 5. Inspired by work that uses gaze position to place labels in mobile Augmented Reality [18, 19], this speculative design suggests a pair of smart goggles that provide Augmented Reality (AR) capability and gaze tracking technology. At stage 1, the smart goggles capture the real-time wave and ocean using embedded cameras and computing vision techniques (as shown in Figure 5-1). In stage 2, the smart goggles represent the real-time view on the screen without any latency so that the view is synchronized with the surfer’s view (as shown in Figure 5-2). At stage 3 in Figure 5-3, smart goggles provide the pre-defined label sets for the instructor surfer. The label set includes 1) Target wave (in green), 2) Best Catching Point (in red; arrows indicate take- off directions), 3) Paddle Route (in blue), and 4) No-go area (in red with cross label). At stage 4, the instructor surfer could use gaze to select a label and move the label to the target area then blink to apply the label to annotate the wave (as shown in Figure 5-4). At stage 5, the smart goggles system synchronizes the annotated wave and ocean wave so that the beginner surfer can easily follow the instructor’s attention on the wave targets. Figure 5: An overview of the proposed design process Figure 5-1: Smart goggles capture the real-time wave and ocean using embedded cameras and computing vision techniques. Figure 5-2: Smart goggles represent the real-time view that is synchronized with the surfer’s view. Figure 5-3: Smart goggles provide label sets for the instructor surfer: 1) Target Wave, 2) Best Catching Point, 3) Paddling Route, 4) No-go Area. Figure 5-4: The instructor surfer uses eye gaze to select a label and move it to the desired area, and then “blink” to apply the label. Figure 5-5: Smart goggles system synchronizes the annotated view between instructor surfers and beginner surfers. 6. Summary Despite surfing has been attracting an increasing number of newcomers, it remains a very challenging outdoor sport to learn. This paper reflects on the author’s personal experience of learning surfing and surfing, as well as the author’s observations of other beginner surfers’ learning experiences. This paper discussed the challenge of synchronizing views of waves in the ocean, using the theory of joint visual attention. This paper also discussed several possible factors related to the dynamic and complicated ocean environment. It presents a speculative design solution of using gaze-tracking and AR technologies on goggles to reach joint visual attention on wave targets. This paper remains a speculative design rather than empirical research. It aims to gain attention from sports HCI researchers to further elaborate the research idea. Further user-centered design studies should be conducted to validate the research problem and to improve the design solution. 7. References 1. Global Industry Analysts, I. A $4.6 Billion Global Opportunity for Surfing by 2026 - New Research from StrategyR. [cited 2022 19 Sep]; Available from: https://www.prnewswire.com/news-releases/a-4-6-billion-global-opportunity-for-surfing-by- 2026---new-research-from-strategyr-301557738.html. 2. Godfrey, C., H. Devine-Wright, and J. Taylor, The positive impact of structured surfing courses on the wellbeing of vulnerable young people. Community Practitioner, 2015. 88(1): p. 26-29. 3. Matos, M., et al., Surfing for social integration: mental health and well-being promotion through surf therapy among institutionalized young people. HSOA journal of community medicine & public health care, 2017. 4(1): p. 1-6. 4. Taylor, J., Giving kids a break: How surfing has helped young people in Cornwall overcome mental health and social difficulties. Mental Health and Social Inclusion, 2013. 5. Moreton, S.G., et al., Exploring potential mechanisms underpinning the therapeutic effects of surfing. Journal of Adventure Education and Outdoor Learning, 2022. 22(2): p. 117-134. 6. Stuhl, A. and H. Porter, Riding the waves: Therapeutic surfing to improve social skills for children with autism. Therapeutic Recreation Journal, 2015. 49(3): p. 253. 7. Brochado, A., O. Stoleriu, and C. Lupu, Surf camp experiences. Journal of Sport & Tourism, 2018. 22(1): p. 21-41. 8. Beal, B. and M.M. Smith, Maverick's: Big-wave surfing and the dynamic of ‘nothing'and ‘something’, in The consumption and representation of lifestyle sports. 2014, Routledge. p. 46- 60. 9. Partington, S., E. Partington, and S. Olivier, The dark side of flow: A qualitative study of dependence in big wave surfing. The sport psychologist, 2009. 23(2): p. 170-185. 10. Ford, N.J. and D. Brown, Surfing and social theory: Experience, embodiment and narrative of the dream glide. 2005: Routledge. 11. Britton, E., ‘Be Like Water’: Reflections on Strategies Developing Cross-Cultural Programmes for Women, Surfing and Social Good, in The Palgrave handbook of feminism and sport, leisure and physical education. 2018, Springer. p. 793-807. 12. Prins, A. and B. Wattchow, Young surfers finding their wave: telling the tale of enskilment in surf places. Leisure Studies, 2022: p. 1-14. 13. Boucenna, S., P. Gaussier, and L. Hafemeister. Development of joint attention and social referencing. in 2011 IEEE international conference on development and learning (ICDL). 2011. IEEE. 14. Scaife, M. and J.S. Bruner, The capacity for joint visual attention in the infant. Nature, 1975. 253(5489): p. 265-266. 15. Schneider, B., et al., Leveraging mobile eye-trackers to capture joint visual attention in co- located collaborative learning groups. International Journal of Computer-Supported Collaborative Learning, 2018. 13(3): p. 241-261. 16. Button, C., M. Schofield, and J. Croft, Distance perception in an open water environment: Analysis of individual differences. Attention, Perception, & Psychophysics, 2016. 78(3): p. 915- 922. 17. How to learn to surf in the whitewater. 2022 [cited 2022 20 Sep]; Available from: https://www.surfertoday.com/surfing/how-to-learn-to-surf-in-the-whitewater. 18. McNamara, A. and C. Kabeerdoss. Mobile augmented reality: Placing labels based on gaze position. in 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR- Adjunct). 2016. IEEE. 19. Rivu, R., et al. Stare: Gaze-assisted face-to-face communication in augmented reality. in ACM Symposium on Eye Tracking Research and Applications. 2020.