=Paper=
{{Paper
|id=Vol-3267/paper003
|storemode=property
|title=I See What You See! Towards Augmented Joint Visual Attention between
Beginner and Instructor Surfers
|pdfUrl=https://ceur-ws.org/Vol-3267/paper3.pdf
|volume=Vol-3267
|authors=Tao Bi
|dblpUrl=https://dblp.org/rec/conf/mhci/Bi22
}}
==I See What You See! Towards Augmented Joint Visual Attention between
Beginner and Instructor Surfers==
https://ceur-ws.org/Vol-3267/paper3.pdf
I See What You See! Towards Augmented Joint Visual Attention
between Beginner and Instructor Surfers
Tao Bi 1
1
University College London, 66-72 Gower Street, London, UK
Abstract
Wave reading and selection are vital, yet challenging, for beginner surfers during the process
of learning to surf. Built on the author's auto-ethnographical diaries, this paper reflects the real-
life challenge of noticing when and where to catch the best wave. More importantly, beginner
surfers often struggle to follow the instructor surfers who are instructing them with references
to a particular wave or specific sections of a wave. The author uses the joint visual attention
theory to provide a tentative explanation for this observed phenomenon. This paper proposes
a speculative design solution of using AR and gaze-tracking-based goggles to foster joint
attention on waves between beginner and instructor surfer, allowing the former to follow the
instructor's directions and the latter to understand where the beginner is looking. This paper
primarily aims to facilitate discussions in the sports HCI community and hopes to provide a
reference to surfing literature in which beginner surfers and their learning experience lack
attention.
Keywords 1
Surfing, Learning to Surf, Wave Reading and Selecting, Surf Coaching, Augmented Reality,
Joint Visual Attention, Smart Goggles, Speculative Design, Auto-ethnography
1. Introduction
Surfing has been one of the fastest-growing water sports [1] that offers spiritual and adventurous
opportunities for being closely connected to the mother ocean. It offers not only fitness benefits but also
mental wellness boost [2-6]. However, surfing remains to be one of the most challenging and slow-
progress sports. Catching good waves consistently requires years of practice. Selecting the right wave
and catching it at the right moment is critical to a successful surfing experience. Taking in-person
surfing lessons from experienced surfers or professional surfing instructors is getting more popular to
effectively gain knowledge about the ocean and waves [7]. However, problems and frustration are still
frequently observed. For instance, beginner surfers often feel confused and struggle to follow an
instructor surfer who is pointing to a particular wave section in the ocean. This not only causes beginner
surfers’ frustration but also extends the learning curve for reading and catching waves.
To further discuss such issues, the author reflects on auto-ethnographical diaries of his own surfing-
learning experience as well as his observations of a local surf community over two years in Newquay,
Cornwall, UK. In this paper, the author will first introduce the “Joint Visual Attention” theory
originating from psychology and child development literature. This theory could provide explanations
for how and why it is challenging for beginner surfers to develop shared and synchronized visual
attention on target waves during their communication with instructors. In addition, the author will
discuss several characteristics of the complex and dynamic ocean waves that potentially led to the
challenge of accurately joining an instructor’s visual attention on a wave. The also paper proposes a
speculative design solution - using AR goggles to help surfers achieve joint visual attention in real-
time. The problem description and design speculations are supported by some explicative figures.
NTSPORT’22: New Trends in HCI and Sports Workshop at MobileHCI’22, October 1, 2022
EMAIL: t.bi@ucl.ac.uk (A. 1)
ORCID: 0000-0002-7406-8943 (A. 1)
©️ 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
As surfing has not been widely studied in the HCI community, this paper aims to gain attention and
discussion from HCI researchers who are interested in sports and outdoor scenarios. Surfing literature
has also been mainly focused on experienced surfing (e.g., surfing big waves [8, 9]) and on the
understanding of its societal aspects [3, 6, 10, 11]. This paper hopes to contribute to a reference to
literature on beginner surfers’ learning experiences [12]. Last but not least, the speculative design
proposal of an AR porotype could also raise discussions and provide inspiration on how to reach joint
visual attention in other outdoor sports (e.g., referring to a snow slope during a ski or snowboarding
session, or referring to a mountain section during climbing or mountain biking).
2. What is joint visual attention?
Joint visual attention means one person looks at where another is looking by following their pointing
gaze or gesture. The pair aims to focus on the same object or event, to interact with each other. Joint
attention can be achieved by using eye contact (looking at another’s eye gaze and face), gestures
(pointing using index finger), or vocalisations (e.g., saying “look over there!”), giving matching
reinforcement by verbal and behavioural confirmation [13]. Joint attention is bi-directional. It can also
occur by one responding to another’s attention invitation, or by initiating joint attention.
Joint visual attention has been researched in areas of children's development (e.g., infant-mother
interaction, cognitive development of children with autism), education and collaborative learning. For
instance, joint attention skills normally start to develop at the child development stage. Infants only
after 8 months start to be able to look towards the region following another’s finger [14]. By the age of
three, children are usually able to gain and maintain joint attention from adults and peers. Being able to
establish joint attention is a crucial social and communicative behaviour [14]. It helps children to see
from another’s point of view so that they can learn and develop cognitive skills during the development
stage. It helps people to maintain effective social interactions with their peers, caregivers and educators,
which is important for building social bonding with others [14]. Children with autism are found to have
difficulty with joint attention, which makes it challenging for them to maintain social interactions while
patting attention to an object (and vice versa).
Joint attention theory also helps HCI researchers to develop interactive tools powered by mobile
eye-tracking technology to assist co-located collaborative learning on a tangible interface [15].
Similarly, researchers in [15] studied joint attention in beginners’ collaborative programming process.
However, such research mainly focuses on the joint attention on static objects and items, and they are
the main indoor settings. It makes sense to say that it is easier to achieve joint attention on static objects
in static indoor environments than on dynamically moving objects outdoors. However, there is little
understanding of what makes it difficult to achieve joint visual attention to dynamic waves in the
complicated ocean environment.
3. Why is hard to achieve joint attention to waves in the ocean?
Many new surfers are grownups or kids who have already developed basic joint attention skills. In
a normal day-to-day case, they should have little difficulty in following another's pointing, gaze or
verbal instructions to reach joint attention. However, what makes it hard to achieve joint attention on a
wave between a beginner surfer and an instructor in the ocean? This section discusses several potential
factors below.
First, the nature of waves and the ocean is dynamic and complex. On the internet, surfing is often
presented with beautiful pictures of a calm ocean, clear weather, gentle wind, and nicely shaped waves.
However, this is not always the case in reality. Conditions for surfing constantly change. For instance,
the wind might pick up and drop down or change its direction within hours. Tide periodically comes in
and goes out every day. Swell travels from and to different directions with varied sizes and power.
Other factors such as air pressure, wind direction, sandbar, reef, beech shape, etc., could affect when
and how waves form and break. In good conditions, waves can be smoothly lined up so that it is easy
to predict and analyse. However, when in tougher conditions as illustrated in Figure 1, waves are messy
and unpredictable. It is difficult to analyse and pick the best one from many waves.
Figure 1: The complicated and dynamic wave condition
In addition, there might be interindividual differences in one’s spatial perception in the ocean
environment. For instance, some people tend to underestimate actual distance in open water [16], but
the such perception of distance also involves high interindividual variance [16]. This variance in spatial
perception could also apply to the sense of angle. The author personally finds it not easy to judge the
direction and the angle of the travelling wave, which might be due to lacking a specific referencing
point in open water. Nevertheless, this could be a matter of inexperience. Further research or literature
is needed to investigate such issues.
A beginner surfer’s cognitive load could also make it difficult to follow an instructor’s visual
attention. Beginner surfers often feel overwhelmed due to overloading multiple types of complex
information. For instance, when a beginner practices surfing in a whitewater area (whitewater refers to
the broken wave that rolls toward the shore [17]), the broken waves could constantly push them back
to shore. This requires intense physical effort as a surfer needs to use strength to keep balance. It is also
cognitively demanding because surfers have to pay attention to incoming big broken waves to avoid
any further impact.
The process of reading, catching, and riding a wave is also very cognitively consuming for
beginners. Within a short period, one has to find a target wave, paddle to gain speed, observe waves
while paddling, and decide when it is the best moment to pop up from a surfboard. For beginner surfers,
they might have to think about the motion of popping up: where should their feet land? Where should
their hands and arms point to? What should they look at? How could they control the board to
manoeuvre? During the process of all these thoughts, beginners often have to pay attention to the
instructors’ guidance. Such high cognitive loads could also lead to emotional stress during wave
catching and popup moments and worries about the incapability of catching a wave.
The waves do not stay still to wait for two surfers to create joint attention. They emerge, evolve, and
disappear very fast, unlike the indoor static environment in which object does not move at all. A
beginner surfer has to pay attention to both waves and an instructor. They often struggle to shift their
gaze between instructors and waves. Normally, to follow another person’s visual attention, the person
has to create eye contact with the initiator. However, setting up eye contact also hinders the beginner
surfer’s ability to pay attention to the wave itself. They might not be able to respond to an incoming
wave fast enough if they have been paying attention to the instructor's eye gaze. Therefore, they might
miss the best moment when a good wave is coming.
As join-attention is bi-directional, it is also equally difficult for the instructor to make sense of how
the beginner perceives the direction and target. The instructor might not understand where exactly the
beginner is looking. Weather and wind also play a role in making it difficult to create joint visual
attention on a wave. For instance, when the wind is very strong, a beginner surfer might struggle to hear
the instructor’s verbal instructions.
As discussed above, a person’s natural capability of joining another’s attention perhaps might not
be the core cause. Instead, it is the complex and dynamic ocean environment that complicates the
communication between beginner surfer and instructor surf. In the next section, I explain such issues
with the support of surfing visualizations.
4. Illustration of Problem
Although little surfing literature investigated the phenomenon of joint visual attention on waves, a
recent paper does document an example scenario. A 7-year-old son had to rely on following his surfer
dad’s guide (e.g., by following the finger-pointing gestures) for making sense of the ocean. Beginner
surfers often lack awareness and attention to emerging signs in the ocean, and they could benefit from
following an experienced surfer’s guide for paying attention to the ever-changing waves.
Figure 2 describes a scenario in which a beginner surfer struggles to join in the instructor’s visual
attention on a target wave. The surfer on the left is the beginner surfer, and the one on the right is the
instructor surfer. Table 1 demonstrates the example conversations between the pair. For instance, the
instructor points to one wave using his left hand and index finger and says, “Look there! Let’s catch
that wave!” The beginner surfer tries to follow the instructor’s pointing direction but still feels confused,
“Which part of the wave is he looking at?”
Figure 2: Difficulty in creating joint visual attention in the ocean between a beginner surfer (left) and
an instructor surfer (right)
Table 1
Example conversations during creating joint visual attention
Instructor surfer: “Look there! It is coming, it’s going to be good! Let’s catch it !”
Beginner surfer: “I am so confused! Which one is he talking about!?”
“Where are you looking at, instructor ?”
“I cannot hear you, Instructor !”
“Which part of the wave should I target?”
Figure 3 and Figure 4 demonstrate two real-life examples of joint visual attention on waves between
an instructor surfer and beginner surfers. In both figures, an instructor surfer (who sits on the left side
and is annotated in green) is pointing at part A of the wave, whereas the beginner surfers (who sit on
the right side and are annotated in red) try to follow his pointing gesture to join the attention on A.
However, both beginner surfers are looking at other points like B and C.
Figure 3: Example 1 - Joint visual attention on waves between instructor and beginner surfers.
Figure 4: Example 2 - Joint visual attention on waves between instructor and beginner surfers.
Figure 3 and Figure 4 consist of two photos by the professional photographer James Gilpin, a
Cornwall-based photographer who loves shooting seascapes, surfing and wildlife (Instagram and
Facebook Page: @acornerofkernow). The annotations were made by the author only for demonstration
purposes for this paper. They do not represent any personal thoughts or experiences from the surfers
included in the pictures.
5. Speculative Design
This section presents a series of low-fidelity sketching to illustrate the speculative design and
interaction process in five stages as overviewed in Figure 5. Inspired by work that uses gaze position to
place labels in mobile Augmented Reality [18, 19], this speculative design suggests a pair of smart
goggles that provide Augmented Reality (AR) capability and gaze tracking technology. At stage 1, the
smart goggles capture the real-time wave and ocean using embedded cameras and computing vision
techniques (as shown in Figure 5-1). In stage 2, the smart goggles represent the real-time view on the
screen without any latency so that the view is synchronized with the surfer’s view (as shown in Figure
5-2). At stage 3 in Figure 5-3, smart goggles provide the pre-defined label sets for the instructor surfer.
The label set includes 1) Target wave (in green), 2) Best Catching Point (in red; arrows indicate take-
off directions), 3) Paddle Route (in blue), and 4) No-go area (in red with cross label). At stage 4, the
instructor surfer could use gaze to select a label and move the label to the target area then blink to apply
the label to annotate the wave (as shown in Figure 5-4). At stage 5, the smart goggles system
synchronizes the annotated wave and ocean wave so that the beginner surfer can easily follow the
instructor’s attention on the wave targets.
Figure 5: An overview of the proposed design process
Figure 5-1: Smart goggles capture the real-time wave and ocean using embedded cameras and
computing vision techniques.
Figure 5-2: Smart goggles represent the real-time view that is synchronized with the surfer’s view.
Figure 5-3: Smart goggles provide label sets for the instructor surfer: 1) Target Wave, 2) Best
Catching Point, 3) Paddling Route, 4) No-go Area.
Figure 5-4: The instructor surfer uses eye gaze to select a label and move it to the desired area, and
then “blink” to apply the label.
Figure 5-5: Smart goggles system synchronizes the annotated view between instructor surfers
and beginner surfers.
6. Summary
Despite surfing has been attracting an increasing number of newcomers, it remains a very
challenging outdoor sport to learn. This paper reflects on the author’s personal experience of learning
surfing and surfing, as well as the author’s observations of other beginner surfers’ learning experiences.
This paper discussed the challenge of synchronizing views of waves in the ocean, using the theory of
joint visual attention. This paper also discussed several possible factors related to the dynamic and
complicated ocean environment. It presents a speculative design solution of using gaze-tracking and
AR technologies on goggles to reach joint visual attention on wave targets. This paper remains a
speculative design rather than empirical research. It aims to gain attention from sports HCI researchers
to further elaborate the research idea. Further user-centered design studies should be conducted to
validate the research problem and to improve the design solution.
7. References
1. Global Industry Analysts, I. A $4.6 Billion Global Opportunity for Surfing by 2026 - New
Research from StrategyR. [cited 2022 19 Sep]; Available from:
https://www.prnewswire.com/news-releases/a-4-6-billion-global-opportunity-for-surfing-by-
2026---new-research-from-strategyr-301557738.html.
2. Godfrey, C., H. Devine-Wright, and J. Taylor, The positive impact of structured surfing courses
on the wellbeing of vulnerable young people. Community Practitioner, 2015. 88(1): p. 26-29.
3. Matos, M., et al., Surfing for social integration: mental health and well-being promotion
through surf therapy among institutionalized young people. HSOA journal of community
medicine & public health care, 2017. 4(1): p. 1-6.
4. Taylor, J., Giving kids a break: How surfing has helped young people in Cornwall overcome
mental health and social difficulties. Mental Health and Social Inclusion, 2013.
5. Moreton, S.G., et al., Exploring potential mechanisms underpinning the therapeutic effects of
surfing. Journal of Adventure Education and Outdoor Learning, 2022. 22(2): p. 117-134.
6. Stuhl, A. and H. Porter, Riding the waves: Therapeutic surfing to improve social skills for
children with autism. Therapeutic Recreation Journal, 2015. 49(3): p. 253.
7. Brochado, A., O. Stoleriu, and C. Lupu, Surf camp experiences. Journal of Sport & Tourism,
2018. 22(1): p. 21-41.
8. Beal, B. and M.M. Smith, Maverick's: Big-wave surfing and the dynamic of ‘nothing'and
‘something’, in The consumption and representation of lifestyle sports. 2014, Routledge. p. 46-
60.
9. Partington, S., E. Partington, and S. Olivier, The dark side of flow: A qualitative study of
dependence in big wave surfing. The sport psychologist, 2009. 23(2): p. 170-185.
10. Ford, N.J. and D. Brown, Surfing and social theory: Experience, embodiment and narrative of
the dream glide. 2005: Routledge.
11. Britton, E., ‘Be Like Water’: Reflections on Strategies Developing Cross-Cultural Programmes
for Women, Surfing and Social Good, in The Palgrave handbook of feminism and sport, leisure
and physical education. 2018, Springer. p. 793-807.
12. Prins, A. and B. Wattchow, Young surfers finding their wave: telling the tale of enskilment in
surf places. Leisure Studies, 2022: p. 1-14.
13. Boucenna, S., P. Gaussier, and L. Hafemeister. Development of joint attention and social
referencing. in 2011 IEEE international conference on development and learning (ICDL).
2011. IEEE.
14. Scaife, M. and J.S. Bruner, The capacity for joint visual attention in the infant. Nature, 1975.
253(5489): p. 265-266.
15. Schneider, B., et al., Leveraging mobile eye-trackers to capture joint visual attention in co-
located collaborative learning groups. International Journal of Computer-Supported
Collaborative Learning, 2018. 13(3): p. 241-261.
16. Button, C., M. Schofield, and J. Croft, Distance perception in an open water environment:
Analysis of individual differences. Attention, Perception, & Psychophysics, 2016. 78(3): p. 915-
922.
17. How to learn to surf in the whitewater. 2022 [cited 2022 20 Sep]; Available from:
https://www.surfertoday.com/surfing/how-to-learn-to-surf-in-the-whitewater.
18. McNamara, A. and C. Kabeerdoss. Mobile augmented reality: Placing labels based on gaze
position. in 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-
Adjunct). 2016. IEEE.
19. Rivu, R., et al. Stare: Gaze-assisted face-to-face communication in augmented reality. in ACM
Symposium on Eye Tracking Research and Applications. 2020.