Towards a Concept of Needs-Based Augmented Reality

Abstract

Augmented reality aims to enhance the real world with computer-generated information. AR technology is both attractive and promising. Current AR experiences depend on external elements to launch, such as markers, images, and location. For an AR experience to be more personalized, this research proposes a scheme to trigger AR experiences based on human needs. This approach should enable capturing human needs, analyzing them to select the most suited experiences that fulfill or aids in fulfilling needs. The contribution of this paper includes (1) a study of current AR technologies and triggers, (2) an analysis of human needs into measurable elements (3) a description of a needs-based AR application process.

Keywords

• Conceptual modeling
• augmented reality
• human needs
• experience trigger
• ontology
• satisfiers

Introduction

 Enhances real environments by adding virtual objects.  Works in real-time and provides interactivity.  Provides the correct placement of virtual objects within the environment.

AR has many application areas such as education and learning [ 4 ], entertainment and gaming [ 5 ], food and beverage industry [ 6 ], health care [ 7 ], manufacturing [ 8 ], museums [ 9 ], space exploration [ 10 ], and tourism [ 11 ].

The study of augmented Reality incorporates many areas such as tracking, interaction, display, mobile AR, authoring, visualization, calibration, and rendering [ 12 ]. AR literature focuses on the development of the technology components; however, little attention is devoted to examining personalized experiences. This research aims to enrich augmented reality paradigms by developing a novel AR experience trigger: A Need’s trigger. Despite the importance of needs and their satisfaction in human life, there is still a shortage of incorporating human needs in information systems and tools [ 13 ]. Augmented Reality is a growing field that may benefit from the utilization of human needs. Similar to the present location, markers, and image recognition triggers, a basic human need may be utilized to activate a more personalized experience.

This research makes several contributions (1) study augmented reality classifications and triggers, (2) analyze human needs and transfer them into sensible data elements (3) present the human needs trigger framework for augmented reality.

The following sections discuss the state-of-the-art in the different types of AR. Provide a discussion of the proposed framework. Finally, demonstrates and discusses the use of this model for creating Needs-based AR experiences. 2

Background

Augmentation is described in previous literature according to human senses [ 14 ]; there is visual, auditory, and tactile augmentation. Each of these is utilized for a particular goal and requires different hardware. This section presents a literature study on AR. 2.1

Technologies in Augmented Reality

It is essential to understand the general process in an AR system, to realize where each of the technologies fits and how they function together. This section provides definitions for the various topics and technologies supporting the AR process (Fig.1). The general process starts by acquiring an image from the real world and sending it to computing and storage devices. This image and user interaction gestures are sent to the tracking technology, which decides what virtual information to display based on the user’s position, viewing direction, and state of motion. The virtual information is retrieved from the virtual objects database and sent to the fusion technology that ensures proper object registration in the real world. The virtual information is then displayed using the display technology [ 15 ]. Table 1 provides definitions for the core AR technologies that remain popular research topics: tracking, interaction and user interfaces, calibration and registration, display techniques, and AR applications [ 16 ].

Definition “Methods of tracking a target object/environment via cameras and sensors, and estimating viewpoint poses.” “Techniques and interfaces for interacting with virtual content.” “Geometric or photometric calibration methods, and method to align multiple coordinate frames.” “Display hardware to present virtual content in AR, including head-worn, handheld, and projected displays.” “AR systems in application domains such as medicine, manufacturing, or military, among others.” 2.2

Classification of Augmented Reality

There are different types of AR systems and apps depending on the concepts and technology. The classification of these types is not unified; other research papers classify the types differently. Edwards-Stewart, Hoyt, and Reger [ 17 ] describe six types of AR under two main categories: triggered and view-based augmentation. The triggered AR technologies include marker-based AR, location-based AR, dynamic augmentation, and complex augmentation. The view-based augmentation includes Indirect and nonspecific digital augmentations. Table 2 describes the classification mentioned above. Other classifications divide AR into marker-based, marker-less, outlining AR, and superimposition AR. The marker-based type of AR depends on a marker that, when scanned by a camera, triggers an AR experience. The marker may be an image, a fiducial marker (Fig.2), or a physical object.

Fig. 2. Fiducial Marker Marker Less AR does not require markers; it uses other triggers such as location-based AR. The projection-based AR is also markerless and works by projecting digital information on objects in the user’s environment. Outlining AR uses the special abilities of cameras in recognizing objects in conditions that may be difficult to recognize for the eyes and provide guidance in the form of outlines on objects. Superimposition AR recognizes objects and superimposes virtual information on them. It includes face filters as done by Instagram and Snapchat. 2.3

Review of Existing Platforms

This section provides a summary of prominent AR platforms. These platforms offer complete AR-experience creation capabilities. The focus is on how the different experiences are triggered. Table 3 summarizes the types of experiences and triggers of the prominent AR platforms, AWE (Augmented Web Experiences), Zap Works, BlippAR, Spark AR, Wikitude, and Unity AR. As clear from the summary above, marker-based AR is the most used type with image and object recognition. While most platforms enable generic experience creation, some offer face tracking, location-based, and context-aware capabilities allowing for customized experiences. 2.4

Triggers

From the previous sections, we observe the variety of AR triggers. Triggers in AR are “stimuli or characteristics that initiate or trigger the augmentation” [ 17 ]. Currently, the top starters of an AR experience are markers, images, physical objects, scene recognition, movement, location, and sometimes the choice to load an experience. In some instances, an experience may be initiated by multiple separate triggers [ 18 ].

In the future, it is expected that more advanced forms of triggers will be utilized in augmented reality, such as sound, temperature, smell, voice recognition, and gesture [ 18 ].

Theoretical Framework

This research proposes the hypothesis, “With advancing sensor technology, basic human needs are viable triggers for augmented reality experiences.” The goal is to create a novel paradigm and provide developers with more options to personalize AR experiences. This section describes the conceptual basis leading to and supporting the hypothesis. 3.1

Personalization in Augmented Reality

In many cases, AR is used to enhance the user experience in museums and tourism [ 19 ] [ 20 ], education [ 21 ] [ 22 ], entertainment, and medicine by adding digital information to the real environment. These approaches usually focus on the experience and the object requiring enhancement. Another way to conceive experiences is to concentrate on the user/users using them. This is the concept of personalization, defined as “a process that changes the functionality, interface, information access, and content, or distinctiveness of a system to increase its relevance to an individual or a category of individuals” [ 23 ]. The core elements of personalization definitions are:  “a purpose or goal of personalization.”  “what is personalized.” Four aspects of information systems may be personalized: the information (content), the presentation of information (user interface), the delivery method (channel), the action  “the target of personalization.” The target can be a group of individuals or a specific individual.

Personalization enables users to acquire information specific to their “needs, goals, knowledge, interests or other characteristics” [ 24 ]. Studying the types of AR experiences in literature, three primary levels of personalization are evident. The first level is generic experiences in which no personalization is implemented. The AR experience displays the same information to all viewers at any time, such as in museum displays. The second personalization level includes experiences receptive to external factors such as location and context [ 25 ] [ 26 ]. The third level displays information about a specific user. An example of this is SentiAR1 an AR experience about a patient in a surgery room or an AccuVein2 device enabling the view of a patient vein in blood sampling.

This research aims to formulate a roadmap that embeds human needs in AR experiences to enhance the personalization. In this sense, personalization elements are defined as:  Purpose: to trigger AR experiences based on user needs  Personalized object: the trigger of the AR experience

1 https://sentiar.com/ 2 https://www.accuvein.com/

 Target: the user viewing the AR experience

The following sections explain how these elements are applied in the proposed system. 3.2

Human needs in Pervasive Environments

The satisfaction of human needs is a core value in pervasive environments. The study of needs in computing has been around for some time now. This topic is addressed from many viewpoints:  Needs representation: Some scholars attempted to represent human needs using ontologies [ 13 ] [ 27 ], others used directed graphs [ 28 ].  Human needs are identified in several methods: interviews, questionnaires, signal processing on brain scans, and prediction methods that depend on sentiment analysis.  From the literature study, it is clear that technology may provide need satisfaction by different means, including providing services, social media use [ 29 ] [ 30 ], internet and mobile use, online relationships, video games, and gamification. 3.3

Proposed Concept

As there are different types of AR experiences with different triggers, this research proposes to add a new type, which is Needs-based AR. For a Needs-Based AR system, a human need triggers and starts the AR experience. Manfred Max-Neef’s model will provide a guide for the categorization of needs. A detailed analysis will enable knowledge about the required sensors for the different needs and the logic necessary to transform sensor data into definite needs (Fig. 3). The focus is on how needs trigger AR rather than how AR satisfies a need. The satisfaction of needs in developing AR could be the responsibility of experience developers and marketers.

Augmented Reality Need Identification

Logic

Sensor Data 3.4

Need Analysis

Capturing human needs in a measurable form is a complex endeavour. Therefore, it requires extensive study and analysis. In this research, Max-Neef’s human-scale development theory and the proposed human needs matrix [ 31 ] (Table 4) are used as a guideline to explore the various needs and dissect them into measurable elements. Furthermore, the Need Context Technology (NCT) framework (Table 5) [ 32 ] is used to map relations and tools in the analysis. The aim is to study the technology that supports need detection, starting with the most basic measurable needs (usually related to health readings, hunger, stress) and evolving into the most complex needs. Hence the goal is to provide the logic to respond to situations such as this example:  If a user needs subsistence on a Being axis, trigger experience A.  This statement leads to the following questions: 

What elements define a need?  How to measure a need?

A trigger in psychology is a factor that activates a need. There are three types of needs triggers [ 34 ] (Fig.4):  Homeostasis Imbalance: is the internal state that reflects a malfunction in the body processes resulting in a rise of a need. (internal) a person. (external)  Incentive: is an external positive or negative environmental stimulus that motivates  Stimulation: is an activity that causes excitement or pleasure.

Connecting these triggers (Fig.5) with the basic needs and the related technology (Table 5) forms the foundation for detecting a human need and establishing a need as a trigger for an AR experience.

Need Category

Trigger of Need

Technology

This section connects the various needs with currently available sensors. To establish the connection, first, the identification of the major categories of needs triggers: internal and external (Fig. 6). The internal type relates to a user’s body readings, including homeostasis, emotions, and stimulation. While the external relates to the surroundings and the environment, incentives, context, and stimulation, these categories also apply to the sensor types. Specific body and wearable sensors can be used for internal triggers, such as Wearable Health Systems (WHS) or Wearable Biomedical Systems (WBS) [ 35 ]. These are termed non-invasive measuring devices that provide continuous monitoring. The external triggers relate to context and environmental sensors, including location, temperature, lighting conditions, and sound. Fig. 6. External and Internal Needs

Triggers 4

Needs-Based Trigger for Augmented Reality Scenario

This section details a specific example scenario to explain the needs-based augmented reality experience, followed by an application process. 4.1

Scenario Analysis

A scenario is a “hypothetical story used to help a person think through a complex problem or system” [ 36 ]. The intention is to use the scenario as an example to explain the use of the model.

For this scenario, the decision is to use the subsistence need on the being axis. Basic subsistence needs include the need for food, water, and accommodation. So, the AR experience developer decides that: when a user is hungry, trigger experience A, an advertisement for the nearby restaurant.

Analyzing this scenario, it is possible to have one of these cases:  There is a homeostatic need for food.  There is a regular habit of eating at a specific time.  There is an external incentive that excites the user to have food.

The homeostatic need requires special sensors to detect the production of the ghrelin hormone that indicates energy scarcity and hunger and triggers the experience. The habit may be predicted from previous user activity; therefore, at a particular time every day, the user needs to eat, and the system predicts that and presents the AR experience. Detecting context information leads to identifying possible incentives that might cause hunger, offering an AR experience in this situation, and asking for feedback can enhance the process of detection and trigger initiation. 4.2

Application Process

Based on the previous scenario, the following application process is developed (Fig. 7). The process starts with selecting the sensor data, analyzing the data, followed by predictions that lead to triggering the AR experience. Afterward, collecting user feedback enhances the analysis and prediction stages. In order to test this application process, the following prototype architecture is suggested (Fig. 8). The architecture is composed of the frontend user interface and the backend experience developer interface. The developer decides the need for which he/she develops the AR experience. Then collects relevant content in the form of audio, image, 3d objects, or animation. Then, the designer designs and produces the experience. These experiences are then saved in the platform database. The user frontend initiates by creating a user profile and collecting all basic user information. The application then collects continuous sensor data and analysis possible needs. The analysis can result in a need-based recognition or prediction. Based on the identified need, an AR experience is recommended and displayed to the user.

Discussion

The present research explores the technologies and classifications of augmented reality. The research connects the concept of AR with that of human needs. In this case, the human need is considered as the trigger of the AR experience which provides more personalized experiences.

While a need is a primary controller of the experience trigger, it might stir an argument: if a user has deficiencies in basic human needs, what makes the user capable of possessing the technology to operate the experience? This question might be answered in two folds.  Technology is now reaching many people in various living conditions  People undergo changing levels of needs daily.

AR experiences are custom-made for specific reasons; however, these reasons are usually generic and not user-related, even in the case of personalized AR, the experiences are product, location, or service related more than user related. This research questions that from a plethora of AR experiences that shall be available in the future, how do we make AR more user-specific and find the best experience for a user at a certain time? This study answers this question by incorporating basic human needs into the equation. Therefore, the novelty is in connecting the concepts of augmented reality, needs detection, and satisfier recommendation.

The goal of this research is to create the means to trigger the experience based on needs. However, the actual design and the AR experience’s purpose are left to the creators and designers of the experience. 6

Conclusions

To realize needs-based Augmented Reality, it is essential to capture the different human needs using computing methods. In this paper, the focus is on the automatic detection of basic needs, and the use of those needs to trigger AR experiences. This research proposes that with advancing sensor technology, basic human needs are viable triggers for augmented reality experiences. The goal is to create a novel paradigm and provide developers with more options to personalize AR experiences. A theoretical framework that reviews and connects the conceptual basis of the research problem is presented.

First, a summary of technologies and classifications of AR is provided. A review of existing platforms is also provided.

Second, a discussion on a theoretical framework leading to the foundation of the proposed concept. This is followed by a need analysis and sensor analysis.

Third, a scenario is presented to demonstrate the proposed concept, followed by the application process based on the scenario.

The key and novel contribution of this research is the proposal to incorporate basic human needs as triggers of augmented reality applications which is significant and necessary in various domains. For pervasive computing, the combination expands the variety of research and enables the personalization of experiences. This research is also practical in marketing, entertainment, and ambient assisted living.

The proposed framework established based on previous research focuses on needs analysis. Future work will concentrate on building a user profile that supports the proposals in this study and testing the framework using a system implementation with a vision of achieving needs-based AR.

References

1. Ronald T. Azuma : A survey of augmented reality . Presence: Teleoperators & Virtual Environments , 6 , 355 - 385 ( 1997 ). doi: 10 .1162/pres. 1997 . 6 .4.355.

2. Carmigniani J. , Furht B. , Anisetti M. , Ceravolo P. , Damiani E. , Ivkovic M.: Augmented reality technologies, systems and applications . Multimedia Tools and Applications , 51 , 341 - 377 ( 2011 ).

3. Azuma R. , Baillot Y. , Behringer R. , Feiner S. , Julier S. , MacIntyre B.: Recent advances in augmented reality . IEEE Computer Graphics and Applications , 21 , 34 - 47 ( 2001 ).

4. Dunleavy M. , Dede C. : Augmented reality teaching and learning . In: Handbook of research on educational communications and technology , pp. 735 - 745 . Springer ( 2014 ).

5. Patricio J. M. , Costa M.C. , Carranca J.A. , Farropo B. : SolarSystemGO-An augmented reality based game with astronomical concepts . 2018 13th Iberian Conference on Information Systems and Technologies (CISTI) . IEEE , 1- 3 ( 2018 ).

6. Waltner G. , Schwarz M. , Ladstätter S. , Weber A. , Luley P. , Bischof H. , Lindschinger M. , Schmid I. , Paletta L.: Mango-mobile augmented reality with functional eating guidance and food awareness . International Conference on Image Analysis and Processing . Springer, 425 - 432 ( 2015 ).

7. Khor W. S. , Baker B. , Amin K. , Chan A. , Patel K. , Wong J .: Augmented and virtual reality in surgery-the digital surgical environment: Applications, limitations and legal pitfalls . Annals of translational medicine , 4 ( 2016 ).

8. Frigo M. A. , da Silva E.C. , Barbosa G.F. : Augmented reality in aerospace manufacturing: A review.“ . Journal of Industrial and Intelligent Information , 4 ( 2016 ).

9. Hammady R. , Ma M. , Temple N. : Augmented reality and gamification in heritage museums . Joint International Conference on Serious Games . Springer, 181 - 187 ( 2016 ).

10. Boy G. A. , Platt D.: A situation awareness assistant for human deep space exploration . International Conference on Human-Computer Interaction . Springer, 629 - 636 ( 2013 ).

11. Han D. , tom Dieck M.C. , Jung T. : User experience model for augmented reality applications in urban heritage tourism . Journal of Heritage Tourism , 13 , 46 - 61 ( 2018 ).

12. Zhou F. , Duh H.B. , Billinghurst M.: Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR . 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality . IEEE, 193 - 202 ( 2008 ).

13. Human S. , Fahrenbach F. , Kragulj F. , Savenkov V. : Ontology for representing human needs . Knowledge Engineering and Semantic Web . Springer International Publishing, Cham, 195 - 210 ( 2017 ).

14. Livingston M. A. : Evaluating human factors in augmented reality systems . IEEE Computer Graphics and Applications , 25 , 6 - 9 ( 2005 ).

15. Juan C. , YuLin W., Wei S. : Construction of interactive teaching system for course of mechanical drawing based on mobile augmented reality technology . International Journal of Emerging Technologies in Learning , 13 ( 2018 ).

16. Kim K. , Billinghurst M. , Bruder G. , Duh H.B. , Welch G.F. : Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR ( 2008 - 2017 ). IEEE Transactions on Visualization and Computer Graphics , 24 , 2947 - 2962 ( 2018 ).

17. Edwards-Stewart A. , Hoyt T. , Reger G. : Classifying different types of augmented reality technology . Annual Review of CyberTherapy and Telemedicine , 14 , 199 - 202 ( 2016 ).

18. Bower M. , Howe C. , McCredie N. , Robinson A. , Grover D. : Augmented reality in education-cases, places and potentials . Educational Media International, 51 , 1 - 15 ( 2014 ).

19. He Z. , Wu L. , Li X.R. : When art meets tech: The role of augmented reality in enhancing museum experiences and purchase intentions . Tourism Management , 68 , 127 - 139 ( 2018 ).

20. Serravalle F. , Ferraris A. , Vrontis D. , Thrassou A. , Christofi M.: Augmented reality in the tourism industry: A multi-stakeholder analysis of museums . Tourism Management Perspectives , 32 , 100549 ( 2019 ). doi:https://doiorg.ezproxy. cc.lut.fi/10 .1016/j.tmp. 2019 . 07 .002.

21. Billinghurst M.: Augmented reality in education. New horizons for learning , 12 , 1 - 5 ( 2002 ).

22. Chen P. , Liu X ., Cheng W., Huang R.: A review of using augmented reality in education from 2011 to 2016 . In: Innovations in smart learning, pp. 13 - 18 . Springer ( 2017 ).

23. Fan H. , Poole M.S. : What is personalization? perspectives on the design and implementation of personalization in information systems . Journal of Organizational Computing and Electronic Commerce , 16 , 179 - 202 ( 2006 ).

24. Zimmermann A. , Specht M. , Lorenz A. : Personalization and context management. User modeling and user-adapted interaction , 15 , 275 - 302 ( 2005 ).

25. Paavilainen J. , Korhonen H. , Alha K. , Stenros J. , Koskinen E. , Mayra F. : The pokémon GO experience: A location-based augmented reality mobile game goes mainstream . Proceedings of the 2017 CHI conference on human factors in computing systems. , 2493 - 2498 ( 2017 ).

26. Tahara T. , Seno T. , Narita G. , Ishikawa T. : Retargetable AR : Context-aware augmented reality in indoor scenes based on 3D scene graph. 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) . IEEE, 249 - 255 ( 2020 ).

27. Ghadiri N. , Nematbakhsh M.A. , Baraani-Dastjerdi A. , Ghasem-Aghaee N.: A context-aware service discovery framework based on human needs model . Service-Oriented Computing - ICSOC 20 . Springer Berlin Heidelberg, Berlin, Heidelberg, 404 - 409 ( 2007 ).

28. Abramovich A. : Theory of needs and problems . ( 2020 ).

29. Zhu Y. , Chen H. : Social media and human need satisfaction: Implications for social media marketing . Business Horizons , 58 , 335 - 345 ( 2015 ). doi://doiorg.ezproxy. cc.lut.fi/10 .1016/j.bushor. 2015 . 01 .006.

30. Houghton D. , Pressey A. , Istanbulluoglu D. : Who needs social networking? an empirical enquiry into the capability of facebook to meet human needs and satisfaction with life . Computers in Human Behavior , 104 , 106153 ( 2020 ). doi:https://doi-org. ezproxy.cc.lut.fi/10 .1016/j.chb. 2019 . 09 .029.

31. Max-Neef M.A. : Human scale development . Apex Press, New York u.a ( 1991 ).

32. Yahya M. , Dahanayake A. : A needs-based personalization model for context aware applications . Frontiers in Artificial Intelligence and Applications , 292 , 63 - 82 ( 2016 ).

33. Max-Neef M. , Elizalde A. , Hopenhayn M. : Development and human needs. Reallife economics: Understanding wealth creation , , 197 - 213 ( 1992 ).

34. Myers D.G. : Psychology. Worth Publishers, New York ( 2013 ).

35. Andreoni G. , Standoli C.E. , Perego P. : Defining requirements and related methods for designing sensorized garments . Sensors 16 ( 2016 ). doi: 10 .3390/s16060769.

36. Cem Kaner J. D. : An introduction to scenario testing . Florida Institute of Technology, Melbourne, , 1 - 13 ( 2013 ).

37. Yahya M. A. , Dahanayake A. : Augmented reality for human needs: An ontology . Frontiers in Artificial Intelligence and Applications , 333 , 275 - 294 ( 2020 ).