Immersive journalism: Extant corpus and future agenda

Abstract

The goal of journalism is to disseminate information to people as accurately and holistically as possible. Therefore, unsurprisingly, the recent advances in multisensory and multimodal technologies have spawned a new research area of immersive journalism (IJ). It is believed that the more holistic and engrossing media experiences afforded by virtual, augmented, and mixed reality technologies can lead to more comprehensive information internalization, both cognitively and emotionally. The literature has increasingly started to propagate since approximately 2016 onward. Therefore, while the domain is still only in its inception phase, and while the related technologies continue to develop, it is already mature enough to both look backwards to what has already been done and forwards to delineate future research agenda. In this review, we investigate what has been investigated in the extant corpus, including: methods and data, technologies and types of content in experiment settings, and dimensions related to the resulting experiences.

Keywords

• virtual reality
• mixed reality
• 360 video
• journalism
• perspective taking
• literature review

Introduction

Immersive journalism (IJ) is becoming increasingly available and popular, primarily due to The Guardian and the New York Times (NYT) and their 360-degree video sections. Additionally, NYT had sent out over a million of Google’s Cardboard VR goggles in 2016, introducing their readers to the medium. However, the idea of merging new technologies and journalism was introduced already in 2010 [A5] with the aim of creating engaging experiences through placing the user in another’s shoes and bridging the gap between you and them or there. Originally, it was envisioned as an immersive virtual reality experience, including embodiment, interactivity, and freedom of movement, which would help represent others’ experience and emphasize promoting empathy and affective understanding [A5, A21]. This trend can also be seen simply as an extension of previous use of new technologies, with the same purpose of enhancing user engagements through the development of newsgames [ 1, 2 ], and the overall drive of gamification [ 5 ].

Closely connected to this idea of engagement is the notion of collapse of compassion [ 8 ] which describes the global phenomenon where the distant suffering of many is not experienced deeply, nor even objectively understood, in terms of its collective individual effects. Instead, news pieces dealing with these topics are often taken as merely information, with possibly some experienced uneasiness. Although there is a psychological defensive reason for this as no one can carry all the burden of the world, it also hinders compassion and action for the betterment of humanity. Immersive journalism emerged as an attempt to use new technologies such as virtual reality to bridge this gap. Virtual reality (VR) is sometimes referred to as an “empathy machine“, particularly in popular discourse [ 9 ], presenting a technology that might be able to enhance human connection by allowing an individual to cross space or even time and walk in another’s shoes [6, A8, A17]. Similarly, Carne y Arena by Alejandro Iñárritu is a unique dramatic experience of which a large portion is in VR and has won a special Oscar in 2017, indicating that there is both recognition and faith in the development of similar projects. It places the user as an immigrant at the U.S.-Mexico border with all the hardships that surround similar feats, diminishing the distance between the user and the immigrant through intuition [ 7 ]. However, as producing fully immersive pieces is still resourceheavy, the majority of IJ available to the general public is in the form of 360-degree videos that are viewable on-screen (2D surface) or in mobile VR which provides further technological immersiveness [A7].

Despite interest and eagerness in the potential of VR, IJ became visible as an interest of academics only from the year 2016 onward, after both an increase in the production of 360-videos and NYT’s initiative which helped popularize VR and the content. Further development and better affordability and accessibility of both the technology and content is heightening interest in related themes, both in public and academia. The domain seems to still be only in its inception considering its breadth and the different possible types of content (360-video, interactive, digital reproduction, filmed, etc.) and technologies (screen size, mobile VR, immersive (embodied) VR with all its variations).

Thus, this review aims at providing an overview of the field, to identify pitfalls and gaps, as well as delineate possible future research avenues. It presents several key points in the literature: methods and data, technologies and types of content in experiment settings, and studied outcomes. 2

Method

This study relies on Webster and Watson’s [ 10 ] process for systematic literature reviews. It supposes a reproducible research consisting of a rigorously defined search, transparent inclusion criteria of the retrieved papers, and a pre-defined concept matrix for analyzing the selected body of literature.

The search for literature was conducted during April 2019 in Scopus, one of the largest databases of peer-reviewed publications. Exploratory searches by the authors had been conducted two years prior with the aim of getting acquainted with the field and terminology. This prior knowledge was used in constructing the search string, which was composed out of two sections: one describing journalism, and the other describing immersive. Both sections were built using multiple related terms and employing wild cards for a comprehensive search where appropriate. Thus, the following search string was construed:

(journalis* OR news) AND (VR OR "virtual reality" OR HMD OR immers* OR embod* OR 360)

A total of 796 results were retrieved, including conference papers, journal articles, and book chapters. The first round of reviews included scanning the retrieved abstracts of the final results, leaving 41 results. Publications were discarded due to referring to virtual reality in the wider sense as digital environments, not being related to news or journalism, or for only mentioning the field in passing. Although immersive news could, additionally, entail augmented reality (AR) applications, and the search string supported that premise, no such applications were found during the search.

Four full papers out of the 41 that were chosen for full analysis could not be accessed, leaving 37 full manuscripts. During this stage, 13 additional publications were discarded for the same exclusion criteria listed above, leaving 24 publications deemed suitable for inclusion in this review. Finally, backward and forward reference searches were conducted revealing 3 new manuscripts. The analysis of the final 27 results was performed using a concept matrix pre-determined by the authors. 3

Results

The analysis was conducted using an adapted concept matrix [ 10 ] and all the results are presented by these investigated aspects of the literature. All of the papers that were analyzed in their entirety (N = 27) were individually coded according to the following pre-defined bases for the matrix: 1. Publication type and year 2. Terminology used 3. Type of study and methods 4. Presented comparisons between media technologies or types of content, and 5. Studied outcomes

Whilst the categories of some of these points for investigation (1 and 3) were predictable (e.g. whether a study is empirical or non-empirical), others (2, 4, and 5) were further defined during the analysis itself. For example, all of the outcome variables that were found in the reviewed body of literature were recorded under Studied outcomes as they appeared in the papers. Using this method, not solely particular pre-defined outcomes were reviewed, but rather all that were studied in this domain and were documented in the retrieved literature. 3.1

Emergence of a field

Before year 2017 the only published articles are from De la Peña and colleagues [A5] which introduced immersive journalism in 2010 and a lone conference paper from 2016 (A20) which drafted the future possibilities of journalism in VR. More prolific academic study of the field started only in 2017 (n = 12) and the number of publications is on a significant rise. During the first quarter of year 2019 only (n = 13), the number of peer-reviewed studies had already reached the total number of those from 2018 (n = 13). 3.2

Terminology

Term

360-degree (video) journalism 360-video news 360-degree VR VR news VR journalism Immersive news Immersive journalism

In most cases, the authors used the term immersive journalism for 360-degree videos on screen or in mobile VR, and for immersive virtual reality applications. However, there are inconsistencies with the terminology which might stem from and contribute to the high granularity of the field with articles scattered in a variety of venues. Therefore, familiarity with the used terminology should ease the cohesion of the research and with time consolidation of the currently vague terminology. Table 1 documents the terms found in the literature. It is worth noting that, while those containing 360 in their name are limited to the 360-degree videos, it is not always clear what is considered under immersive journalism, immersive news, and VR news and VR journalism. These can, but do not necessarily, denote both immersive and mobile VR content. Furthermore, 360-video news has only been used in conjunction with immersive news (A3), and immersive news only in conjuction with immersive journalism (A23). Several other studies have used different terms together with no particular pattern (A8, A11, A13, and A25). 3.3

Types of studies and methods

All of the manuscripts were classified either as empirical (n = 17) or non-empirical (n = 10), and according to the methods used. In the further sections of this review, only the results of the empirical studies are examined and presented. These are further noted as quantitative (n = 8), qualitative (n = 6), or mixed methods (n = 3) studies.

The majority of the quantiative studies analyzed the data using statistical inference (i.e. hypothesis testing), and only 3 (A3, A13, and A15) presented it using solely descriptives such as mean values. Almost all of the measurements were collected via psychometric tests. Interestingly, one study analyzed users’ behavior using objective, publicly available data from the streaming platform YouTube (A27).

Qualitative studies mainly investigated the content (A4, A10, A22, and A26), or conducted interviews or focus groups with users (A6, and A10) or practicioners (A1). 3.4

Comparisons

Treatments

VR embodied 360, cardboard VR 360, screen 2D, mobile VR 2D, screen Article No treatment in lower quality and needs to be held to the head; screen – a common label for 2D screens, regardless of the size and technological specifications.

Empirical studies often employed comparison of the effects of different media and/or media technologies. The most represented comparisons employ 360-degree videos in mobile VR on one side, and a variety of treatments on the other. The least studied in comparable settings are immersive VR (A19, A21, and A23), as imagined immersive journalism, and written articles (A24), as a more traditional form of journalism. 3.5

Studied outcomes

Category

Affect

Cognition and attitudes

Engrossment

Behavior

For better readability of the output, the studied outcomes are loosely divided into five categories – affect, cognition and attitudes, engrossment, behavior, and production and journalistic norms. There were coined by the authors and do not represent concrete analytical value. On the other hand, categories in the Measure(s) column were taken directly from the investigated literature and represent their measured outcomes. Unsurprisingly, users’ engrossment is studied the most, with (tele)presence as the most studied outcome, However, it is highly granulated across different media and media technologies, as can be seen from Section 3.4. Only one study employed a measure of attitudes towards the topic of the content (A6), and two tracked whether users showed interest beyond the experiment and have followed up to learn more (A21 and A23). Finally, the category of production and journalistic norms entails studies on, for example, use of annotations (A14) or subtitles (A12). 4

Discussion

This review is the first attempt to summarize empirical research on the topic of immersive journalism, which is gaining increasing interest in academia. The field is multidisciplinary and highly topical, and studies are greatly dispersed and disconnected, as can be seen throughout this study, starting with the inconsistent terminology. it is hoped that this review will serve as a step toward consolidating the field by representing the state of the art and identifying gaps and points for further research.

However, it should only be taken as a stepping stone toward a more nuanced one. Considering the speed at which the field is expanding, it is necessary that it is updated and expanded when possible so as to provide more solid grounds for examining the effects of immersive journalism. 4.1

Identified gaps and future directions

Already from this short review there are several issues and gaps identified in the literature. Some are minor but expected as they mostly stem from the field being novel and multidisciplinary; others pertain to methodological drawbacks and overlooked central concerns in immersive journalism. 1. Authors rarely define the variety of terms used, making it difficult to denote what immersive journalism is and what it is not. Some more clearly denote it as embodied immersive VR experiences (A5, A21, A23), but it would seem that the majority refers to 360-videos commonly available to the general public. A more transparent approach while at the same time contextualizing the research in the wider field could aid in structuring it at this crucial time of growth. 2. Even though not limited to this field [ 4 ], quantitative data and results are not always well and clearly presented, succumbing to various misconceptions when drawing conclusions. It is of particular relevance here, because of the breadth of the technologies as well as content, to diligently lay out both descriptive and inferred results. This practice would allow for meta-analyses that would additionally enable reviewers to gather higher level implications from the studies. 3. Similarly, as seen from Table 3, there are rarely multiple studies employing same pairs of treatments, as out of sixteen comparisons only three pairs are to an extent comparable. Instead, it would be beneficial if treatments are replicated, while for example using different type or topic of the stimuli. Such a practice is incremental, but necessary for strengthening the findings. 4. Furthermore, as the VR technology is becoming more available and fully immersive experiences gaining more popularity, it is imperative that these are investigated in a timely manner beside the 360-videos. On the other hand, a comparison of immersive and traditional, written news pieces has only been found in one example (A24), revealing a dearth of knowledge in how they compare to each other. 5. Only a handful of the reviewed studies investigated palpable outcomes of these immersive experiences. This is particularly unexpected in the light of immersive journalism’s aspirations to engage and induce empathy, as well as the popularization and recognition of similar content in the artistic domain through Carne y Arena [ 7 ]. The attitudinal and behavoural effects are vastly hypothesized but rarely investigated. Considering that empathy is a highly problematic concept [ 3 ], it might be more beneficial to examine measurable outcomes such as attitudinal changes (A6) and following-up (see A21 and A23). Notable by their absence are longitudinal and behavioral studies showing whether these possible preliminary outcomes can truly affect an individual and the society [ 9 ]. 6. Finally, the most crucial and largest gap in the reviewed empirical literature on immersive journalism is the lack of scrutiny of users’ media literacy - in particular when it comes to critical evaluations of the consumed content. Although it can be argued that there are benefits to the emphasized individuality and the subjective experience of immersive journalism [ 7, 8, 9 ], there should also be a counterbalance ensuring that the public is at the same time informed and vigilant. Future empirical studies should weigh these two aspects – subjectivity and objectivity – in order to obtain a more comprehensive account of the effects and ethics of immersive journalism. 4.2

Limitations of the review

As with any review, there are certain drawbacks to this one that ought to be noted. With a wide field such as immersive journalism there is no way of making certain that all published studies are taken into account despite the best efforts in constructing the search string. However, it is meant as a broad overview of the state of the field and its findings rather than aiming at one particular aspect in detail. Moreover, due to the length constraints, it was mainly focused on empirical research while leaving conceptual and theoretical discussions in the background. Finally, even though the number of publications included in this review is not negligible, there are not enough comparable studies that would enable a deeper discussion of the results and whether or not immersive journalism truly is more effective in engaging users and bringing about positive change.

Acknowledgments References

This work was partially supported by Business Finland (5479/31/2017).

Reviewed publications

References

1. Bogost , I. : Persuasive games . MIT Press ( 2007 ).

2. Bogost , I. , Ferrari , S. , Schweizer , B. : Newsgames . Journalism at play. MIT Press ( 2010 ).

3. Cuff , B.M. , Brown , S.J. , Taylor , L., Howat , D.J..: Empathy: a review of the concept . Emotion Review 8 ( 2 ), 144 - 53 ( 2016 ).

4. Greenland , S. , Senn , S.J. , Rothman , K.J. , Carlin , J.B. , Poole , C. , Goodman , S.N. , Altman , D.G. : Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations . European journal of epidemiology 31(4) , 337 - 350 ( 2016 ).

5. Hamari , J.: Gamification. In Ritzer, G., Rojek , C. (eds.), The Blackwell Encyclopedia of Sociology . New York John Wiley & Sons ( 2019 ).

6. Oh , S.Y. , Bailenson , J. , Weisz , E. , Zaki , J.: Virtually old: Embodied perspective taking and the reduction of ageism under threat . Computers in Human Behavior 60 , 398 - 410 ( 2016 ).

7. Raessens , J.: Virtually Present, Physically Invisible: Alejandro G. Iñárritu's Mixed Reality Installation Carne y Arena . Television & New Media 20 ( 6 ), 634 - 648 ( 2019 ).

8. Slovic , P. : If I look at the mass I will never act: Psychic numbing and genocide . In Emotions and risky technologies , pp. 37 - 59 . Springer, Dordrecht ( 2010 ).

9. TED. How virtual reality can create the ultimate empathy machine . https://www.ted.com/talks/chris_milk_ how_virtual_reality_can_create_the_ultimate_empathy_machine, last accessed 2 /11/ 2019 .

10. Webster , J. and Watson , R.T.: Analyzing the past to prepare for the future: Writing a literature review . MIS quarterly , pp. xiii - xxiii ( 2002 ).

A1. Aitamurto , T. : Normative paradoxes in 360° journalism: Contested accuracy and objectivity . New Media & Society 21 ( 1 ), 3 - 19 ( 2018 ).

A2. Baía Reis , A. , Coelho , A. F. V. C. : Virtual Reality and Journalism . Digital Journalism 6 ( 8 ), 1090 - 1100 ( 2018 ).

A3. Brautović , M. , John , R., Potrebica , M. : Immersiveness of News: How Croatian Students Experienced 360-Video News . In Lecture Notes in Computer Science 2017 , pp. 263 - 269 . Springer International Publishing ( 2017 ).

A4. Chen , T.Y., Liu , H.C. , Hsu , C.Y.: Film language analysis in society news-A case study of The New York Times . In Pacific Neighborhood Consortium Annual Conference and Joint Meetings (PNC) 2017 . IEEE ( 2017 ).

A5. De la Peña , N. , Weil , P. , Llobera , J. , Spanlang , B. , Friedman , D. , Sanchez-Vives , M. V. , Slater , M. : Immersive Journalism: Immersive Virtual Reality for the First-Person Experience of News . Presence: Teleoperators and Virtual Environments 19 ( 4 ), 291 - 301 ( 2010 ).

A6. Delmazo , C. : The Use of 360-Degree Video to Provide an Alternative Media Approach to Paralympic Sports . In Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering , pp. 200 - 205 . Springer International Publishing ( 2019 ).

A7. Hardee , G. M. , McMahan , R. P.: FIJI: A Framework for the Immersion-Journalism Intersection . Frontiers in ICT 4 ( 2017 ).

A8. Hassan , R.: Digitality, Virtual Reality and The “Empathy Machine” . Digital Journalism 1 - 18 ( 2019 ).

A9. Hendriks Vettehen , P. , Wiltink , D. , Huiskamp , M. , Schaap , G. , Ketelaar , P. : Taking the full view: How viewers respond to 360-degree video news . Computers in Human Behavior 91 , 24 - 32 ( 2019 ).

A10. Jones , S. : Disrupting the narrative: immersive journalism in virtual reality . Journal of Media Practice 18 ( 2-3 ), 171 - 185 ( 2017 ).

A11. Kang , S. , O 'Brien , E. , Villarreal , A. , Lee , W. , Mahood , C. : Immersive Journalism and Telepresence . Digital Journalism 7 ( 2 ), 294 - 313 ( 2018 ).

A12. Li , K. , Yang , D. , Ji , S. , Liu , L. : The Impacts of Subtitles on 360-Degree Video Journalism Watching . In International Joint Conference on Information, Media and Engineering (ICIME) 2018 . IEEE ( 2018 ).

A13. Mabrook , R. , Singer , J. B. : Virtual Reality, 360 ° Video, and Journalism Studies: Conceptual Approaches to Immersive Technologies. Journalism Studies 1 - 17 ( 2019 ).

A14. Meira , J. , Marques , J. , Jacob , J. , Nobrega , R. , Rodrigues , R. , Coelho , A. , de Sousa , A. A. : Video annotation for immersive journalism using masking techniques . In 23rd Portuguese Meeting on Computer Graphics and Interaction (EPCGI) . IEEE ( 2016 ).

A15. Pérez Seijo , S. : Immersive Journalism: From Audience to First-Person Experience of News . In Advances in Intelligent Systems and Computing , pp. 113 - 119 . Springer International Publishing ( 2016 ).

A16. Pérez-Seijo , S. , López-García , X. : Five Ethical Challenges of Immersive Journalism: A Proposal of Good Practices' Indicators . In Advances in Intelligent Systems and Computing , pp. 954 - 964 . Springer International Publishing ( 2019 ).

A17. Sánchez Laws , A. L. : Can Immersive Journalism Enhance Empathy? . Digital Journalism 1 - 16 ( 2017 ).

A18. Sánchez Laws , A. L. , Utne , T. : Ethics Guidelines for Immersive Journalism . Frontiers in Robotics and AI 6 , 28 ( 2019 ).

A19. Shin , D. , Biocca , F. : Exploring immersive experience in journalism . New Media & Society 20 ( 8 ), 2800 - 2823 ( 2017 ).

A20. Sirkkunen , E. , Väätäjä , H. , Uskali , T. , Rezaei , P. P.: Journalism in virtual reality . In Proceedings of the 20th International Academic Mindtrek Conference on - AcademicMindtrek '16 . ACM Press ( 2016 ).

A21. Slater , M. , Navarro , X. , Valenzuela , J. , Oliva , R. , Beacco , A. , Thorn , J. , Watson , Z. : Virtually Being Lenin Enhances Presence and Engagement in a Scene From the Russian Revolution . Frontiers in Robotics and AI 5 , 91 ( 2018 ).

A22. Soler-Adillon , J. , Sora , C. : Immersive Journalism and Virtual Reality . In Interaction in Digital News Media , pp. 55 - 83 . Springer International Publishing ( 2018 ).

A23. Steed , A. , Pan , Y. , Watson , Z. , Slater , M. : “We Wait” -The Impact of Character Responsiveness and Self Embodiment on Presence and Interest in an Immersive News Experience . Frontiers in Robotics and AI 5 , 112 ( 2018 ).

A24. Sundar , S. S. , Kang , J. , Oprean , D. : Being There in the Midst of the Story: How Immersive Journalism Affects Our Perceptions and Cognitions . Cyberpsychology, Behavior, and Social Networking 20 ( 11 ), 672 - 682 ( 2017 ).

A25. Van Damme , K. , All , A. , De Marez , L., Van Leuven , S. : 360 ° Video Journalism : Experimental Study on the Effect of Immersion on News Experience and Distant Suffering . Journalism Studies 1 - 24 ( 2019 ).

A26. Vázquez-Herrero , J. , & López-García , X. : Immersive Journalism Through Mobile Devices: How Virtual Reality Apps Are Changing News Consumption . In Advances in Intelligent Systems and Computing , pp. 3 - 12 . Springer International Publishing ( 2017 ).

A27. Wang , G. , Gu , W. , Suh , A. : The Effects of 360-Degree VR Videos on Audience Engagement: Evidence from the New York Times . In HCI in Business, Government, and Organizations, pp. 217 - 235 . Springer International Publishing ( 2018 ).