AUGMENTED REALITY AND VISUAL ARTSMaria Papadopouloumaria_artistry@yahoo.gr1Ioannis Deliyannis0Assistant Professor, Department of Audio and Visual Arts, Ionian UniversityMA, Department of Audio and Visual Arts, Ionian University
This research explores the use of augmented reality technologies in the field of visual arts, focusing on the way that visitors and artwork interact. The bibliographic research focuses on exploring the definitions of augmented reality and interaction. The historical overview defines the two above-mentioned fields and looks into their connection with art, via exploration of their visual characteristics. We present the implementation methodology for such enhanced artwork and the process of interaction with the observer. Our research extends beyond typical presentation and citation of augmented and interactive visual works as it sets two main objectives a) discuss the way the artistic work which use augmented reality technologies interacts with the visitor, as well as b) present how human senses are stimulated through visual art work when new interactive augmented reality technologies are employed.
software named “ARART” to end-to-end solutions such as “HP Reveal”
(formerly known as Aurasma)
(List of augmented reality software, 2019)
. Our
research questions moves beyond the technical elements of the software to the
user-experience sector: 1) How is the interaction between visual art and
augmented reality and the visitor implemented? 2) Which of the five senses are
stimulated through the artwork? 3) How are those senses stimulated?
Augmented Reality
Augmented Reality AR is a way to link physical items from the environment to
virtual content. The main difference between AR and VR lies in the fact that AR
integrates the user into the physical environment in real time, offering the
possibility of improving all the five senses even if AR applications usually use only
the vision, while VR isolates the user from it. Augmented reality was created to
enhance the view of the natural world and this is accomplished by using
computers, sensors and image processing technologies
(Kipper & Rampolla, 2012)
.
AR techniques appeared in 1962 when Morton Heilig designed a machine
which immerse the user in a full range of sensory stimulation while
watching short films. This machine – sensorama- is the first known multi sensory
technology. Followed in the 1970s, by the establishment of Videoplace by
Myron Krueger. Krueger created a virtual reality and interactive art environment
in which by using augmented reality, users could interact with virtual objects
(Kipper & Rampolla, 2012)
. These two events are what triggered the
revolution in the development of AR technologies and applications. As technologies
evolve, in 1994 Julie Martin created a theatrical production called “Dancing In
Cyberspace”. During the show and while the dancers performed acrobatics on
the stage, virtual objects appeared on them and around them
(Dils & Cooper
Albright, 2001)
. Today, such theater and performance productions are common
and Projection-based AR techniques are used. In the field of art this technology
is called “Projection Mapping” that covers a smaller field than Projection-based
AR
(Jaewoon et al., 2015)
. In recent years, researchers managed to develop
techniques that these three-dimensional projections could applied not only to
fixed objects but also to moving objects, as this technique requires automatic
alignment of accurate moving image objects
(Jaewoon, et al., 2015)
.
Interaction and Art
Active participation of the observer in the formation of an artwork and the need
for his presence for the development or existence of the artwork itself is an
embodiment of interaction. AR clearly enables this to be implemented as it
provides a tool to interact with artwork. Of course this concept is not something
new, as by researching art history, it emerges that during the period of the
Futurism and Dadaism, there was a need from artists to physically and emotionally
engage the public in their artwork.
Artists of that period (Futurism and Dadaism) sought the interaction, in
other words the reaction of the public towards their artistic action. At this time,
the Happenig’s predecessors could be identified along with the first
attempt to integrate the visitor into the artistic work. Then follows Allan Kaprow
(Charalambidis, 1995), the father of Happening, who was the first to engage the
visitors actively in his artwork. His idea was born when he noticed in the
environments he was creating that each visitor was becoming part of his artwork.
Based on this observation, he decided to give to every visitor the opportunity to
be actively involved in shaping the project. This was led to Happening. Today,
the interaction of a project with the observer can be achieved through the use of
technology and can become more complex and multileveled than it could have
happened with Happening.
Interaction and AR
For artists, how the observer feels or what he/she experience, is a key factor for
their artwork. This goes beyond the logic of comparing one artwork item with
the other or in other words ‘’this piece sounds like or looks like’’. Specifically,
interactive digital art is purely about interactive experience
(Edmonds, 2014)
.
The way in which visual art interacts with the public is a critical part of the
essence of the artist and his work. The reaction towards the project is the core of it
and is more important than any other aspect in an artwork. The creative practice
of the artist who chose this path is quite different from that of a painter. In the
case of interactive art, the public’s response is the key element that will concern
the artist, rather than the color or the perception of perspective. The public’s
commitment to the project is based on what the public does, how it interacts
with the project, and so forth
(Edmonds, 2010)
. A basic part of the interactive
arts is also part of the immersion of the visitor.
In the art of technology, observer’s immersion does not only function as a
visual-spatial illusion but also as the sense of conscious immersion in a fake
world and/or artistic processes. This is particularly true in the case of interactive
art, which requires awareness of the things that are offered through interaction
and focuses on the reactions of the observer who is actively involved. Here, the
focus is not on escaping from everyday life, but on the emotional and mental
intensity of the experience. In interactive art, the way that the
participant-observer acts is important for him to perceive the artwork
(Kwastek, 2015)
. The
need to study the interaction between art and public has revealed the need to
create a study model.
Zafer Bilda in 2011 developed a model for studying the connection of the
public with the visual interactive artwork. This was the result of the research he
conducted when he examined the interaction of art with the public. He found
that, the observer’s engagement with the project shifted from involuntary
actions to deliberate actions that led the observer into a sense of control. In some
projects observer’s involvement continues through developing levels of
exploration and uncertainty. It should be noted that Bilda has recognized four phases
of interaction in an interactive visitor system: adaptation, learning, anticipation
and deeper understanding
(Edmonds, 2014)
.
Through the above research it is observed that a series of issues concerning
the audience experience are important to the interactive artist and the research
on these points is an important part for the creative process of an artwork. When
creating an interactive visual artwork, artists should take into account the
engagement processes and engagement types of the visitors involvement
(Edmonds, 2010)
.
Aside from the above, which mainly concerns the artist and what needs to
be taken into account in order to activate and engage the visitor with his
interactive digital artwork. The key question remains how the visitor is affected
when he/she receives the above information from an interactive artistic work
that is connected with some real-time technology. Part of the answer to this
question comes from the interactive artwork created by Diaa Ahmed Mohamed
Ahmedien (2017)
who created a neural interactive work of art in the form of a
holographic puzzle in an attempt to expand the functional role of brain activity
in interactive artistic processes.
The experiment connects the imaging system of the puzzle pieces with the
electrical activities of the participant’s brain by the means of an
electroencephalography system. This experiment reveals the effect of the functional expansion
of the participants’ nerve responses and the ability to quantitatively analyze the
interactive processes in an operational interpretation of the neural dimension in
the interactive arts. If the brain is properly stimulated by the participant then the
holographic tiles will appear and the participant can solve the puzzle. As a result,
participants recognize how the system works and exercise self-observa- tion and
self-control in order to manipulate the holographic pieces of the puzzle. Finally,
data from brain signals was summed up in comparative tables to reveal the
reciprocal interaction processes between each participant’s neurological
response and performance. This technique is called “Neurofeedback” and is a
technique that affects the nervous system of the brain, in which brain activity is
recorded using electrodes and is presented visually so that the person can
functionally know the state of his brain nerve signals and their performance.
Comparing Zafer Bilda’s method of analyzing the interaction of the visual
work and the steps that the visitor should take in order to engage with the
interactive work, the way that Diaa Ahmed Mohamed Ahmedien analyzes in his
research of the brain activity of each participant from his acquaintance with the
system as the solution of the interactive holographic puzzle, we see an analogy
of the steps and an increase in brain activity when the brain is in the final stage
of solving the puzzle. As the author says, in order to solve the puzzle, the brain
should produce Gamma waves that represent the state of hyper-brain activity
(Ahmedien, 2017)
. In this case, intense brain activity is been prodused by the
person through his interaction with the AR artwork.
AR applications have been applied to language studies, social sciences, the
arts and humanities, entertainment and recreation, as well as advertising and
marketing. Many studies have found that AR offers visitors interesting,
entertaining and exciting experiences as well as shocking sensations
(Chang et al.,
2014)
. As far as educational applications are concerned, it has been reported that
both teachers and students believe that AR not only promotes participation and
motivation but also creates a realistic and new learning environment by
combining both the real and the virtual
(Chang et al., 2014)
. Studies have shown
the effectiveness of using AR in education. The findings seem to suggest that it
has contributed to increased academic achievement and promoted positive
emotional experiences compared to traditional teaching
(Ibáñez, Di Serio, Villaran,
& Kloos, 2014)
.
Artwork
Nowadays, artists in order to make the interaction between the visual work and
the observer, make use of various technologies and, above all, AR technologies.
A well-known artist working with AR is Masayuki Akamatsu, an example of
his work from 2012 is the ‘’Uroboros Torch’’
(Masayuki, 2012)
. This exhibition
consists of eight paintings in which the augmented reality is achieved by the use
of a mobile phone. That is, the visitor has the ability to see through his cell
phone parts of the paintings come to life, also the tables are connected with
speakers that augment the reality for the visitor through the sound. It is observed
that through these technologies the visitor interacts with the project by
activating in addition to his vision, touch and hearing.
Artist John Craig Freeman also deals with the visual arts and the
combination of augmented and virtual reality unlike Akamatsu though, his works are
large-scale outdoor installations One of his recent works is “Mexico Border
Migration Stories” 2017
(Freeman, 2017)
. The artist through AR allows viewers
to be in two different worlds simultaneously offering mostly visual stimuli.
Another artist who deals with the political form of art and the use of
augmented reality is Patrick Lichty. In his “Love Bombers’’
(Lichty, 2012)
project
of 2012 made a visual art political statement against the NATO summit in a
Chicago demonstration where, over the protesters mob sing a cell phone the
observer could see virtual planes bombing the mob with virtual hearts. This
project connects a historical event to a geographical environment
(Lichty, 2014)
.
And in this project AR technologies provide visual stimuli to the public. This is
not the only artwork of this artist, he has made a series of works linking the
natural surroundings of Alaska with the present and the past criticizing
industries exploiting the mineral wealth and destroying the environment. As the artist
himself mentions in his work “Into the Wild / Virtual Kenai”of 2014, he portray
another form of conquest in the Anthropocentric Age
(Lichty, 2014)
. In this
way, his work falls within a historical context while still having a critical
position. In this work the artist also used a QR-code technology.
Similar political, social and environmental concerns produse the work of
Nathan Shafer ‘’Kenai Tapestry’’ of 2013-14. This artwork consists of a
largesized woolen tapestry that portrays the Kenai Fjords National Park in Alaska
and is connected with real-time augmentation techniques by providing
information on the location through the forest site. This artwork reflects the artist’s
concerns about nature and its destruction
(Lichty & Shafer, 2016)
.
Subsequently, a different form of AR is found in the artwork of Golan Levin
and Zachary Lieberman ‘’Footfalls’’
(Levin & Lieberman, 2006)
. In this
interactive projection the sounds that produced by visitors’ feet are detected by
microphones under the floor and used to determine the size and number of
virtual objects (bubbles) that fall from a height of six meters. The less number of
visitors experience the stumbling blocks, the more virtual elements fall from the
top by projection. Also, visitors using their silhouettes can “move” the projected
bubbles around them. It is noticed that when the visitor joins in (with all his
body within the artwork and) his movements and the way he walks, are those
elements that influence and shape the artwork. The visitor literally becomes one
with this artwork, as he physically becomes part of his evolution, mainly by
visual stimuli.
In addition to the image and motion AR is also associated with sound. One
such example is the artwork of Mene Savasta Alsina and Hernan Kerlleevich
‘’Ahora’’
(Alsina & Kerllenevich, 2013)
which is an interactive sound
installation. This sonic artwork varies according to the movement of each visitor. In
other words, the presence of the visitor and his movement in the space modifies
each time the composition, in a sense re-writing each time this composition in a
different spacetime. This way, the visitor is physically integrated into the
artwork and remodes the project through its movement. The stimuli that a visitor
receives are mainly acoustic.
Another work worth mentioned is the ‘’water shapes’’
(Papadopoulou,
2018)
, which is an audiovisual - visual artwork that deals with the subject of
water and creates a visual interactive environment that connects sound, image
and augmented reality in order to help the audience interact and immerse. The
project is divided into two parts. In the first part, nine photographs are
presented in which the use of a mobile phone achieves the augmentation of reality, as
the visitor has the opportunity to see via his mobile phone these photos come
alive. In the same space (part) there is a video art that enhances the sound. In
the second part of the project, as the visitor moves into space, he activates a
sensor and a perfume diffusion mechanism. In this way the sense of smell of the
visitor is activated. As the visitor enters the second space of the work, light
diminishes and the only light sources at this point is a holographic projection
which is projected inside an inverted pyramid and an interactive projection with
a virtual rain in which when the visitor approaches, he sees himself inside and
his body drips up with water drops. The water drops follow the movement of the
visitor’s body in a rainy environment. Through holographic projection, a stable
rain sound is provided in the room. Through five distance sensors connected to
an Arduino system, different rain and water sounds are activated depending on
the visitor’s movement, each time setting up a different sound depending on
the traffic and the crowd of visitors. It is important to notice that through this
artwork an effort is made to activate the visual, tactile, acoustic and olfactory
senses of the visitor.
The Royal Academy of Art (KABK) and the Delft University of
Technology Foundation set up an AR workshop in which two related research projects
were developed. The aim of the first was to research and apply the AR technique
in the arts, design the technology and dissemination of the technology to the
creative industry. The aim of the second project was to combine AR with the
interaction and diffusion of technology into public institutes such as museums.
Through this collaboration, they have emerged a series of works that combine
art AR technologies interaction with the visitor and work
(Caarls, Jonker,
Kolstee, Rotteveel, & Eck, 2009)
. This research will only mention a 2007 artwork
titled “Out of the Blue’’, which is an augmented reality audiovisual
environment in which the exhibition area was transformed into a blue landscape where
elliptical shapes were floating and creating the landscape. As the visitor was
moving around, he could see and hear around him a various of flying objects
(Caarls, et al., 2009)
.
Finally, it should be noted that art and AR technologies are not tools
exclusively for visual artists. They can be tools of expression and artistic cre- ation
for students as well. An example (a protype one) is the four students of 1st
E.E.E.EK. Pilea-Chortiatis (Greek secondary special education school) who
they presented at the 10th Student Computer Science Conference of Central
Macedonia (2018) their visual work which was connected with AR
technologies. The students took their own photos and they edited them using
various techniques such as collage, or coloring, and they linked them through
AR application to their new atrworks. In this way they recreated their natural
environment. With the use of a mobile phone, the observ- er could see these
works.
Conclusion
Beyond the purpose and research questions raised in this paper, small-scale
historical references were sourced in order to connect the concepts of interaction,
AR and the arts. Historically the incorporation of AR technologies into the
visual artwork and the need for the artwork to interact with the visitor, a series of
artworks have been presented and analyzed. The artists with their artworks attempt
to incorporate the observer into them by activating his senses both physically
and psychologically.
Through this brief research it is realized that the visitor ceases to be just a
simple observer but acquires an active participation role in the project, by
formulating through his choices the form of the project he is visiting. It is also
observed that artists are trying to activate as much as possible the senses of each
visitor and not just his vision.
Most of the aforementioned projects note the need for an observer to use
his/her mobile phone devices so that he/she can read the augmentation of reality
in each project. In two of the examples, it is observed that through an interactive
projection the observer takes an active part in shaping the project by acting with
the projection through the motion of his body. A similar configuration of the
artwork, referred to the two examples mentioned was the sound work that was
presented using different forms, analogous to the crowd and movement of each
observer. Apart from the visual, tactile and sonic stimuli that were reported as
offered examples to the visitors, one of the projects offered olfactory instigation.
It is remarkable to see that artwork through AR technologies is also
evolving as a tool of social and political criticism by offering artists a basic
expression for their positions and the involvement of visitors in the political and visual
work. It was also noticed that as technology evolves, art evolves as well.
Technology offers solutions and art exploits them. As a natural continuation of this
research a larger-scale research should follow that will deal with the collection
of visual artworks making use of AR technologies and promote as much as
possible the interaction between the visual work and the observer. Through this
new research, safer conclusions can be drawn about the use of these
technologies in interactive art environment.
10ο 10th Student Computer Science Conference of Central Macedonia -
17
/
4
- 20/4/
2018
. (
2018
).
Retrieved 20 October2019
from https://www.youtube.com/watch?v=fJbuw- jSDbTg&feature=share
Pagoulatou
,
Ch.
,
Lazou
, Α., &
Ntourma
,
Δ.
(
2017
).
Virtual Reality Systems (Τechnological Εducational Ιnstitute of Western Greece)
. Retrieved from http:// repository.library.teimes.gr/xmlui/bitstream/handle/123456789/6235/ %CE%
A3%CE%A5%CE%A3%CE%A4%CE%97%CE%9C%CE% 91%CE%A4%CE%91%20%CE%95%CE%99%CE%9A%CE%9F% CE%9D%CE%99%CE%9A%CE%97%CE%A3%20%CE%A0%CE%A1%CE%91%CE%93%CE%9C%CE%91%CE%A4
%CE%99%
C E%9A%CE%9F%CE%A4%CE%97%CE%A4%CE%91%CE%A3.. pdf?sequence=1
&isAllowed=y
Papadopoulou
,
Μ.
(
2018
).
Directing Another Reality (Augmented Reality) Unpublished master's thesis
, Ionian Universtiy, Corfu, Greece.
Charalabidis
,
Ά.
(
1995
).
The Art of 20th Century
(Vol.
3
). Thessaloniki: University Studio Press.
Ahmedien
,
D. A. M.
(
2017
).
Reactivating the neural dimension role in interactive arts
.
LEONARDO
,
50
(
2
),
182
-
183
.
Alsina
,
M. S.
, &
Kerlleñevich
,
H.
(
2013
).
AHORA. Retrieved 20 October2019
from http://cargocollective.com/operadora/ahora
Boulos
,
K.
,
Lu
,
Z.
, &
Guerrero
,
P.
(
2017
).
From urban planning and emergency training to Pokémon Go: Applications of virtual reality GIS (VRGIS) and augmented reality GIS (ARGIS) in personal, public and environmental health
.
International Journal of Health Geographics
,
16
(
7
),
11
.
Caarls
,
J.
,
Jonker
,
P.
,
Kolstee
,
Y.
,
Rotteveel
,
J.
, &
Eck
,
W. van.
(
2009
).
Research Article Augmented Reality for Art, Design and Cultural HeritageSystemDesign and Evaluatio
.
Hindawi Publishing Corporation EURASIP Journal on Image and Video Processing
,
16
.
Chang
,
K.-E.
,
Chang
, C.-T.,
Hou
, H.-T.,
Sung
, Y.-T.,
Chao
,
H.-L.
, &
Lee
,
C.-M.
(
2014
).
Development and behavioral pattern analysis of a mobile guide system with augmented reality for painting appreciation instruction in an art museum
.
Computers & Education
,
71
,
185
-
197
.
Dils
,
A.
, &
Cooper Albright
,
A.
(
2001
). Moving History/Dancing Cultures:
A DanceHistoryReader
. USA: Wesleyan University Press.
Edmonds
,
E. A.
(
2010
).
The Art of Interaction. Digital Creativity
,
21
(
4
),
257
-
264
.
Edmonds
,
E. A.
(
2014
).
Human Computer Interaction, Art andExperience. Στο L.Candy
&
S.Ferguson (Επιμ.), Interactive Experience in the Digital Age Evaluating New Art Practice
. Switzerland: Springer International Publishing.
Freeman
,
J. C.
(
2017
). Virtual U.S./Mexico Border:
Migration Stories, geo-located augmented reality public art
.
Retrieved 20 October2019
from https://johncraigfreeman.wordpress.com/
2017
/11/18/exploring-anaugmented-reality/
Ibáñez
,
M. B.
,
DiSerio
, Á.,
Villarán
,
D.
, &
Kloos
,
C. D.
(
2014
).
Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness
.
Computers & Education
,
71
,
1
-
13
.
Jaewoon
,
L.
,
Yeonjin
,
K.
,
Myeong-Hyeon
,
H.
,
Dongho
,
K.
, &
Byeong-Seok
,
S.
(
2015
).
Real-Time Projection-Based Augmented Reality System for Dynamic Objects in the Performing Arts
. Symmetry, (
7
),
182
-
192
.
Kipper
,
G.
, &
Rampolla
,
J.
(
2012
).
Augmented Reality: An Emerging Technologies Guide to AR
. Rockland, MA: Syngress.
Kwastek
,
K.
(
2015
).
Immersed in Refection? The Aesthetic Experience of Interactive Media Art
.
Studies in Intermediality
,
9
,
67
-
86
.
Levin
,
G.
, &
Lieberman
,
Z.
(
2006
).
Footfalls. Retrieved 20 October2019
from http://www.flong.com/projects/footfalls/
Lichty
,
P.
(
2012
).
Love Bombers. Retrieved 20 October2019
from http://www. voyd.com/ar.html
Lichty
,
P.
(
2014
).
The Aesthetics of Liminality: Augmentation as Artform
. Leonardo,
47
(
4
),
325
-
336
.
Lichty
,
P.
,&
Shafer
,
N.
(
2016
).
AR,Alaska andAugmenting the Circumpolar
.
Media-N New Journal of the Media Caucus
. Retrieved fromhttp://median. newmediacaucus.org/caa-conference-edition-2016
-
washington-dc/aralaska-and
-augmenting-the-circumpolar/List of augmented reality software [Encyclopedia]
. (
2013
).
Retrieved 20 October 2019 from Wikipedia website
: https://en.wikipedia.org/wiki/List_ of_augmented_reality_software
Masayuki
,
A.
(
2012
).
Uroboros Torch. Retrieved 20 October2019
from http:// akamatsu.org/aka/works/uroboros/
West
,
D. M.
(
2012
).
Digital Schools: How Technology Can Transform Education
. Washington, D.C: Brookings Institution Press.