This article overviews services and platforms applied for application of technology of augmented reality (AR) to improve human-machine and painting-human interaction during observing abstract paintings. Besides, scenarios of AR application for fluid abstract paintings are systemized and discussed. The service based on project Artivive implementing AR for paintings is analysed. The approaches and architecture patterns for service, based on different scenarios are discussed. The developed architecture designs can be used to make the galleries of abstract paintings more interactive and engage more visitors. The suggested architecture solution (the project AR for Smart Abstract Art, ARSmArt) for the selected scenario “AR by Individual Visitor” is presented in case study. The ARSmArt architecture is compared to the existing Artivive based service. Future steps of research and development are described
1.1
Nowadays information technologies (IT) are a very fast-growing and powerful tool
that affects almost all areas of human life. One of the most impressive are
experiments with augmented reality (AR). Augmented reality is the technology to create a
“next-generation, reality-based interface” [
AR is a technology that augments physical environments on a digital device screen by overlaying them with digital content. For the recent several years, AR has been gaining popularity and developing by leaps and bounds, and it is impossible not to notice the possibilities of augmented reality applications. In this paper, attention is concentrated on art. There a lot of possibilities to use AR and create new dimensions of imagination. With AR, museums/galleries can reach a completely new level of interaction and help you rediscover the world. AR can be used to foster greater possibility to discover an artwork’s meanings and resonant impact.
There are no difficulties to determine an artist’s idea looking to the pictures of defined visual objects of modern art, it’s harder with abstract ones. Therefore, museums have been among the first public institutions to find a practical application for AR to enhance understanding and improve the connection between artists and visitors. The smartphone of visitor is the easiest means of engagement.
The concept of interactive pictures is already applicable to museums. This technology enhances the perceiving of abstract paintings. Augmented reality technologies allow making the picture “get alive” on the screen of a visitor’s smartphone. Exiting inventions of this area push scientists to do their own research in this industry 1.2
Right now there are over 2.5billion smartphone users in the world [
When we think of the arts, we tend to associate it with visiting an art gallery or taking a trip to a theater. But when augmented reality and art collide, it opens up new artistic possibilities. This is not just to enhance a visitors experience but also to free the art from the confines of a gallery, subsequently reaching a whole new audience.
Augmented reality art, as a new media subset, distinguishes itself through its peculiar mechanics of exhibition and performative re-contextualization. It allows the artist to translocate the borders and constraints of the experience from physical to virtual, expressing the piece onto spaces independent of physical or locative constraint, yet still tethered to the real world.
Early works of AR Art can be found in [
There are several approaches to help visitors to get an additional sense of the
painting. Modern museums use not only AR technologies to reach this aim, they use audio
systems (e.g. headphones, speakers) to play sounds that influence the visitors’ minds
and help them to get extra feelings. There are exhibitions that solve the problem of
understanding the paintings by playing with the lights. In the same time, AR shows
the best results in helping artists to make visitors see what the artists see. Existing
solutions use AR with a wide variety, among them applications that seamlessly
recognize a selection of two-dimensional artworks throughout the museum and provides
additional curatorial and interpretive content (including video) about the artwork [
Successful and already popularized projects are “ARtGlass”, “GuidiGO”,
“Artivive” [
Many AR systems that are used by museums act in a similar way they have some set of paintings (digital model of paintings) and unique predefined visual objects linked with this digital model (DM). The main drawback of this solution is that a visitor can perceive only one subjective vision that belongs to the artist. This approach can be improved by allowing visitors to make their own visual objects using imagination. It is a next step ahead in the virtual reality world.
AR technology are developing actively and becomes more popular around the
world. The number of applications and AR tools is growing, it allows artists brake the
boundaries and widespread their ideas on completely new level. The area AR in the
art is popular among researches because existing famous solutions have a lot of
drawbacks and restrictions in usage. The scenarios of for abstract paintings and
visitorpainting interaction developed in [
In this paper a new approach for interactive painting is suggested. Visitors of the gallery will have an opportunity to see visual effects based on their own vision not only the artist’s vision. Various scenarios of the interaction between artists and visitors are presented in the 2nd section. Possible architecture solutions for the selected scenario are presented in the 3rd section. 4th section describes case study based on comparing of the known project Artivive and the suggested architecture ARSmArt. Section 5 concludes research results and describes the future direction of research and development. 2
To develop the architecture of service for augmented reality added abstract painting it is important to connect all parts of the system in the correct way. It can be done using scenarios of interaction all parts of the system between each other.
There are possible scenarios can be chosen as a base for architecture development:
“AR by artist”, “AR by visitor group”, “AR by individual visitor”, “AR by artist and
individual visitor” [
The scenario “AR by artist” It based on the artist's title of the painting. In this scenario artist plays the main role (Fig. 1): an artist gives a title to loaded picture; the artist loads digital model for every loaded picture; the artist pays for registering on AR platform;
AR platform transforms a digital model to AR picture and connects it to the title of the picture. Exhibition visitors should just set up the AR app to own a smartphone and see on the picture on the gallery.
The scenario “AR by visitor group” (ScARG) is based on the artist title of the painting. It consists of the following procedures (Fig. 2): A group of n visitors (GV, GV = {Vi}, i = 1,...,n) after learning/looking of the painting give a set of titles (GTP, GTP = {VTPi}, i = 1,...,n) for it according with their understanding, feeling, etc. GTP is processed using technique/tool of semantic similarity analysis (SST) and divided on uncrossed semantic subsets ΔGTPj. For each subset, ΔGTPj is determined as a generalized word as a selected title (STPj). According to STPj DM of the painting, which can be developed using DT according to P (similar Scenario 1) is transformed into PAR-STPj using AT. Visitors of the groups can see different PAR-STPj using ATV. The artist and visitors of the group can analyze and compare ATP, PAR-A and GTP, PAR-STPj.
The scenario “AR by individual visitor” (ScARI) is based on the artist title of the painting. It consists of the following procedures (Fig. 3): A visitor (V) after learning/looking at the painting gives a title for it (VTP) according to his/her understanding, feeling, etc. The visitor introduces VTP by one of the means (sound, word, etc.) into tool AT . DM is transformed into a painting of AR (PAR-IV) using AT according to VTP. The visitor can see painting and PAR-IV using tool ATV, and the artist can see one by use of AT.
Each scenario of augmented reality application for the abstract fluid paintings that are described in the 2nd section needs own architecture to fit all requirements that it provides. So it is necessary to develop an individual architecture that will handle all parts of the scenario (inputs, flows, data processing). All scenarios have their advantages and disadvantages, but each of them is somehow better than the previous one, so the architecture solutions called as ARSmArt will follow from simple to more complex. 3.1
The first scenario that is defined as “AR by Artist” is the simplest scenario that can be used for solving the problem with getting an extra sense in the paintings. However, the result that it is provided by this scenario is enough to make visitor perceive the painting the way the artist do it. The architecture solution for this scenario is presented on Fig. 4. The architecture consists of the following roles: Artist, Visitor. Artist does a big part of the whole data processing flow. He is responsible for creating the painting, searching and processing the visual effect for certain picture and uploading these parts to the web application. The visitor’s role is to use application on the smartphone to get the AR visual effects over the surface of the painting.
The main functional elements of the system design are the following: Web application; Video storage; Painting to ID converter service (P2ID); Software for a smartphone. The web application is designed to process all inputs (picture, visual effects for picture) from the artist; sending requests to service that converts painting to ID; saving visual effects to the storage. Video storage stores visual effects (videos) for the painting by ID that is received from P2ID, it is necessary to get the right visual effect in a gallery. Painting to ID converter service is the service that converts the array of bytes that represents a picture, to a unique identifier. Software inside a smartphone is designed to show AR visual effect over a painting surface. The flow that describes how parts of the system work together is described below: 1. Artist uploads picture and a visual to the web application; 2. Web application sends the picture to P2ID; 3. P2ID converts picture to a unique identifier (can be GUID) and sends it back to web app; 4. Web app save visual effect by ID received from P2ID; 5. Visitor use smartphone with installed application and take a picture of a painting; 6. Software inside smartphone sends the picture to P2ID and receive ID; 7. Software inside the smartphone gets a visual effect from video storage by ID and projects it over a painting surface using AR techniques.
The scenario “AR by Visitor Group” has more flexibility it can satisfy multiple visitor's thoughts. Due to the fact that this scenario allows seeing the author’s insight to more visitors, it is more recommended to use in galleries. The architecture solution for this scenario is presented on Fig. 5.
The architecture consists of the following roles: Artist; Title experts; Visitor. Artist is responsible for creating the painting; uploading a picture to the web application for AR technologies applying; uploading a picture to the web application for aggregating possible titles for the painting; searching and processing the visual effect for his painting. Title experts help the artist to understand how visitors perceive his art. Using titles for certain painting obtained from title experts, the artist can upload multiple visual effects and give an opportunity to the visitors to choose between predicted AR visual effects. The visitor’s role is the same as in Scenario A.
The main functional elements of the system design are at the following list: Web application for processing pictures and visual effects; Web application for aggregating titles for the paintings; Video storage; Painting to ID converter service (P2ID); Software for a smartphone.
Web application for processing pictures and visual effects is designed to process all inputs (picture, visual effects for picture) from the artist; sending requests to service that converts painting to ID; saving visual effects to the storage. Web application for aggregating titles for the paintings serves for collecting possible titles for the painting, process them and group them into the set of the most popular ones. Video storage, stores visual effects (videos) for the painting by ID and titles that were obtained from the web application for aggregating titles for the paintings. Now every visual effect has a composite unique identifier that consists of ID from P2ID and a title.
Painting to ID converter service has the same role as in Scenario A. Software inside a smartphone is designed to get input from visitors and show AR visual effects over a painting surface.
The flow that describes how all parts of the system work together is the following: 1. Artist uploads picture both to the web application for processing pictures and visual effects and to the web application for aggregating titles for the paintings; 2. Title experts give the tiles for a painting; 3. Web application for titles aggregating process them and group into the set of the most popular ones and send it to the artist; 4. Artist searches and processes video effects based on titles obtained from title experts; 5. Web application sends the picture to P2ID; 6. P2ID converts picture to a unique identifier (can be GUID) and sends it back to web app; 7. Web app save visual effect by ID received from P2ID and titles; 8. Visitor use smartphone with installed application and take a picture of a painting; 9. Visitor choose the title that he wants to see over the painting; 10. Software inside smartphone sends the picture to P2ID and receive ID; 11. Software inside the smartphone gets a visual effect from video storage by ID and title and projects it over a painting surface using AR techniques. 12. Visitor can choose another title and see the AR for the painting. The scenario “AR by Individual Visitor” is the next step in developing scenarios of augmented reality application for the abstract fluid paintings. This algorithm allows each visitor to choose an individual visual effect. The data processing flow for an artist is also improved. The architecture for this scenario is presented on Fig. 6. The architecture also consists of three following roles: Artist; Title experts; Visitor. Artist is responsible for creating the painting; uploading a picture to the web application for AR technologies applying; uploading a picture to the web application for aggregating possible titles for the painting; filling in video storage with different visual effects according to possible titles for all his paintings. Title experts and visitors have the same roles as in Scenario B but visitors also should input the title that describes their perceiving of the painting.
The main functional elements of the architecture are: Web application for processing pictures and visual effects; Web application for aggregating titles for the paintings; Video storage; Video adaptation service; Software for a smartphone. Web application for visual effects processing is developed for uploading visual effects and saving them to the storage. Web application for aggregating titles has the same responsibilities as in Scenario B. However it is more valuable in this case. The reason is the absence of a link between a picture and a video. This link will be set on in realtime by the visitor’s input. Video storage is designed to store any possible visual effects that can be applied for paintings after some adaptation process. Video adaptation service use obtained picture and video to process the video to fit pictures sense and dimension. Software inside a smartphone serves as in Scenario B.
The algorithm for defined architecture can be described by the following steps: 1. Artist uploads picture to the web application for aggregating titles for the paintings; 2. Title experts give the tiles for a painting; 3. Web application for titles aggregating process them and group into the set of the most popular ones and send it to the artist; 4. Artist searches and processes video effects based on titles obtained from title experts; 5. Artist fills in video storage with different universal visual effects; 6. Visitor use smartphone with installed application and take a picture of a painting; 7. Visitor inputs the title that he wants to see over the painting; 8. Software inside smartphone gets video from video storage by title; 9. Software inside sends a picture and a video to video adaptation service to get the adapted visual effect; 10. Software inside the smartphone obtained visual effect projects over a painting surface using AR techniques; 11. Visitor can choose another title and see the AR for the painting. Both scenarios and architectures that describe them have advantages and disadvantages. They differ by the complexity and functional elements they consist of, but every architecture can help artists to make visitors to see his insight. The architecture developed for Scenario A is the simplest among others. The main advantages are: high-quality AR because of thorough visual effect processing by the artist; artist can popularize his insight; low complexity.
Disadvantages of this architecture are the following: AR is based on only one point of view; searching and preprocessing visual effects for a painting by the artist is necessary (time-consuming process); reusing existing visual effects is unable.
Architecture for Scenario B has some improvements compared to Scenario A. A few cons were fixed: multiple points of view is used for AR creation; high-quality AR because of thorough visual effect processing by the artist; visitor can choose AR visual effect; artist can use titles from title-experts for visual effects searching. However, there are several gained cons: higher complexity of the system; preprocessing of more amount of visual effects is needed; reusing of existing visual effects id still unable.
The most complex architectural pattern has the highest usability, but it requires more resources. Pros (P1-P4) are the following: (P1) Artist can use titles from titleexperts for visual effects searching; (P2) Preprocessing visual effects is not necessary; (P3) Visitor can apply any AR visual effect; (P4) Ability to reuse stored visual effects in video storage. Cons (C1-C5) are the following: (C1) Less quality of the AR because of fact that visual effects are processed no for certain pictures; (C2) Difficult process of automatic visual effect adaptation; (C3) Probability of necessary visual effect absence; (C4) Visual effect adaptation process can take much time; (C5) High complexity. 4
The architectural approaches allow extending functionality of existing solutions. The service “Artivive” can be improved in several ways according with suggested ARSmArt architecture. Advantages and disadvantages of both “Artivive” and the developed architecture ARSmArt are presented on Table 1.
The prototype described in the table above can be more useful and applicable in modern galleries. Visitors can be more interested in interactive paintings.
We described the scenarios and possible architecture approaches for application of technology of augmented reality (AR) to improve human-machine and paintinghuman interaction during observing abstract paintings.
In this work the approaches and architecture patterns for service based on different scenarios are suggested and described. Case study for developed scenario “AR by Individual Visitor” is compared to one of the existing solutions. Using system based on the developed architecture, museums can reach a completely new level of interaction and help exhibition visitors rediscover the world.
Future steps and improvements are the following: - searching visual effects in real time straight on the Internet; - sell paintings with added AR effect; - using the presented approach in different fields (e. g. psychology); - creating virtual gallery concept.