TravelGANs. A design approach to style transfer Azzurra Pini 1 Abstract.1 Generative AI has been used and researched mostly for implementation, the user interface and finally we will propose some artistic purposes and with a technology driven approach. We present directions for future research. an application of style transfer with GANs to design a service for the travel industry. Our solution is intended to support a mindset shift in the travel booking process, which is claiming for a more visual based 2 A-EYE SEASON TRAVEL search. TravelGANs is designed for sourcing local photos from the web and seasonalize them on the fly based on users’ preferred travel The project started observing a mindset change in travel booking dates. towards a more visual experience, while the formats around communicating destinations have stayed static. Some recent initiatives by a few travel companies are have already acknowledged 1 INTRODUCTION this cultural and behavioural shift, and started to move in that Generative AI has been around for quite a while to support human direction, providing for example users with the possibility to book in specific tasks, mainly for data augmentation, to fill gaps of data travels through Instagram. [11][4][9] With this idea in mind, the concept originated in particular from and information in various contexts. Although the military has been the idea of gathering travel inspiration from online pictures, and a primary field of application for most AI techniques, the research specifically to answer the question on ‘how does (place) look like in has then evolved mostly in academia for computer vision research (season)?’. Even the most visual platforms, more or less specifically and applied to artistic projects. dedicated to travel, do not in fact provide alternative photos of a Our project has been focusing in particular on the application of place in different conditions – a functionality that we claim could be generative models, such as Generative Adversarial Networks, a class extremely relevant for travel planning, particularly for certain of machine learning systems invented in 2014 [6], that is now one of destinations with high seasonal variability. the most popular techniques used to learn patterns from data and From a technology perspective several options have been reproduce its characteristics into new visual samples. Since then the explored in order to tackle this challenge, ranging from computer technique has been used mostly to generate realistic photographs, vision techniques to more complex machine learning models. While text to image, video simulation, face ageing, image blending, super classic computer vision techniques would support the analysis and resolution and similar applications. At the technical level the editing of existing pictures, they are not accurate enough for capabilities of GANs, VAE and generative models in general have transforming images without paired examples provided. improved substantially over the years and we are now witnessing an An interesting opportunity for experimentation has been impressive level of accuracy in reproducing realistic content. identified in Generative Adversarial Networks (GANs), in particular Nevertheless, only a few examples of real-world applications can be we have been exploring the applications in relation to style transfer, found, mostly for the fashion industry [3][13], for generating an increasingly popular sub-set of algorithms originally started for realistic models and apply different styles to clothing items. Some artistic purposes [5] and now used to translate different types more rare ones are registered in the science domain [10] and pictures into different styles, e.g. converting a photograph into the engineering such as for autonomous driving, to simulate the possible style of a famous painter or a horse into a zebra. More specifically, different environmental context of roads in terms of lighting and recent research has provided examples of application of GANs style weather. [1] transfer for translating photos of locations from winter to summer As opposed to the majority of the current research, instead of and vice versa. [14] Within this framework more recent research has taking a technology driven approach, we tackled generative AI from also been made on transferring an image from night to day [1] and a design perspective, looking for different ways where generative into different lighting or weather conditions. [1] [8] models can support a user experience or a mindset change. The CycleGAN style transfer [14] is an approach for unpaired TravelGANs tackles a behavioural change in the travel booking Image-to-Image translation developed at UC Berkley to learning to process which we claim to be more and more relying on visual translate an image from a source domain to a target domain in the features. The prototype aims at providing an AI powered visual absence of paired examples. Among the different examples of booking service, capable of quickly sourcing local photos and seasonalise them on the fly in response to visitors’ preferred travel period. In this paper we will introduce the design concept, some experiments with the CycleGAN [14] style transfer and its 1 Fjord, Accenture The Dock, Dublin, Ireland, email: azzurra.pini@fjordnet.com Copyright ©2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) transfer presented in the paper – the most popular being the artistic applications – we explored a later implementation into a season transfer for transforming pictures from winter to summer and vice versa. The current model is trained on 854 winter photos and 1273 summer photos of Yosemite downloaded from Flickr. 3 TravelGANs With our prototype the challenge was to design a service to support a visual exploration of holiday destinations and for this purpose to exploit the potential of generative network. The service has been conceived mainly to support the travel exploration phase and it’s meant to complement a regular booking platform (such as Booking.com). TravelGANs is meant to provide inspiration for holiday booking, in situations where there are less constraints in terms of destination and period with respect to other travel use cases. Specifically designed for ad-hoc holiday bookers, the service is meant to support an experience where users can find holiday destinations based on visual preferences rather than time and budget. The core concept is for them to understand how a certain destination will look like at the moment of the trip. The service is also tackling the use case of a B&B or small hotel owner as well as travel agents assembling packages. In this second case TravelGANs can help them rapidly assemble, customize and seasonalize their listings to Figure 1. The implementation pipeline. increase the user engagement. In order to test the concept we developed a web app prototype, the transfer. The details of the season transfer tests and using the current season transfer model of CycleGAN [14]. The implementation will be discussed in the following section. design and development were performed in a rapid prototyping environment by a small team and delivered in a 6 weeks period. 3.2 A CycleGAN application 3.1 Building a travel DB The transformation of our pictures was made using the pretrained summer2winter_yosemite model2. From a software perspective we were aiming at the following high- Because the model has been trained mostly on pictures of level pipeline: Yosemite, when used to transfer seasonal style for other types of 1. Selecting a set of popular travel destinations organized by type: locations it visibly revealed a fundamental bias towards mountain seaside, mountain, cities, nature; landscapes of a “continental” latitude, with particular colour and 2. Building a static dataset for those destinations in different lighting conditions. One of the challenges of our research was also seasons, 3. Selecting winter and summer alternatives; to test the performance of the model on different types of landscapes. Aware of this bias we also made some initial tests with a dataset 3. Run the images through the pre-trained model for season containing a weather classification3 to complement and improve the transfer; performance of the model, forcing associations between 4. Feed back the transformed images to the data base. meteorological conditions and seasons. With the support of this Our initial database has been populated by a web app we developed classification we selected pictures tagged with peculiar seasonal to access the FlickrAPIs, which allowed us to automate the weather conditions before feeding them to the model for the transfer. collection of images, and in particular: Some tests returned particularly positive results [Figure 2], regardless of the similarity of the images with the training set, • Select a location suggesting a valuable direction for future improvement of the season • Select a season transfer. In order to gauge the applicability of the CycleGAN for the • Toggle for hemisphere (to support the seasons inversion) design, we tested the model also for other types of landscapes. • Filter licences for usage rights Because of the limitations of the original training dataset we proved • Pick preferred pictures and download a .zip file that the application of the transfer generally works better for pictures • Feed the downloaded pictures to the CycleGAN season of mountains and similar natural landscapes. In particular we transfer model. confirmed the hypothesis that the winter2summer transfer works The web app is initially intended to support the development of the pretty well with strong seasonal traits, e.g. snow, clouds, cold prototype, aiming at a future integration with a travel portal, but also colours, thus supporting the utility of a complementary weather to support the use case of the host, who can explore and select classification. At the same time, we discovered that generally the royalty free photos to add to his/her property page. transfer doesn’t work well when the original picture is very different As displayed in Figure 1, based on the results of our test we also from the examples of the training set, in the cases where the image included some pre-processing steps that we found useful to improve doesn’t present strong classic seasonal traits, i.e. if it’s rather 2 https://github.com/junyanz/CycleGAN. 3 https://www.cs.ccu.edu.tw/~wtchu/projects/Weather/index.html colourful and sunny. In this case sometimes the perceived effect is a to translate the images in a more straightforward way. Without the reversed transfer, converting the image to winter (instead of confusion created by unexpected colours, the model can simply summer). (Figure 3) apply the ‘seasonal’ style. Some tests of the winter2summer model on our dataset produced pretty good results. For this reason, we real (winter) fake (summer) implemented the black and white conversion as a step in our pipeline before feeding the images to the model as displayed in Figure 1. As shown in Figure 4, after the conversion the hypothesised colourisation recognises and assigns pretty much all the right colours. Sometimes the fake images can be difficult to distinguish from the original colour version, but in terms of season transfer the results appear definitely more accurate and realistic. real (winter) fake (summer) before Figure 2. Examples of season transfer in combination with weather classification real (winter) fake (summer) after Figure 4. winter2summer with black&white conversion 4 The User interface TravelGANs’ user interface is designed to support two different use a. cases. The main one we designed for is for a regular user who’s exploring options for choosing a holiday destination. The user is presented with a set of photos of travel destinations, randomly selected from our database through the Flickr API web app. The photos are arranged in an image board layout without any description indicating the location or season, to be used for visual b. inspiration. The user can refresh the selection and is prompted to select one that resonates with her/him. After selecting a photo, the destination and the season of the image is revealed to the user. We use the metadata automatically associated to each photo - geographical coordinates, type of location and timestamp - to provide the user for other suggestions of similar destinations to explore. c. Figure 3. winter2summer failure cases A similar phenomenon unexpectedly happens also for environments that should have been similar to the training dataset, suggesting the need of a wider training also for these locations. (Figure 3b) Since the prototype was designed to include different types of locations, further experiments have been made also with radically different landscapes. When an unexpected natural element is found (e.g. sand) the model fails to recognise it, and returns more “creative” results, mistaking for example a desert for sea and therefore replacing wrong colours (Figure 3c). As we expected the model doesn’t also work very well with pictures of interiors, such as hotel rooms, that could be rather common for travel photos, especially as the pictures are gathered in an automatic way (e.g. through APIs). Some of these experiments also revealed that the performance of Figure 5. Photo board, destination selection the model increases significantly when the original picture is in black and white. In this case we hypothesize that the model is able When a GAN generated image is selected, the original photo is • Dominant color extraction and similar techniques would displayed for comparison and the user is then guided towards the allow to explore travel locations by their most frequent traditional booking process. hues, responding to users’ questions like “I want to go to a place that looks mostly yellow and blue”. • All these techniques can be integrated and provide live updated personalized visual cues and trends of the most searched terms or characteristics. The project also became a starting point to deepen the reflection on human-machine interaction, in particular in relation to service design applications of generative AI and the need for parameters for the exploration and representation of the latent space. REFERENCES [1] Anoosheh, A., Sattler, T., Timofte, R., Pollefeys, M., & Gool, L. Van. (2019). Night-to-day image translation for retrieval-based localization. Proceedings - IEEE International Conference on Robotics and Figure 6. ‘Seasonalization’ of selected destination Automation, 2019-May, 5958–5964. [2] Caesar, H., Uijlings, J., & Ferrari, V. (2018). COCO-Stuff: Thing and Stuff Classes in Context. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1209–1218. 5 FUTURE WORK [3] Dietmar, J. "GANs and Deepfakes Could Revolutionize The Fashion TravelGANs is a prototype of a service that suggests a new approach Industry". Forbes. at holiday exploration, based on users’ visual preferences. The [4] Coffey, E., Easyjet launches app allowing passengers to book flights implementation exploits the potential of deep generative networks based on Instagram pictures, (2018), Indipendent.co.uk to provide seasonal alternatives of travel pictures. https://www.independent.co.uk/travel/news-and-advice/easyjet-app- From a design perspective there is planned future work for an look-and-book-instagram-pictures-flights-social-media- extensive user research, especially on the perception of AI generated a8588481.html images in relation to a travel booking service. Broader research is [5] Gatys, L., Ecker, A., & Bethge, M. (2016). A Neural Algorithm of Artistic Style. Journal of Vision, 16(12), 326. also required to design functionalities more focused on visual [6] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David similarity and personalization of features. Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio From a technology perspective we discovered a great potential "Generative Adversarial Networks", in NIPS 2014 for further exploration with respect to GANs and more in general on [7] Liu, M. Y., Breuel, T., & Kautz, J. (2017). Unsupervised image-to- deep learning models for generative design. image translation networks. In Advances in neural information More specifically, we identified several opportunities for future processing systems (pp. 700-708). work in: [8] Lu, C., Lin, D., Jia, J., & Tang, C. K. (2017). Two-Class Weather • Training of different models for different location types, Classification. IEEE Transactions on Pattern Analysis and Machine e.g. Seaside, cities and interiors. Intelligence, 39(12), 2510–2524. • Training models on clustered locations based on similar [9] Miller, C., How Instagram is changing travel visual characteristics, using external classifications such as https://www.nationalgeographic.com/travel/travel-interests/arts-and- culture/how-instagram-is-changing-travel/ the weather dataset [8]; [10] Mustafa, M., Bard, D., Bhimji, W. et al. CosmoGAN: creating high- • Implement computer vision techniques to analyse images fidelity weak lensing convergence maps using Generative Adversarial upfront, such as image segmentation and classification Networks. Comput. Astrophys. 6, 1 (2019). integrating stuff classes [2]; [11] Ritschel, C., When booking holidays think of Instagram photos above all other factors, Indipendent.co.uk, (2017) • Contextual latitude/longitude time classification to https://www.independent.co.uk/life-style/millennials-holiday- improve seasonal accuracy; decision-instagram-photos-factors-think-first-social-media- • Semantic analysis and content recognition: recognising a8131731.html semantic elements in pictures would allow to apply [12] Chu, W. T., Zheng, X. Y., & Ding, D. S. (2017). Camera as weather different style rules to different objects (e.g. sea VS sand) sensor: Estimating weather information from single images. Journal of Visual Communication and Image Representation, 46, 233–249. • Perform tests with similar generative techniques as [13] Wong, C. "The Rise of AI Supermodels". (2019) CDO Trends. Variational Auto-encoders (VAE). [14] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired Image- to-Image Translation Using Cycle-Consistent Adversarial Networks. The exploration of these technologies would allow also to expand Proceedings of the IEEE International Conference on Computer the design of a more visual travel booking experience with Vision, 2017-Octob, 2242–2251. complementary use cases to be explored. • The simple application of content recognition would allow users for a semantic search of specific visual elements into travel photos. Such a travel service would therefore be able to respond to questions like “I want to go to a place with trees and museums”.