G.E.M.I.X.: Game Engine Movie Interaction eXperience Tommaso Antonio Laura Anna Davide Dario Celata Notarangelo Ripamonti Gadia Maggiorini University of University of University of University of University of Milan Milan Milan Milan Milan Milan, Italy Milan, Italy Milan, Italy Milan, Italy Milan, Italy ripamonti@di. gadia@di. dario@di. unimi.it unimi.it unimi.it ABSTRACT produced renderings and animations, but there is also a con- The use of Computer Graphics in the production of movies stant request for novel tools aimed at supporting an efficient and Visual Effects is a well-known and established practice and optimized production of the final contents. in the today industry. Due to the articulated structure of a In particular, the role of previsualization (also known as pre- movie production pipeline, there is a continous need for ad- vis or previz) is becoming increasingly crucial in the pre- vanced tools aimed at speeding-up and optimizing the work production stage. A previsualization in today industry is basi- of each pipeline department. In particular, previsualization cally a “rough” version of the final, more complex, scene [4]. represents a crucial step of the pipeline, because it helps to This preliminary version of the scene is usually used by the visually evaluate the potentiality of a scene before the final production team to analyze and tune its features. For example, production by using an approximated version of its features a previs can be used to test the efficacy of a particular light- (models, animations, etc). ing setup, the placement of actors or objects in the scene, a In this paper, we describe the design and implementation camera setting and movement, etc. In any case, the final goal choices at the basis of G.E.M.I.X. (Game Engine Movie In- of previsualizations is to avoid as much as possible changes teraction eXperience), a tool for the interactive production of to the scenes content in production or post-production stages, previsualizations, implemented as an extension of the Unreal thus speeding-up the overall work, and lowering the budget. 4 game engine. In particular, we have focused our efforts With the increasing complexity of today movies and special in the analysis and implementation of the visual interface of effects, there is a growing need for flexible tools for previsu- G.E.M.I.X.. We have adopted a Human-Centered Design ap- alization production, allowing the full interactive control of proach, collaborating with professionals in the visual effects every feature, with the possibility to automatically adjust the production, and analyzing their specific needs and require- overall preview to different levels of approximations. The ments, in order to allow a rapid adoption of the G.E.M.I.X. game engines [5] currently used in the video game industry as the main previsualization tool in the production pipeline. seem to satisfy these technical requirements and to provide most of the desired features for a previsualization tool. ACM Classification Keywords H.5.2. Information Interfaces and Presentation (e.g. HCI): However, a relevant problem is that these environments are User Interfaces not specifically designed for previs production; as a conse- quence, members of the production team have to face a tough Author Keywords learning curve in order to become proficient in the use of Previsualization; Human-Centered Design; Game Engine; these tools. Therefore, this can reduce the control over the Movie Production Pipeline creative process, and, as a consequence, also the final effec- tiveness of the produced previsualization. INTRODUCTION In this paper, we present G.E.M.I.X. (Game Engine Movie The current state of movie and visual effect production in- Interaction eXperience), a tool for the interactive production dustry sees a massive use of Computer Graphics-based tech- of previsualizations, implemented as an extension of the Un- niques. As a consequence, not only the research on Computer real 4 game engine. G.E.M.I.X. has been designed adopting Graphics is particularly active in investigating and proposing a Human-Centered Design approach, collaborating with pro- novel algorithms and techniques to enhance the quality of the fessionals in the visual effects production, and analyzing their specific needs and requirements, in order to propose a previ- sualization tool characterized not only from a complete set of customization features, but also from an usability based on the users skills, needs and preferences. The remainder of this paper is organized as follows: in the fol- GHItaly18: 2nd Workshop on Games-Human Interaction, May 29, 2018, Castiglione lowing sections we will present a brief overview of the state della Pescaia, Italy. of the art related to the production of previsualizations, and Copyright © 2018 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. then we will present the overall design process of G.E.M.I.X.. G.E.M.I.X. DESIGN AND IMPLEMENTATION Finally, we will draw conclusions and discuss major future While most of the works considering high-level game engines developments. for the production of previsualization address mainly techni- cal aspects, less efforts have been given to address the usabil- ity of these systems. In this paper, we describe the design and implementation choices at the basis of G.E.M.I.X. (Game Engine Movie In- RELATED WORK teraction eXperience), a tool for the interactive production If compared to the situation in the production and post- of previsualizations. G.E.M.I.X. has been designed not only production stages, where several well-established tools, tech- considering how to adapt or to integrate the production tools niques and technologies have been proposed in the last years, available in the game engines from a low-level development it is evident how there is a relevant lack of standardization in point of view, but mainly focusing on a Human-Centered De- the previsualization production field. Indeed, often previsual- sign (HCD) approach [9, 18, 7] in the overall design of the izations are managed internally at the pre-production depart- whole production environment. To this aim, we have collabo- ment, with ad-hoc approaches and tools. rated with EDI - Effetti Digitali Italiani 1 , an Italian company In analogical productions, previs were usually based on with almost 20 years of experience in the production of visual sketches and illustrations, produced and used in relation to the effects for cinematography and commercials. EDI staff has overall storyboarding process. In more recent digital produc- expressed in the last years interest in the integration of tech- tions, different approaches are adopted. In some cases, previs nologies and techniques from different fields (video game, are produced using standard 3D modeling and animation soft- Virtual Reality) in the production pipeline. In particular, EDI wares and a set of preliminary predefined assets. However, has decided to radically change the current pre-production this approach requires the presence of a skilled professional (a pipeline, by adopting a game-engine based previsualization modeler or an animator), who usually is not involved fulltime as close as possible to the actual needs of the professionals at this level of the production pipeline. Some specific com- involved in this process, and with a particular focus on a high mercial softwares for previsualization have been proposed: in level of usability. To this aim, in collaboration with EDI staff, some cases, they allow to produce a more advanced version we have structured the design process of G.E.M.I.X. follow- of 2D storyboards, based on animated sequences of digital ing the HCD principles [7, 3, 2] as resumed in Figure 1. drawings with audio and sound. Other tools allow the pro- duction of rough 3D scenes by selecting assets from a set of predefined models and textures, and setting up animations at different levels of sophistication. However, these tools often limit the possibilities in the customization of the previsualiza- tion features, and are characterized by interfaces comparable to other typical 3D animation softwares, thus not easily acces- sible to not-experts in 3D content production. Some academic researches have been proposed to investi- Figure 1. G.E.M.I.X. design process gate different solutions and aspects related to the production of previsualizations. In some cases [13, 6], the goal was G.E.M.I.X. requirements to propose some level of standardization in the process, to achieve an efficient interoperability between different produc- The first stage of the design of G.E.M.I.X. has been the tion tools and among pre-production and production depart- analysis of EDI requirements, and the understandings of the ments. Other works [12, 14] have investigated instead the needs of the professionals involved in the pre-production potentiality of currently available high-level video game en- stage. This process was based at first with meetings with gines as tools for the efficient production of previsualizations. EDI managers and staff, to discuss about the goals and re- The influence of cinematography on video game production is quirements of the overall project. At the end of this briefings not a novelty, from both thematic and technical point of view. session, a detailed Project Charter document was produced. A well-known example is the use of cutscenes, used to break Then, several stakeholders have been asked to fill out a sur- gameplay in order to show not-interactive moments like e.g., vey regarding the G.E.M.I.X. project. Using the survey, the conversations, effects of a player’s action, etc. These short goal was to collect separately the personal vision of differ- animations usually follows conventions taken from movie ent professionals involved with different roles in the project. production, and were generated in the past using the same The survey has been structured with a common section to be tools and techniques. In the last years, due to the continuous proposed to all the stakeholders, related to general comments growth of the computational power of graphics cards, real- and opinions on the overall G.E.M.I.X. project, followed by a time rendering has seen a relevant enhancement, such to al- section specifically prepared for different kinds of stakehold- low the use of the same rendering engine for both gameplay ers. Professionals directly involved in the pre-visualization and cutscenes. As a consequence, current game engines like process were asked to discuss about the current setup in pre- Unity and Unreal Engine provide specific tools to create and vis production, and about the decisions taken to choose the manage both interactive than offline animations. 1 https://www.effettidigitali.it/en/ technology and tools to adopt, while professionals involved for the availability of an advanced multi-track editor (the Se- in production stage were asked to discuss about the interop- quencer Editor) providing specific tools for the production eration between pre-production and production stages, and and preview of cinematic sequences in real-time, which is to express their opinion on the current previsualization setup one of the requirements indicated by EDI. Due to the strict used in EDI. Finally, managers were asked to fill out a dedi- constraints imposed by Unreal Engine related to the exten- cated section related to the strategies and goals for the future sion of its features, it was not possible to develop G.E.M.I.X. of the company. as a stand-alone application; rather, it has been implemented as an application to be run inside Unreal Engine. Regard- Once collected all the surveys, a detailed list of the require- ing the production features required by EDI, the adopted ap- ments and goals of EDI about the G.E.M.I.X. project has been proach was to develop only the tools not already provided by defined. We resume the main points: Unreal Engine, and to collect together those already present, • Adoption of a game engine as the main environment for but providing an alternative and intuitive visual interface de- interactive previs production signed specifically for the identified personas. At the end of the prototyping process, G.E.M.I.X. was composed by four • Achievement of a high visual realism still maintaining real- main modules, each dedicated to specific functions, which time interaction with the previs features will be described in the following subsections. • High interoperation between differen production depart- ments, through the possibility to import, visualize, and Model Viewer modify 3D assets of various standard formats, already The Model Viewer module is used to visualize one of the 3D available and used in the production stage models placed in the previs scene, and to modify its charac- teristics and material. Moreover, it allows to import a model • Possibility to create, personalize and animate all the fea- created with an external 3D modelers, or from a database of tures of 3D human models available assets. • Possibility to automatically create large outdoor environ- Character Creator ments, with full control over the morphology of the terrain The Character Creator allows to create and customize a 3D and its features, and with illumination based on daytime model of a human character, to be animated and placed inside • Full control of the virtual camera settings and movements the previs scene. The editor (Figure 2(a)) allows to modify in an intuitive way several characteristics of the model, like e.g., • Possibility to export the previs as an animation height, hair color, or the amount of body muscles (Figures 2(b) and 2(c)). Moreover, it is possible to load previously • The adoption of a high-usable and high-intuitive visual in- modified models, and to apply further customizations. terface, in order to make the tool and its features easily un- derstandable also to the stakeholders directly involved in previs production, usually not suited to 3D assets produc- tion softwares G.E.M.I.X. context of use On the basis of the analysis of the company requirements, and of the professionals involved in the design and production of previsualizations, we have decided to better define the final stakeholders of G.E.M.I.X. before the beginning of the actual design of the application. We have decided to adopt the per- sonas approach [11], in order to identify a small set of poten- tial target users of a previsualization tool.: the film director, (a) the director of Photography, the previs artist, and the scenog- rapher. In Table 1 we present a very synthetized resume of the main details of each persona. G.E.M.I.X. design After the conclusions of the activities described in the previ- ous sections, we had all the necessary informations in order to begin with the design and development of a first prototype of G.E.M.I.X.. First of all, we have evaluated the currently available game (b) (c) engines, in order to select the most appropriate for the inclu- Figure 2. The Character Creator (Figure 2(a)) allows to customize a 3D sion of the needed functionalities. We have decided to use Un- model of a human character. Figures 2(b) and 2(c) show the results of real Engine as the basis for the implementation of G.E.M.I.X., different settings of the Muscles parameter because of the very realistic level of rendering quality, and Persona Role Behaviour To include in G.E.M.I.X. To avoid in G.E.M.I.X. Film • To decide the content • Impatient • Choice among differ- • To lose time with irrel- Director of each scene and of • She wants her work to ent scenes/locations evant details the whole movie be recognized • Predefined models of • Slowness in the pro- • She trusts her staff humans duction of the final re- • Production of effec- sult tive previsualizations Director of • To set up lights • Perfectionist • Presets of customiz- • Not-customizable Photography • To collaborate with • She surrounds herself able lights tools the scenographer to with people who sup- • Full control of camera • Simulated lighting not design the scene port her artistic talent movements and shoot- close to reality • To set up the framing • She loves experiment- ing parameters ing with framing Previs artist • To set up the scene for • Very aware of produc- • Easily accessible pro- • Limited flexibility of previsualization tion times duction tools the software GUI • To visually communi- • She exploits her artis- • Possibility to cus- • Differences from stan- cate the film director’s tic background to un- tomize the production dard movie produc- ideas destand the film direc- process tion pipeline tor’s needs • She is skilled with 3D production tools Scenographer • To design a scene and • She follows her own • A database of differ- • Limited customiza- its features on the ba- personal aesthetic ent models tion of materials sis of the film direc- taste • Freedom to place ob- • Not-realistic preview tor’s needs • She has different artis- jects of objects • To coordinate the tic skills • Customization of ob- other staff members ject materials responsible for the scene preparation Table 1. Main characteristics of the identified personas Level Editor Cinematic Level The Level Editor is the more sophisticated module imple- Once the previs scene and models have been created and mented for G.E.M.I.X.. It allows the automatic generation tuned, the Cinematic Level gives to the user specific tools of an outdoor terrain to be used as the main environment for to set up the camera settings and movements, using param- the previsualization. This module applies techniques from eters and techniques directly inspired by cinematography. the Procedural Content Generation (PCG) [1] field in order The user can then “register” the previsualization, apply post- to create a complex scenario controlled by a limited set of pa- processing, and export it as a movie. The main idea at the rameters, like e.g., dimension, roughness, etc. The adoption basis of the Cinematic Level is to give to the user a feedback of a PCG-based approach allows several future extensions of as close as possible to the real cinematography experience, the Level Editor: in fact, PCG is applied in different fields from both the technical than visual point of view. of Computer Graphics, like e.g., the automatic generation of buildings and cities [16, 17], or of game levels for platform G.E.M.I.X. evaluation games [15, 8]. Once generated, the morphology of the pro- We have then focused our attention on the evaluation and fi- cedural terrain can be further modified by the generation of nalization of the visual interface, whose usability and easiness roads (Figures 3(a)), or the application of a Sculpting tool to use were two of the most important requirements indicated (Figure 3(b)). Moreover, models and characters generated by EDI. with the Model Viewer and Character Creator modules can be loaded inside the environment. We have implemented a first wireframe version of the visual interface, and we have conducted a Nielsen’s heuristic evalu- (a) (b) Figure 3. The Level Editor allows the procedural generation of a terrain as the main scene for the previsualization. It is possible to modify the scene by generating roads (Figure 3(a)), or sculpting the morphology of the terrain (Figure 3(b)). ation [10] with usability experts. A total of 36 violations of CONCLUSIONS AND FUTURE WORK the heuristics were detected, in particular in the Level Editor In this paper, we have presented the design and implementa- and Cinematic Level modules. Figure 4 shows a graph of the tion details of a tool for the production of previsualizations violations of the Nielsen’s heuristics. based on Human-Centered Design approach. G.E.M.I.X. has been developed as an extension of a currently available game engine, in collaboration with an Italian company (EDI - Ef- fetti Digitali Italiani) with almost 20 years of experience in the production of visual effects. The goal of the project was to achieve an innovative tool characterized not only from a high flexibility in the interactive generation of the previs scene with a high level of visual realism, but also on the accurate design of a usable and intuitive visual interface, in order to make it easily understandable to previs professionals with lim- ited skills in 3D assets production. The final version of G.E.M.I.X. is currently used in EDI, after Figure 4. Violations of the Nielsen’s heuristics of the first prototype of an iterative process of validation of its visual interface. In the G.E.M.I.X. visual interface. future, we plan a second evaluation stage, based on the opin- ion and comments from EDI professionals after the adoption The evaluation has led to a clear identification of strengths of G.E.M.I.X. as main tool for previs production. and weaknesses of the visual interface of G.E.M.I.X.: Future extensions of G.E.M.I.X. functionalities are possible, • Strenghts: like e.g., the integration of procedural tool to generate other – Good level of attractiveness environments (cities, indoors environments), and the support for Virtual Reality visualization devices, due to the growing – Coherent distribution of functionalities interest in the production of immersive movies. – Easiness of navigation ACKNOWLEDGMENTS • Weaknesses: The authors would like to thank Francesco Grisi della Piè and – No coherence in the layout of some sections all the staff at EDI - Effetti Digitali Italiani for their collabo- – Difficult reachability of some sections ration and support. – Sporadic problems with terminology – Lack of accelerators to speed up the interaction We have proceeded to revise the visual interface, with partic- ular attention to correct the issues in the two more problem- atic modules. We have performed a second heuristic evalua- tion, which had a positive result. The last step was to finalize the visual interface with a visual identity based on the EDI logo. Figures 2, 3 and 5 show the final visual interface of G.E.M.I.X. EDI is currently using G.E.M.I.X. in some selected projects, and a further evaluation stage will be performed as soon as relevant data could be collected. Figure 5. G.E.M.I.X. main menu REFERENCES Conference on Human Factors in Computing Systems 1. David S. Ebert, F. Kenton Musgrave, Darwyn Peachey, (CHI ’94). ACM, New York, NY, USA, 152–158. Ken Perlin, and Steve Worley. 2003. Texturing & 11. Lene Nielsen. 2013. Personas - User Focused Design. Modeling: a procedural approach. Morgan Kaufmann. Springer. 2. International Organization for Standardization. 2002. ISO/TR 16982:2002 -Ergonomics of human-system 12. Michael Nitsche. 2008. Experiments in the Use of Game interaction -Usability methods supporting Technology for Pre-visualization. In Proceedings of the human-centred design. (2002). 2008 Conference on Future Play: Research, Play, Share https://www.iso.org/standard/31176.html. (Future Play ’08). ACM, New York, NY, USA, 160–165. 3. International Organization for Standardization. 2010. 13. Lesley Northam and Craig S. Kaplan. 2012. Designing a ISO 9241-210:2010 - Ergonomics of human-system Real Time Visualization Tool for Previsualization and interaction -Part 210: Human-centred design for Virtual Production of Film and Games. In Proceedings interactive systems. (2010). of Graphics, Animation and New Media (GRAND) https://www.iso.org/standard/52075.html. Conference. Montreal, Canada. 14. Brian J. Pohl, Andrew Harris, Michael Balog, Michael 4. Jean-Marc Gauthier. 2005. Building Interactive Worlds Clausen, Gavin Moran, and Ryan Brucks. 2017. Fortnite: in 3D: Virtual Sets and Pre-visualization for Games, Supercharging CG Animation Pipelines with Game Film & the Web. Focal Press. Engine Technology. In Proceedings of the ACM 5. Jason Gregory. 2015. Game engine architecture (Second SIGGRAPH Digital Production Symposium (DigiPro Edition). CRC Press. ’17). ACM, New York, NY, USA, Article 7, 4 pages. 6. Yvonne Jung, Sebastian Wagner, Christoph Jung, 15. Laura Anna Ripamonti, Mattia Mannalà, Davide Gadia, Johannes Behr, and Dieter Fellner. 2010. Storyboarding and Dario Maggiorini. 2017. Procedural content and Pre-visualization with X3D. In Proceedings of the generation for platformers: designing and testing FUN 15th International Conference on Web 3D Technology PLEdGE. Multimedia Tools and Applications 76, 4 (Web3D ’10). ACM, New York, NY, USA, 73–82. (2017), 5001–5050. 7. Martin Maguire. 2001. Methods to support 16. Manlio Scalabrin, Laura Anna Ripamonti, Dario human-centred design. International Journal of Maggiorini, and Davide Gadia. 2016. Stereoscopy-based Human-Computer Studies 55, 4 (2001), 587 – 634. procedural generation of virtual environments. In 8. Claudio Mazza, Laura Anna Ripamonti, Dario Proceedings of IS&T’s Stereoscopic Displays and Maggiorini, and Davide Gadia. 2017. FUN PLEdGE 2.0: Applications XXVII (28th Symposium on Electronic A FUNny Platformers LEvels GEnerator (Rhythm Imaging : Science and Technology). 1–7. DOI: Based). In Proceedings of the 12th Biannual Conference http://dx.doi.org/10.2352/ISSN.2470-1173.2016.5.SDA-042 on Italian SIGCHI Chapter (CHItaly ’17). ACM, New York, NY, USA, Article 22, 9 pages. 17. Michael Schwarz and Pascal Müller. 2015. Advanced 9. Toru Mizumoto, Atsuko Kuramochi, and Ryota Mori. Procedural Modeling of Architecture. ACM Trans. 2014. Viewpoints to Introducing the Human-Centered Graph. 34, 4, Article 107 (July 2015), 12 pages. Design (HCD) Process to the Development Process. 18. Kazuhiko Yamazaki. 2009. Approach to Human Springer International Publishing, Cham, 48–52. Centered Design Innovation by Utilized Paper 10. Jakob Nielsen. 1994. Enhancing the Explanatory Power Prototyping. Springer Berlin Heidelberg, Berlin, of Usability Heuristics. In Proceedings of the SIGCHI Heidelberg, 367–373.