To tackle the challenge of synthesizing and modifying realistic 3D avatars on mobile devices while maintaining optimal display performance, a new method for modifying the geometry of three-dimensional models of user faces is presented. This method combines a basic model with software adaptation and generates a set of morphs representing human facial features, enabling the creation of various unique facial configurations. It is described the working algorithm of the developed method on the iOS platform with the use of built-in tools such as the SceneKit game engine. The result of applying of the developed method and algorithm is a mobile application for the iOS platform that allows users to modify their digital 3D avatars by dynamically altering the geometry of model surfaces. The presented system is compatible with augmented and virtual reality solutions, offering enhanced user experience and promoting long-term use of mobile applications in various industries, including entertainment, video games and augmented/virtual reality systems.
The synthesis of a three-dimensional model of the user's face (3D morphable model [
Modern approaches to creating 3D models of user faces achieve different levels of realism - from realistic to cartoon or semi-realistic. Although realistic 3D models most accurately reproduce the human face and may be a better solution for certain applications such as medical simulations, their synthesis and use may be related with ethical problems. For example, creating a realistic 3D face of a person without his consent can be a violation of his privacy. In addition, using such models for certain purposes (such as impersonating someone or creating fake videos) can have negative consequences for the person whose face was used in the model. Also, synthesizing a realistic 3D face model can require a significant number of detailed and personal data, such as high-quality photos, videos and 3D scans, to accurately convey all the features of a person's facial features.
The question of cultural and ethical biases in the synthesis of realistic digital avatars attracts attention as a potential problem of recent studies [18, 22]. Convolutional networks trained on specific datasets can have difficulty accurately modeling dark-skinned individuals, leading to biased or inaccurate results. Mitigation of these problems requires diversification of training data and constant consideration of ethical reasons throughout the process of creating and using digital avatars. On the other hand, cartoon or semi-realistic avatars can offer some protection from a number of ethical concerns, as they are less likely to be confused with real people and are much easier to programmatically adapt the model's colors. Another problem of synthesized 3D avatars is the mesh geometry editing process due to the limitations of the statistical models used for generating these models.
These models are based on large datasets of 3D scans and can have various artifacts, inconsistencies and non-uniform geometry, which can make editing difficult. Additionally, the generated digital avatars may not have a consistent topology, further complicating the mesh editing process. Despite significant progress in research on the synthesis of three-dimensional avatars in recent years, the issue of the performance of rendering avatars on mobile devices has not been given due attention. With the growing popularity of mobile devices as a platform for consuming 3D content, especially in solutions using augmented reality [23-26], optimized 3D models, as well as solutions for their effective display on these devices, have become a necessity.
When displaying 3D content on mobile devices, the size of 3D models is an important aspect. It affects rendering efficiency and the experience of user interacting with the models. The balance between the size of the model and the completeness of the data, contained in it, must be provided to ensure efficient use of disk space on mobile devices while maintaining the necessary rendering quality, 3D models of the user's avatar.
Achieving such balance is a difficult task, as reducing the size of the model often leads to a deterioration in the quality of the form [21], that significantly reduces their effectiveness and negatively affects the user experience.
Therefore, it is important to develop methods that can optimize the size of 3D digital avatars without compromising their accuracy and quality. These methods should be designed to work optimally on mobile devices with their limited computing power and memory resources. Due to such optimization, it is possible to achieve improvements in the ease of interaction and the effectiveness of displaying digital avatars on mobile devices. It can help in solving a number of product tasks. For example, the next tasks: increasing the level of engagement and retention of product users or wider use of these solutions on the platform.
This work focuses on the development of methods of dynamically change the geometry of digital user avatars, taking into account the problem of rendering performance of three-dimensional objects on mobile platforms.
In researches focused on photorealistic 3D face reconstruction [12, 13], artificial neural network systems and proprietary methods of capturing and reconstructing detailed 3D models of human faces are usually used. This allows for a high level of realism and customization. It is common to use the transformation of a 2D user image [13] to create a 3D mesh, or even neural network convolutions in a 3D mesh space directly [12].
Although such an approach currently shows the most realistic result in the industry, such solutions still have a number of drawbacks. They are taken into account in the methods developed in this work. For example, it is the difficulty of correctly reproducing dark skin tones, which is related to the data sets used during the training of neural networks. In addition, there is an ethical concern about the possibility of using photorealistic 3D models for malicious purposes.
An alternative to photorealistic 3D reconstructions is the synthesis of three-dimensional caricature faces, which include distortion and exaggeration of certain facial features for comedic or artistic effect [14, 15]. Although such result may be desirable in some industries (such as entertainment or advertising), the results of these methods are highly specialized and have many limitations in use. For example, it is the use in medical imaging or in cases of advertising goods and services in the field of beauty.
It is also important to note that the extremeness and unrealistic nature of the distortion must be taken into account when designing ethical products. This is especially important to ensure that these decisions do not have unintended negative consequences.
This article proposes methods that take into account these ethical implications. In turn, this enables development of products and solutions that are not only efficient but also ethical.
Most similar to our developed solutions are studies related to the creation of semi-realistic 3D avatars. Some of them also use the approaches of training their own neural networks to determine the characteristics of the user's face and create a three-dimensional model, as in works [16, 17].
In work [18], these decisions are strengthened by searching for the most suitable properties of the user's image among the three-dimensional models available in the library, such as glasses, hair on the head or face.
However, it is important to note that the authors of the mentioned works did not consider the possibility of changing or customizing the obtained 3D face, and also left open the question of mirroring the user's facial expressions for augmented or virtual reality systems.
Work [19] is aimed at photorealistic 3D face reconstruction. However, here the authors also consider the issue of displaying the user's facial expressions, but in a slightly different way than in our research. Also, the mentioned work considers a mobile device for a more mobile and affordable face synthesis tool. However, it is used only as a reading device, while all calculations and the synthesis take place on a PC.
It is worth noting that all related works focus on the creation of a 3D model of the user's face, but usually do not describe the problems of integration into a specific game engine and the means of interaction between the user and the synthesized face, especially on mobile devices.
Accordingly, the aim of our research is to improve the efficiency of the process of dynamically changing the geometry of 3D surfaces based on the development of a method for the SceneKit game engine.
The object of the work is the process of dynamically changing the geometry of 3D surfaces for the SceneKit game engine.
The subject of research is the method and means of dynamically changing the geometry of 3D surfaces for the SceneKit game engine.
To realize the goal, the following tasks must be solved: to conduct a literary analysis of the methods of dynamic change of the geometry of 3D surfaces; to develop a method of dynamically changing the geometry of 3D surfaces for the SceneKit game engine; to develop an algorithm for system operation using the method in point 2; to conduct research on the developed method of dynamically changing the geometry of 3D surfaces in the SceneKit game engine on the iOS platform.
The developed method is based on the combination of a ready-made 3D model and its software adaptation. This approach differs from previous studies that relied exclusively on fully software implementations. As a result, this approach gives the user the opportunity to adjust the synthesized model.
The algorithm of the developed method is presented in Figure 1. It includes the preliminary creation of a set of morphs of human facial features.
This makes it possible to fill a three-dimensional model with a large number of geometric combinations, keeping the size of the original model relatively small for porting to a mobile OS [27] (5 MB without taking into account the set of textures).
Expanding the set of existing morphs Creating a basic
3D model Creating a set of morphs of facial
features
Combining and correcting morphs with
each other Software adaptation of
the model No
Is the desired number of element variations achieved?
Yes Creating a 3D model in
USDZ format Further model rendering and integraion
End
In the course of the work, it was developed a set of morphs, which are used to create unique configurations of various facial elements, such as the nose, eyes, lips, cheeks, eyebrow shape and chin.
This set is represented by the formula = ∑ ∈ , (1) where is a set containing all morphs, is the weight coefficient of the specific morph setting, is a geometric representation of each designed morph of the user's face, is a number of morphs.
Since adjusting the weighting coefficients of one morph leads to change of the model geometric grid, the formula is represented by the sum of morphs. The change one of them will geometrically affect all other available morphs. It is worth noting that the given set of morphs can be modified and expanded if necessary. At the same time, changes at the model level will not require additional changes in the implementation logic of the software part.
The base model and morph set of human facial features are designed using Blender open-source software. The purpose of this work is to use the developed model on a mobile device, so 3D files must be supported by the system game engine.
According to studies in work [20], the USDZ format is the most optimal for porting to the iOS mobile operating system, and is also supported by built-in game engines.
However, work [21] investigates the problems of exporting three-dimensional models from their development environments to the USDZ format, in particular, the formation of folds (geometry deformations) and the problems of displaying model animations.
Also, in the course of this research, the problems of presenting the developed morphs in the source file of the three-dimensional model in the USDZ format were revealed. To solve the listed conversion problems, three-dimensional model adaptation software was used.
It was developed the algorithm of the system of dynamic change of three-dimensional surfaces of the user's avatar, taking into account the built-in methods and tools in the iOS operating system.
Adjustment of the weighting coefficients
Start
Integration of the model on iOS plaform
Processing of the model with the SceneKit game engine
Aggregation of the
model’s morphs Rendering of the model in the SceneKit game engine No
Is the desired accuracy of the model achieved?
Yes Further integration of the configured model
End
Step 3. Search for all available morphs in the model. The developed at the modeling stage morphs that responsible for adjusting the features of the user's face (shapes of the nose, eyes, etc.), are grouped into data sets for adjustment in the following steps.
Step 4. Render the SCNNode object of the 3D model using the SceneKit game engine.
Step 5. Due to interacting with the customization interface, configure the existing morph sets until the accuracy of the avatar display satisfies the user.
The block diagram of the algorithm of the system for dynamically changing the geometry of the user's three-dimensional avatar on the iOS platform is shown in Figure 2.
The proposed scheme (Figure 3), which consists of three main parts, intends to outline how the created method should be used from the user's or client-side perspective.
Firstly, it employs a predefined model with modification morphs, allowing users to easily adapt and customize the model according to their needs.
Secondly, the schema considers user input for morphs weight modification, enabling individuals to fine-tune the model with personalized adjustments, creating a more tailored experience.
Lastly, to ensure optimal performance on mobile devices, the method incorporates mobile rendering through built-in tools such as SceneKit. This approach, with its user-centric focus, seeks to revolutionize the 3D modeling experience by providing a seamless and intuitive method that caters to the needs of users and clients alike.
The developed system, as outlined in the provided text, presents a comprehensive method for generating and modifying customizable avatars within a virtual environment. By utilizing a tuple of data, the system effectively incorporates a basic three-dimensional model ( ) that serves as the foundation for the avatar creation process. The set of morphs ( ) offers a collection of customizable elements that enable users to personalize their avatars according to their preferences.
The result of the developed method and algorithm is a mobile application for the iOS platform. The application is developed in the Swift programming language, and allows the user to uniquely customize a digital 3D avatar by dynamically changing the geometry of the model's surfaces.
An example of the work of the developed method is presented in Figure 4 and Figure 5.
As can be seen in Figure 5, the geometric mesh of the model changes according to the user-defined settings of morphs weight coefficients.
To ensure compatibility across different platforms, the system includes a set of software and hardware ( ) specifically designed for displaying the three-dimensional model. Lastly, the set of morph modification operations ( ) provides a range of actions for further refining the avatar's appearance, granting users the flexibility to make adjustments as desired.
This robust approach to avatar creation not only simplifies the process but also fosters a more immersive and engaging user experience in virtual environments.
The developed system is suitable to scalable. Due to the use of the USDZ format, it saves morphs in an optimized way compared to common formats in the industry, such as Collada, FBX or GLB.
The developed system can be implemented in augmented and virtual reality solutions, as the rendering process takes place using the SceneKit game engine. Combined with the ARKit system library, the game engine allows to display and manipulate 3D models in the real world.
Unlike existing solutions [12, 13], which rely on convolutional networks to create realistic digital avatars, the proposed our methods avoid the ethical problems associated with the creation of unauthorized digital avatars that can be used against a person whose face was synthesized.
Compared to the caricature synthesis methods presented in works [14, 15], the developed our solutions are more balanced and carry a neutral (calm) expression of the geometry of the user's digital avatar in order to prevent any potential harm from excessive extremization of the synthesized face.
Available semi-realistic solutions in works [16, 17] also focus on neutral digital avatars, but do not include methods for editing a three-dimensional model by the user.
The work [18] also presents the methods of synthesis of a semi-realistic three-dimensional model, which make it possible to select and adjust such additional characteristics of the model as glasses or a hairstyle. But these methods do not describe the means of editing the geometry of elements of the user’s face, such as the nose or mouth, as was done in the course of this scientific work.
The paper [19] also describes the importance of the three-dimensional avatar synthesis system for optimization on mobile platforms. However, the mentioned work does not present optimization solutions for a specific mobile platform. And the synthesis, apart from the collection of input data, takes place on a PC. The work [19] also uses the approach of combining pre-developed morphs with software settings already during synthesis, but the mentioned approach is not used for user emotion morphs.
The methods described in our work, in turn, involve the use of a similar approach to change the geometry of the avatar's face, and also involve the combination with morphs of the user's emotions, which significantly expands the user functionality compared to work [19].
In addition, the method developed in this work describes the algorithm for rendering and manipulating a synthesized avatar on the iOS mobile platform using the USDZ file format and the SceneKit game engine. This approach provides a more optimal way to store both the model mesh and the morphs, which leads to increasing of efficiency when porting the model to the platform, as well as when scaling the model to a larger number of morphs.
Size of the synthesised model (MB) Collada
Fbx
Glb USDZ 0 20 40 60 80 100 120 140 General
Resources
With morphs
Without morphs
In addition, using SceneKit as a game engine it is ensured smooth and accurate rendering of 3D models, thereby providing a more immersive and realistic user experience. Along with this, the USDZ file format is optimally supported by the iOS platform, which ensures wider compatibility and integration with other applications.
Thus, the method proposed in this paper is an optimized and effective tool of visualization and manipulation of the user's digital avatar on the iOS platform, which improves the user experience while maintaining a high level of efficiency and scalability.
To compare the volume of the source files, we considered the developed three-dimensional model (Figure 4) with the same set of morphs. This model has been exported to the most common 3D file formats such as Collada [28], Fbx [29], GLB and USDZ [30].
As can be seen in the comparison diagram (Figure 6), the model in the USDZ format takes up the least amount of disk space compared to the others.
It is worth noting that the yellow and orange colors on the diagram correspond to the size of the model without including morphs and with their addition to the file structure, respectively.
According to the obtained data, the use of models in this format shows better results in both cases.
Three-dimensional models can often use textures (UV maps) to give visual color to a 3D object. In the synthesized model, the set of texture resources is unchanged and occupies the same amount of disk space regardless of the file type. It is represented in green in the diagram.
Thus, the output size of the file that will be used when porting and rendering a 3D model on a mobile device, is equal to the sum of the amounts of resources and the storage structure of the 3D model. That is brown color in the diagram.
The developed method of dynamically changing the geometry of 3D models using the USDZ format gave better results compared to other popular methods using such formats as COLLADA, FBX and GLB. Using the capabilities of the USDZ format it is created a 3D model with less weight, ultimately resulting in faster load times and improved overall performance. This is especially important for applications in areas such as AR/VR, where reaction speed is important.
Additionally, the USDZ format provides seamless compatibility with Apple platforms, that provides an advantage over other approaches.
Thus, using USDZ to create lightweight 3D models proved to be a more efficient and versatile solution compared to traditional methods based on COLLADA, FBX or GLB file formats.
The approach highlighted in the research marks a significant stride amidst the escalating demand for interactive and immersive mobile applications. By giving users more control over avatar customization, it not only transforms the experience in AR and VR environments, but also adds depth to the virtual user experience, leading to increased user engagement. Furthermore, the innovative development outlined in the research isn't exclusive to a particular domain. It has potential applications spanning across diverse sectors like gaming and social media. The resulting improvement in visualization and customization of digital avatars can be a game-changer in enhancing user engagement and satisfaction. This could significantly contribute to higher retention rates in digital platforms, a key metric in the success of these applications. As such, the approach detailed in the study suggests the possibility of subtly influencing the direction of mobile applications and virtual platforms towards a more user-centric customization strategy.
The paper analyzes the existing methods of synthesis of three-dimensional avatar of the user on mobile devices. The problems of existing solutions for modeling realistic, semi-realistic and animated 3D avatars of the user have been formulated.
It has been established that high-quality realistic solutions based on neural networks have a number of unresolved ethical problems in the field, and are also difficult to undergo geometric modifications. Existing semi-realistic and animation solutions may include methods for modifying the model by the user, but do not describe methods for optimizing the display of a digital avatar on mobile devices.
It was developed a method and was described the algorithm for modifying the user's synthesized digital avatar by combining the basic model with software adaptation.
The approach differs from previous studies that relied solely on software implementations, allowing the user to solve the problem of editing the synthesized model. The method involves creating a set of morphs of human facial features that can be used to generate various unique facial configurations. The set is represented by a formula containing weighting coefficients and a geometric representation of each morph.
It was developed the algorithm for the operation of developed methods on the iOS platform using built-in methods and tools of the platform, such as the SceneKit game engine.
The result of the developed method and algorithm is a mobile application for the iOS platform that allows the user to modify a digital 3D avatar by dynamically changing the geometry of the model's surfaces. The developed system is scalable and due to the use of USDZ format, it optimally preserves morphs compared to industry-wide formats such as Collada, FBX or GLB. The system can be implemented in augmented and virtual reality solutions.
Using the example of the developed method, it was demonstrated that the geometric mesh of the model changes according to the settings of the morph weights specified by the user.
The proposed solutions can be applied in various fields, including the entertainment industry, video games, as well as augmented and virtual reality systems. In addition, the use of developed techniques for creating digital avatars can improve the user experience and facilitate the long-term use of mobile applications in numerous commercial contexts.
This work was realized within the framework of the program Erasmus+ Jean Monnet Module «Augmented Reality for Education: implementation of European experience» (101085772 – AR4EDU – ERASMUS-JMO-2022-HEI-TCH-RSCH).
[12] A. Ranjan, T. Bolkart, S. Sanyal, M.J. Black. "Generating 3D faces using convolutional mesh autoencoders", Proceedings of the European conference on computer vision (ECCV), 2018, pp. 704–720, doi: 10.48550/arXiv.1807.10267. [13] S. Lombardi, J. Saragih, T. Simon, Y. Sheikh, Deep appearance models for face rendering. ACM
Transactions on Graphics, 37 (2018), 1–13, doi: 10.1145/3197517.3201401. [14] Z. Ye, M. Xia, Y. Sun, R. Yi, M. Yu, J. Zhang, ... Y.J. Liu. 3D-CariGAN: an end-to-end solution to 3D caricature generation from normal face photos. IEEE Transactions on Visualization and Computer Graphics, 2021, doi: 10.48550/arXiv.2003.06841. [15] H. Cai, Y. Guo, Z. Peng, J. Zhang, Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model. Graphical Models, 115 (2021), 101103, doi: 10.48550/arXiv.2004.09190 [16] T. Shi, Z. Zuo, Y. Yuan, C. Fan. "Fast and robust face-to-parameter translation for game character auto-creation", Proceedings of the AAAI Conference on Artificial Intelligence April 2020, 34(2); pp. 1733–1740, doi: 10.1609/aaai.v34i02.5537. [17] J. Lin, Y. Yuan, Z. Zou. "Meingame: Create a game character face from a single portrait", Proceedings of the AAAI Conference on Artificial Intelligence, May 2021, 35(1); pp. 311–319, doi: 10.1609/aaai.v35i1.16106. [18] S. Sang, T. Zhi, G. Song, M. Liu, C. Lai, J. Liu, X. Wen, J. Davis, L. Luo. "AgileAvatar: Stylized 3D Avatar Creation via Cascaded Domain Bridging", SIGGRAPH Asia 2022 Conference Papers, November 2022; pp. 1–8, doi: 10.48550/arXiv.2211.07818. [19] A.E. Ichim, S. Bouaziz, M. Pauly, Dynamic 3D avatar creation from hand-held video input.
ACM Transactions on Graphics (ToG), 34 (2015), 1–14, doi: 10.1145/2766974. [20] D. Ostrovka, T. Teslyuk, P. Vesely, I. Prots’ko. "The analysis and comparsion of file formats for constructions of iOS three-dimensional objects for augmented reality systems", 1st International Workshop on Digital Content & Smart Multimedia, Ukraine, Lviv, 2019, pp. 325–334. [21] D. Ostrovka, V. Teslyuk. "The analysis of file format conversion tools for storing 3D objects for the iOS platform", MoMLeT&DS, Ukraine, Lviv, 2020, pp. 341–350. [22] B. Egger, W.A. Smith, A. Tewari, S. Wuhrer, M. Zollhoefer, T. Beeler, ... T. Vetter. 3d morphable face models – past, present and future. ACM Transactions on Graphics (TOG), 39 (2020), 1–38, doi: 10.1145/3395208. [23] J. Ferrão, P. Dias, B.S. Santos, M. Oliveira, Environment-aware rendering and interaction in web-based augmented reality. Journal of Imaging, 9 (2023), 63, doi: 10.3390/jimaging9030063. [24] W. Broll. Augmented Reality. Virtual and Augmented Reality (VR/AR): Foundations and
Methods of Extended Realities (XR), Springer: Amsterdam, The Netherlands, 2022, 291–329. [25] M.J. Liberatore, W.P. Wagner. Virtual, mixed, and augmented reality: A systematic review for immersive systems research. Virtual Real, 25 (2021), 773–799. [26] D. Ostrovka, V. Teslyuk. The current state and prospects of advanced reality technology in smart mobile devices. Selected aspects of digital society development: monograph 45. Publishing House of University of Technology, Katowice, 2021, pp. 179–190. [27] M.M. Mirza, A. Ozer, U. Karabiyik. Mobile cyber forensic investigations of web3 wallets on
Android and iOS. Applied Sciences, 12 (2022), 11180: 1–35, doi: 10.3390/app122111180. [28] Collada File Documentation Homepage. URL: https://www.khronos.org/collada/ (date of access: 10.04.2023). [29] Fbx File Documentation Homepage. URL: https://www.autodesk.com/products/Fbx/overview (date of access: 10.04.2023). [30] USDZ File Format Explanation Topic. URL: https://medium.com/techinpieces/all-aboutapplesnew-usdz-file-format-simplified-12dff29f3fc0 (date of access: 10.04.2023).