Digital Culture & Audiovisual Challenges: Interdisciplinary Creativity In Arts And Technology DYNAMIC GENERATION AND EXPLORATION OF SYMMETRICAL TEXTURES FROM REAL-TIME CAMERA INPUT ON MOBILE DEVICES Nikitas M. Sgouros Professor, Dept. of Digital Systems University of Piraeus, Greece sgouros@unipi.gr Abstract: Symmetrical patterns play significant roles on the development of the aesthetics of various art forms. The wide use of such patterns in modern design creates the need for digital tools that can facilitate their creation, experimentation and use by artists and designers. Responding to this need, we describe Symmetricon, our Android art/design app for mobile devices that supports real-time visual exploration of a multitude of symmetry types. In particular, we explore animated texture design based on the application of plane symmetry groups on real-time camera input and projected on various type of surfaces. Our art/design app is freely available on the Google Play store. We provide example videos for its use and possible application on YouTube. Keywords: Geometric design, symmetry, digital authoring tools, mobile computing. Introduction Symmetrical patterns have been a constant source of inspiration for various visual art forms by evoking feelings of order, harmony or balance [1]. Furthermore, the existence of detailed mathematical descriptions for generating all possible symmetries on the plane has opened up the possibility of efficient, algorithmic generation of such patterns. However, the kinds of iconography that can be generated from symmetry is vast and can be made even bigger by combining symmetrical forms with color or motion animation or by introducing noise in such patterns in a principled way. In this case, the important question that concerns us is how we can efficiently help and motivate digital artists to create and experiment with symmetrical designs that are visually interesting and aesthetically pleasing. Our efforts in dealing with this question focus on the creation of digital environments for creative visual exploration of symmetrical patterns and their interplay with geometry and animation. We want to create tools that can be easily deployed by artists and designers, therefore we target mobile devices as our main development platform. To this end we have created and deployed Symmetricon [2], a free Android app for generating, animating and displaying, in real-time, plane symmetry patterns. In the rest of this paper, section 2 gives a brief description of the types of Copyright © 2018 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) DCAC 2018. Digital Culture & Audiovisual Challenges: Interdisciplinary Creativity In Arts And Technology symmetry generated by Symmetricon, while section 3 explains the algorithms used for the efficient creation of such patterns in real time. Section 4 describes the use of the system and provides some examples of the designs that are possible with it. Finally, section 5 is a conclusions and future work section. Symmetry We think of symmetry as a system for organizing design components, referred to as tiles or cells, that are repeated in space. The repeated filling of space with tiles creates tessellations, i.e., complete coverings of the space without any gaps or overlaps. When we restrict ourselves to the two- dimensional space there are 17 ways by which cells can be combined to generate tessellations. We refer to them as the 17 wallpaper groups [3]. The basis of symmetrical surface design is to have a cell that orients in the same direction and repeats with no gaps or overlaps. Therefore, only certain shapes can be used as primary cells (i.e. the shapes that will be combined to form the final repeating cell). These are the triangles, quadrilaterals (all planar shapes with four sides) and the hexagon. In the case of quadrilaterals, if opposite sides are not equal in length or parallel to each other, then the tessellation process becomes more involved with various translation and rotations applied to each one of them in order to fit seamlessly in the tessellation. Triangles must also be combined with one of them turned upside down to form a quadrilateral cell. All final repeating tiles are generated by four motions of the primary cell (see Fig. 1); translation, rotation, reflection and glide. Translation moves the primary cell sideways, horizontally, vertically or diagonally without a change in its orientation. Rotation turns the cell a certain number of degrees either around its geometrical center or around one of its vertices. Reflection flips the cell either horizontally or vertically. Finally, glide is a combination of a translation and a reflection. In this case, the cell can either be translated horizontally/vertically or then flipped vertically/horizontally. All 17 possible symmetry groups on the plane are labeled with a notation system stemming from the one used by crystallographers to classify crystals but restricted to the plane [5]. The notation describes the combination of all the operations that are performed on a primary cell (translation, rotation, mirror, glide) to create a final tile that repeats through horizontal and/or vertical translation to produce the tessellation. Symmetricon uses this standard notation. It works as follows [6]: • The first letter in all the groups is either a p or a c. The p stands for the primary cell, the c stands for centered cell which is akin to a primary cell. • A number in the second position indicates the maximum order of rotation that is done in the symmetry. The center of rotation is determined by the letters that follow the number. An m directly following the number means that the cell must be mirrored first and then rotated along the angle formed by the mirror. If m comes in the fourth position after another letter, the primary cell must first be rotated and Digital Culture & Audiovisual Challenges: Interdisciplinary Creativity In Arts And Technology then the mirror applied to the result. For example, p2 means the cell is rotated two times (180°); p3 means it is rotated three times (120°); p4 means the cell is rotated four times (90°); and p6 means it is rotated six times (60°) to produce the final repeating cell. If there is no number in the label, it means that no rotation is applied during the creation of the final tile. • The third letter (or second if there is no number in the label) indicates the first operation that is done to one of the sides of the primary cell (m = mirror, g = glide, 1 = no operation prior to rotation). • The fourth position (or third if there is no number in the second position of the label) is also a letter and indicates what is done to an adjacent side of the cell. For example, p4g means that the final cell consists of the primary cell rotated four times and then the resulting cell is glided, while pgg means that the primary cell is glided both horizontally and vertically. Generation of Symmetrical Textures The symmetrical texture generation process in Symmetricon receives as input the frames captured from the camera stream of a mobile phone. These frames comprise the primary cells for the subsequent operations. Symmetricon then generates the final repeating tile which is either a rectangle or a hexagon depending on the symmetry group specified by the user. This tile is a bitmap image which is passed as a texture unit to the fragment shader of an OpenGL environment. This is straightforward in the case of rectangular tiles. In the case of hexagon tiles, given that textures are rectangular in OpenGL, the hexagon tile is turned into a rectangle by adding the top left triangle to the bottom right side of the hexagon and so on. The rest of the operations consisting of all the motions (translations, rotations, reflections and glides) applied to the repeating cell in order to create the tessellation are being performed in the fragment shader in parallel thus speeding the tessellating process and achieving real-time performance. In parallel with the generation of the tessellation the fragment shader computes the animation pattern specified by the user in the GPU as well. This can either be a horizontal/vertical translation of adjacent rows/columns of the tessellation, a rotation of a 4x4 tile block around its geometrical center or non-linear vortex and tunnel effects. The resulting tessellation is being pasted upon either a square surface, a sphere surface, a square grid (i.e., a square surface sampled with a grid of points) or a combination of the above depending on the user choice. These surfaces are created in the vertex shader of the OpenGL environment and are centered on a 3D coordinate system. The user can select a variety of animations that affect the position of the vertices of each surface displayed thus creating various wave and other geometrical deformation special effects. Digital Culture & Audiovisual Challenges: Interdisciplinary Creativity In Arts And Technology Figure 1: Examples of (a) horizontal translation, (b) rotation about the point of intersection of the diagonals, (c) vertical reflection, and, (d) glide. The black line in (c) and (d) depicts the axis of reflection. Digital Culture & Audiovisual Challenges: Interdisciplinary Creativity In Arts And Technology User Interface (UI) Symmetricon allows the user to select the types of symmetry groups he wants to use, the size of the tessellation along with the specific texturing surface (square, sphere or rugged plane) he wants to create and the type of animation he wants to employ on the pattern (none, horizontal, vertical, cyclical, vanishing, tunnel or pulsating effects). Based on these user specifications, the system creates a tessellation of the desired symmetrical pattern and embeds the resulting texture in a 3D scene. The user can then view the generated texture from any distance or viewing angle in this 3D space. He can also store the textures he finds interesting as images in his phone. Given that Symmetricon is a mobile app, we have sought to minimize all UI elements so that we can maximize the display of the tessellation on the device screen. The app window consists of a toolbar at the top that contains all the user interaction options and the main screen for displaying the tessellations (see [7] for a YouTube video depicting the system in use). Toolbar The toolbar contains a series of menu items for interacting with the app: • the Symmetry option displays a menu of the 30 possible symmetry types supported by the system. Selecting any of the symmetries in this menu results in the application of this patricular symmetry on the app input. The names for each symmetry type correspond to their crystallographical labeling. The complexity of generating each symmetry group is not the same, therefore some groups (e.g. P6) may take more time to be displayed than others. • the Repeats option determines the size of the tessellation. The size is the number of times that the cell will repeat in the horizontal and vertical directions. The current version of Symmetricon creates only square tessellations in which the number of repetitions is the same horizontally and vertically. • the Input option opens a menu for selecting the input for creating the primary cell. Possible inputs are the front/back camera or a user-supplied image/video. • the Animation option opens a menu for selecting the animation pattern that will be applied • to the tessellation. Possible animation patterns include sliding of the tessellation rows or columns along with cyclical motions of adjacent cells. Furthermore, we offer versions of the vanishing, tunnel and vortex effects. • the Surface option opens a menu for selecting the surface on which the resulting image will be textured. Possible options include a square (default), a sphere, a combination of both or a rugged square surface. • the Take Snapshot option allows the user to take a jpeg snapshot of the screen contents and store the resulting image. • the Set/Exit full screen options allow the app to use all available space in the device screen. • the Help option displays a help dialog. Main Screen The main screen allows the user to change the angle and distance from which he Digital Culture & Audiovisual Challenges: Interdisciplinary Creativity In Arts And Technology observes the pattern by dragging gestures left/right and up/down. In each of these cases the textured surface always appears at the center of the screen. Using pinching gestures the user can zoom in or out of the scene. Examples Symmetricon provides the artist/designer with the opportunity to generate and test various motifs under an unlimited set of continuous and constantly changing imaging conditions. This enables the app to be used as a visual exploration environment that allows its users to create infinite color and geometrical variations of symmetrical textures using as raw material real-world scenes captured by the camera of a mobile device. In this case, the artist/designer can control the camera position and motion of the mobile device during the capturing of the raw material while also applying on it various animation patterns and projecting it on various surfaces. A video showing a selection of patterns created by an extended version of Symmetricon while randomly scanning a typical office space with the camera of the mobile device can be found in [4]. The particular app version used in the video is able to project the textures simultaneously on an unlimited set of texturing surfaces where each of these surfaces is specified by the user using 3D mathematical functions. Figure 2 provides some more examples of the possible tessellations that Symmetricon can create. The images used for the creation of these patterns are the frames captured from the camera stream of a mobile phone. Figure 2: Three examples of tessellations generated by Symmetricon. Digital Culture & Audiovisual Challenges: Interdisciplinary Creativity In Arts And Technology Conclusions & Future Work We have described Symmetricon, our art/design app for creative exploration of planar symmetrical textures and their interaction with various surface geometries and animation patterns. Symmetricon is able to create 30 types of symmetries in real time from either the camera stream, or images and videos stored in a mobile device. There exist a variety of mobile apps that focus on the generation of symmetrical patterns (e.g. Kaleidoscope Pro, Picasso, etc.). Most of them are drawing apps that allow the user to create the primary cell and then generate mainly kaleidoscope tessellations based on it. To the best of our knowledge Symmetricon is the only existing app that can generate animated, symmetrical tessellations from real-time camera stream and apply them as textures to various surfaces. Our future research efforts will focus on speeding up further the tessellation process by parallelizing the primary cell generation which currently is executed on the CPU. In addition, we are exploring the dynamic generation of symmetrical patterns and the iconography they produce in their use in non-Euclidean geometries and in mobile virtual reality environments. Furthermore, we are pursuing collaborations with visual artists for the use of this technology in common projects along with the development of novel methods for the integration of such camera-based symmetrical texturing in virtual and/or mixed reality environments. Acknowledgements Presentation of this work was partially supported by the University of Piraeus Research Center. References Beyer, J. (1999). Designing Tessellations: The Secrets of Interlocking Patterns. McGraw- Hill. https://www.youtube.com/watch?v=Sseob7zSNDU&feature=youtu.be Farris, H. A. (2015). Creating Symmetry: The Artful Mathematics of Wallpaper Patterns. Princeton University Press. Kappraff, J. (1991). Connections: The Geometric Bridge between Art and Science (2nd edition). World Scientific Publishing Company. https://play.google.com/store/ apps/details?id=org.epinoetic.symmetricon Schattschneider, D., (1978). The plane symmetry groups: Their recognition and notation, The American Mathematical Monthly, 85 (6): 439–450.