While there have been many advancements in generative models for 3D design, there has been a limited amount of user interface work in this co-creation domain. The user interface controls and interaction paradigms emerging in this field tend to be unintuitive and hard to standardize, as they are often based upon complicated work related to latent space disentanglement, dimensionality reduction, and other bespoke computational techniques. We demo a user interface that provides intuitive controls for the generation of basic 3D animal shapes. These controls, a set of semantic sliders, map to simple and universal operations such as scale and rotation. By adjusting these parameters over animal limbs, users can semantically guide generative models towards their goals, optimizing the mapping between AI action and user intention. Our user interface operates over a generative model that implements Wei et. al.'s semi-supervised architecture for learning semantically meaningful embeddings [1]. To train it, we collected artist data and generated synthetic data by authoring a parametric animal shape generator. This generator produces low-fidelity, abstracted animal shapes we refer to as metashapes. Our system is an instance of a neurosymbolic generative system, which is when the generative system learns dually from data as well as from symbolic, algorithmic constraints. We conclude with an analysis of the benefits and drawbacks of neurosymbolic generation for 3D animal shapes and the utility of metashapes for user control over AI.
1.1. Related Work
works such as variational autoencoders and GANs can learn latent spaces and generatively produce 3D shapes [2]. However, the dimensions of these latent spaces are often highly entangled and too hyperdimensional to be human interpretable. Reverse engineering meaningful signals from latent spaces has become an active area of research. Techniques such as beta-variational autoencoders, InfoGAN [3], and latent space factorization have been developed for the purposes of disentangling the latent space. However, all these methods work to varying degrees of success and tend to be contingent upon the curation and training of large datasets.
Recently, Wei et. al. (2020) proposed an architecture that challenges latent spaces with a semi-supervised model that learns a semantic space. A semantic space can generate instances of 3D object classes after being trained jointly on synthetic and real data; within this space, users can carry out 3D shape editing operations [1].
The demo we present implements the ar2. System chitecture and methods proposed by Wei et. al on the domain of quadruped animals. Wei et. al. demonstrate the success of their method Our system utilizes point clouds as our 3D using state of the art academic datasets on shape representations, as neural networks have well-established generative task domains such shown success on a number of 3D tasks inas chairs, airplanes, and human bodies. We volving point clouds from instance segmenchoose to focus on the domain of animals, be- tation [4] to shape editing to interpolation cause animals are one of the most common [2]. classes of 3D assets created. They display a To create a dataset of animals, we first webhigh variance in their shapes, which make scraped 228 3D mesh assets made public on them far harder to statistically parameterize Sketchfab by artists [5] and sampled point than easier shapes like human silhouettes. In clouds from these assets. Using open3d, a spite of this variance, animals share struc- Python package for 3D graphics, we rotated, tural similarities that humans can intuitively normalized, and scaled our data to fit within characterize. For example, we can generalize a unit sphere, center around the 3D origin, quadrupeds to be four-legged animals with a and face the same direction. head, neck, and a tail, and teach this abstraction to our system. We refer to this abstrac- 2.1. Metashape Generator tion as a metashape: a generic, low-fidelity shape that can abstractly characterize a class of 3D objects. We utilize these metashapes and the aforementioned architecture for our neurosymbolic generative system.
To create a synthetic dataset of 20,000 animal metashapes in accordance with Wei et. al’s architecture, we utilized Blender’s module for Python scripting to spawn metaballs that coagulate into basic animal shapes. Metaballs are 3D primitives common to computer graphics software that can additively or sub- are all in accordance with best practices protractively react to one another to form organic- posed by Llano et. al. for explainable compulooking shapes. Our metashapes were sym- tational creativity systems [8]. bolically parameterized by vector directions, Author interactions with the system efilimb lengths, and limb rotations. Our inspi- ciently exposed the generative model’s abilration for this approach comes from an idea ities and shortcomings. It was moderately long theorized by cognitive science that 3D successful at learning from its semantic sushapes can be decomposed into more basic pervision and able to produce transformations primitives known as geons [6]. in scale and rotation over torso length, neck
We created two versions of the generative length, and tail length. Exploration through model for this demo that work with one con- the user interface produced the varied results sistent user interface. The first is parame- as pictured in Figure 1. However, the model, terized by six semantic axes, the second by in its first iteration with six semantic paramtwenty one semantic axes. These parameters eters, failed to completely disentangle the secorresponded to length, width, height, rota- mantic space. The model showed an on and tion, radius, position, spacing and other la- of ability to control parameters such as neck bels that characterized the primitives corre- rotation, tail rotation, leg length, and tail length. sponding to parts of the animal metashape. By on and of, we mean that while in cerThe exact labels can be found in the appendix. tain clusters of parameters the 3D transfor
These labels supervise the learning of the mations over rotation and scale were accusemantic space and teach the model 3D oper- rate, in other clusters the sliders produced ations such as scale and rotation over specific suboptimal behavior. For example, one speparts. While generating a synthetic dataset cific problem was that the model sometimes from a template is a source of inductive bias, mixed up the posterior extrusion of the neck we attempted to mitigate this by informing with the anterior extrusion of the tail. Anour template with results from Superquadrics, other problem was that the model seemed una recent part-segmentation model that was able to capture the extreme ends of our synapplied successfully to animal meshes [7]. thetic training data (i.e. long legs). More examples of malformed edits are illustrated and captioned in Figure 2.
We present a user interface designed in the 4. Discussion Unity3D game engine, which abstracts over the generative model and allows users to in- In this section, we discuss the following learnteract with it in real-time using semantic slid- ings from this demo. ers. These sliders map to the original axes of the semantic supervision and ofer explain- 4.0.1. Eficient design space exploration ability for the model’s actions. We additionally demo preliminary features for direct ma- Though many generative models can now be nipulation, camera view movement, user his- interacted with in realtime [9, 10], it is often tory, generative AI history, and transparency. still intractable to completely visualize the deThe real-time nature of the interactions, the sign space these models sample over. Howexplainability of the model through the se- ever, our design space is low-dimensional; sixmantic sliders, and the concept of memory dimensional in one version, 21-dimensional in another. Users have access to the entire de- over the design space with universal concepts sign space and can traverse through it within like scale and rotation, mental operations that minutes. They can find best and worst case are intuitive and shared by everyone. The seoutputs within seconds. The eficient explo- mantic meaning attached to each slider optiration that we allow implements the follow- mized the translation of user intention into ing principle established by Gero et. al.: in- AI output. While metashapes are 3D conteractions between humans and AI are im- cepts, they translated well to a user interface proved when humans can eficiently explore that would be intuitive even to non-technical and understand the global knowledge distri- end users. The minimal interface is a good butions underlying generative models [11]. counterexample to the many heavier generative model user interfaces which encourage 4.0.2. Using metashapes as abstractions users to interact with lower-level complexibetween users and AI ties like data distributions [12] and hyperparameters [10].
generation
We present a demo of a neurosymbolic generative system that allows users to create 3D animal shapes with semantically meaningful controls. Additionally, we illustrate how symbolically generated metashapes can be a useful abstraction going forward for human-AI interaction.
neurosymbolic generation with metashapes.
One of the most significant challenges is finding the right metashape abstraction to encapsulate a class of 3D shapes. While methods to find these abstractions do exist [13, 14, 7], it is hard to evaluate the correctness of their abstractions. Furthermore, these methods do not often lend to intuitive user interface con- Acknowledgments trols and metaphors.
Additionally, by incorporating symbolic con- Vivian Liu is supported by a National Science straints from a template, we put a concrete Foundation Graduate Research Fellowship. The number on the degrees of freedom users may authors thank Panos Achlioptas (@optas) for access. While we set this number to generate open-sourcing the loss function used for model a tractable low-dimensional design space, a training, Felix Herbst (@herbst) for contributnumber that is too small limits users and a ing to the coloring function on the point clouds, number that is too big can overwhelm them. and David O’Reilly for open sourcing a large Even if we gave users an exorbitant number pack of 3D animal assets. of degrees of freedom, users could still very well want to go beyond what is ofered and
define their own axes.
The neurosymbolic generation in this system could also benefit from understanding the interplay between real artist and synthetic data. For example, the synthetic dataset behind our ifrst iteration of the generative model created straight legs that pointed downwards. However, the generative model altered the parameter of leg length by extending them not only downwards but also outwards, reproducing patterns present in the shapes of amphibians and reptiles. We optimistically believe that the model was able to generalize some natural variance and establish some correlation between real and synthetic data. However, there were certainly corner cases in which the output was distinctly “real” or “synthetic”. More work could be done to figure out how to investigate and mitigate overfitting, as the generative model was built on a highly asymmetric composition of datasets. [1] F. Wei, E. Sizikova, A. Sud, S. Rusinkiewicz, T. Funkhouser, Learning to infer semantic parameters for 3d shape editing, 2020. arXiv:2011.04755. [2] P. Achlioptas, O. Diamanti, I. Mitliagkas, L. Guibas, Learning representations and generative models for 3d point clouds, 2018. arXiv:1707.02392. [3] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel, Infogan: Interpretable representation learning by information maximizing generative adversarial nets, 2016. arXiv:1606.03657. [4] C. R. Qi, H. Su, K. Mo, L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017. arXiv:1612.00593. [5] D. O’Reilly, Animals, 2020. URL: http:// www.davidoreilly.com/library. [6] I. Biederman, Recognition-bycomponents: a theory of human image understanding., Psychological review 94 2 (1987) 115–147. [7] D. Paschalidou, A. O. Ulusoy, A. Geiger, Superquadrics revisited: Learning 3d shape parsing beyond cuboids, in: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. [8] M. T. Llano, M. d’Inverno, M. YeeKing, J. McCormack, A. Ilsar, A. Pease, S. Colton, Explainable computational creativity, in: ICCC, 2020. [9] A. Ghosh, R. Zhang, P. K. Dokania, O. Wang, A. A. Efros, P. H. S. Torr, E. Shechtman, Interactive sketch fill: Multiclass sketch-to-image translation, 2019. arXiv:1909.11081. [10] Runway AI, Inc., Runwayml, ???? URL: https://runwayml.com. [11] K. I. Gero, Z. Ashktorab, C. Dugan, Q. Pan, J. Johnson, W. Geyer, M. Ruiz, S. Miller, D. R. Millen, M. Campbell, S. Kumaravel, W. Zhang, Mental Models of AI Agents in a Cooperative Game Setting, Association for Computing Machinery, New York, NY, USA, 2020, p. 1–12. URL: https://doi.org/10.1145/ 3313831.3376316. [12] J. Matejka, M. Glueck, E. Bradner, A. Hashemi, T. Grossman, G. Fitzmaurice, Dream Lens : Exploration and Visualization of Large-Scale Generative Design Datasets (2018) 1–12. [13] K. Genova, F. Cole, A. Sud, A. Sarna, T. Funkhouser, Local deep implicit functions for 3d shape, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4857–4866. [14] R. Huang, P. Achlioptas, L. Guibas, M. Ovsjanikov, Limit shapes – a tool for understanding shape diferences and variability in 3d model collections, Computer Graphics Forum 38 (2019) 187–202. doi:10.1111/cgf. 13799.
Iteration 2 with 21 semantic parameters from the following set of parameters: torso length, (front) torso width , (front) torso height, (back) torso width, (back) torso height, a choice between head type 1 (which emphasizes ear variation) and head type 2 (which emphasizes jaw variation), head size, head feature (ear / jaw) prominence, mouth angle, neck length, neck rotation, neck size, leg length, position of front legs, position of back legs, leg gap, leg angle, tail length, tail rotation, tail radius, tail variance, a choice between a tail that increases in width or decreases, and leg radius.