User interface prototyping is an eficient approach for eliciting user requirements in the early phase of software development. It is an iterative analysis technique in which users are actively involved in the mocking-up of the UI for a system. It is typically done in a drag-and-drop fashion by professionals, which is time-consuming and requires specialized knowledge. This paper presents a prototype tool SAIP (Semi-Automatic user Interface Prototyping), which allows users to express their natural language requirements and semi-automatically generates a corresponding graphical user interface. Based on a systematic literature review of the user interface prototype, we design a framework that maps natural language to the user interface prototype. Furthermore, develop a demo version of the SAIP - a user interface prototyping tool served from natural language processing and neural network for the user interface automatic generation. Our tool aims to help users who have little knowledge in user interface design to express their ideas and requirements freely.
User Interface prototyping is an essential part of software engineering, efectively facilitating requirements elicitation. It is typically done in a drag-and-drop fashion, using prototyping software such as Adobe XD1. However, it is time-consuming to learn and master such techniques. The prototypes of mentioned software are designed by individual users and then uploaded on the oficial website to allow other users to edit them, which will duplicate styles.
In recent years, many researchers and developers have experimented with the automatic
generation of user interfaces. As early as 2005, Nichols et al. proposed model-based concepts
to solve automatic interface generation from a theoretical level. [
This paper presents a practical prototype tool that takes natural language as input and directly generates corresponding user interface prototypes using NLP (Natural Language Processing) and image processing techniques. It can generate customizable interface prototypes that are non-repetitive.Compared with commercial products generated by models such as Canva, its prototype recommendations and functions are recommended based on user popularity and downloads. This means that the more exquisite and the higher the prototype’s ranking, the more users it is, the more likely it is to be reused by diferent people. When designing products, designers hope to refer to more innovative prototypes.
For products such as WordPress that provide static templates, we believe that in the early stages of prototyping, the introduction of visual templates will afect users’ freedom of expression and limit the width of users’ thinking. Using text, a more abstract and faster way, can give users more room for innovation and design more novel prototypes. Specifically, our proposed prototype functions in a pipeline manner, which recommends various interface designs to users.(e.g., layouts and colors of diferent components) If users’ requirements are not satisfied by the recommended solution, our tool also provides a self-development mode, allowing users to customize the designed interfaces. At present, SAIP already has a demo version - a demonstration video can be found here. 2
According to interaction patterns in user interfaces, it can be understood that the layout of user interfaces can be broadly classified into a limited number of categories. Moreover, these categories correspond to website features that are distinct and well-diferentiated. For example, we naturally think of YouTube for streaming sites, which uses a peer-to-peer grid pattern to show as many video covers as possible to attract viewers.
We then classify the natural language input of users by manually labeling the corresponding layouts of websites with diferent features and then training the framework through neural network. In this form, we can determine what type of website is roughly described by the user’s natural language and adopt the mainstream layout to meet the user’s needs. Specifically, our proposal starts from word tokenization, which is an inevitable step for processing Chinese. Then, we conduct TF-IDF (Term Frequency–Inverse Document Frequency) to extract key words, identifying the core descriptions of user inputs.
The core descriptions are then inputted into the neural network model. Our labeled training data-set trains this model in advance, which can classify the user input requirements and map them to a specific layout. If there is no successful mapping (i.e., the classification probability is below a threshold), it defaults to the most widely adopted center stage layout.
Impressive color style is essential to catch the user’s attention at first glance. For better aesthetics and simplicity, modern single-page applications should ideally have no more than three colors, with Primary Color for mainframes and components and Secondary Color for secondary accents.3 So the question turned to - how to extract the color preference from the natural language description?
The human brain can imagine diferent colors directly through spoken words. For example, people can quickly think of colors like black, gray, and navy for business style. For another example, steak, people can easily associate with red or brown. Nevertheless, computers are dificult to associate colors directly from natural language except for much training for computers. In daily life, people often use natural language in search engines to search for images. Images 3https://material.io/design/color/the-color-system.html#color-theme-creation (a) Color before and after de-duplication (b) Improved ItemCF algorithm can be transformed into colors using extraction algorithms. Thus, our approach is to use search engines to transform natural language into images and then use a color extraction algorithm to extract the potential colors from the images.
However, it would be very messy to give this result to users directly. First, there are multiple
images for each term. In this case, too many colors will be recommended to the user. Secondly,
the duplication rate of colors would be very high. The color extraction algorithm is segmented
based on pixel points, and many colors have slight diferences in RGB, but the colors are almost
indistinguishable to the naked eye. So the tool needs to de-duplicate the colors to remove the
similar ones based on their Euclidean distances in RGB space [
In de-duplication process, if a color is removed because of duplication, it means that it is too similar to another color, meaning that this color appears more often. So our prototype tool records the weights of diferent colors. When a color is de-duplicated, then tool will boost its weight and prioritize the subsequent recommendations.
The impact of doing this are twofolds. First, if the user input requirements are too distinct,
all the colors are more similar, and fewer colors are available after de-duplication. The efect
of de-duplication is shown in Figure 2(a). Second, the tool’s de-duplication algorithm keeps
the higher weight of the two colors, which may remove the color that the user wants. The last
step of color processing is that our tool uses an improved Item-based Collaborative Filtering
(ItemCF) algorithm [
After the above two steps, the tool extracts the layout and color scheme from the user’s requirements, and all that is left is to render it to the front-end. The approach we currently use is to pre-set up some front-end templates that can be inserted. These templates have distinct layout features, and each template corresponds to the layout strategy. The tool modifies the color scheme of the diferent components to meet the user’s color scheme requirements. Simultaneously, a drag-and-drop page layout feature is embedded in the tool to help users make adjustments to the laid-out pages.
Based on the above research, we have developed a prototype tool, implementing semi-automatic user interface prototyping. The architecture of our tool is shown in Table 1. Technically speaking, given the requirements text in Chinese, we first perform tokenization based on Trie tree structure, Viterbi algorithm, and HMM model4. Then we use FastText, which is fast and robust for small data-set, to process word vectors and embed them into neural networks.
Moreover, to complete the classification of diferent words and phrases on the layout, we manually label more than 2000 words and sentences. The tag types are mainly diferent web page layouts (e.g., peer-to-peer grid mode, heading area mode) and diferent template names. Meanwhile, to solve the ambiguity caused by user input errors, we use the Tencent NLP library to help us do text error correction and find synonyms. 5
For color extraction, we use the mainstream Median Cut Method [
This paper introduces SAIP, a prototype tool for Semi-Automatic user Interface Prototyping. It takes layout and color requirements as input in natural language and generates corresponding user interface prototypes, assisting in requirements elicitation.
Our tool is mainly aimed at prototype creators with a light technical background or designers
who need to prototype quickly. First of all, it follows the idea of rapid prototyping in software
engineering. A brief natural language input could automatically generate a preview interface
prototype for the users. This quickly and easily helps users to define their requirements,
eliminating the need for multiple iterations for further development. Secondly, each interface
prototype generated by the system can be adjusted and modified, and software engineers can
4https://github.com/fxsjy/jieba
5https://cloud.tencent.com/document/product/271
6https://github.com/STRML/react-grid-layout
iterate and improve. Thirdly, according to human-computer interaction theory[
In the future, our primary task is to use machine learning techniques to replace the manual labeling approach to more accurately obtain the layout and colors mapped by user needs. For example, refer to the VIPS algorithm [8] to extensively learn the layout and color scheme of websites, optimize the classification model, and recommend more reasonable and richer layouts for users. And derive the DOM tree of the prototype interface from helping further software engineers to develop it. Besides, we will improve our functions, and by exploring natural language processing techniques, we hope to meet more user needs, such as analyzing user needs for functions, to derive a set of interface groups with internal logical relationships, including jumps and links to diferent pages. Finally, we will improve the robustness and usability of our system to make it more mature.
For the empirical evaluation, our plan is as follows: First, we will improve our current tool’s stability and usability and make it available to any Internet user by using our server to launch our product. After that, we will publish a survey about our tool and collect feedback from users to extend our tool from a functional perspective. After several iterations, our ultimate goal is to make it commercially available and accessible to all prototype designers.
This work is partially supported by the National Natural Science of Foundation of China (No.61902010), Beijing Excellent Talent Funding-Youth Project (No.2018000020124G039), and the Project of Beijing Municipal Education Commission (No.KM202110005025). [8] D. Cai, S. Yu, J.-R. Wen, W.-Y. Ma, VIPS: a Vision-based Page Segmentation Algorithm, Technical Report MSR-TR-2003-79, 2003. URL: https://www.microsoft.com/en-us/research/ publication/vips-a-vision-based-page-segmentation-algorithm/.