The article contrasts manual and automated identification of elements in images of web user interfaces (UIs), which is essential for machine learning (ML) models that describe user behavior. We consider the principal advantages and disadvantages of the two methods and compare linear regression models. The constructed ML models describe users' subjective perception of web UIs in such dimensions as complexity, aesthetics and ordering. Somehow unexpectedly, the resulting R2s of models built with certain factors obtained from automated labeling turned out to be slightly higher. Particularly, shares of text and images in the web UI, as well as the sizes of the elements, were rather influential. We believe that the main disadvantage of the manual labeling is the human factor, as mistakes made by the labelers and diversity of their outcome affect the quality of the models. In turn, the automated process has a number of drawbacks that must be taken into account and that we discuss in the paper. The results of our work might be of interest to both ML researchers and to usability engineers who seek to improve the subjective satisfaction of users with websites.
Any design object needs effective presentation, in which structuring of textual and visual
information is highly important [
The identification of UI elements in a website page screenshot for further assessment of visual
complexity can be obtained through either manual labeling or automated recognition process [
Thus, the purpose of the current work is to determine the types of elements that affect the subjective perception of websites, as well as to compare the models built with the factors’ values obtained via automated vs. manual labeling of web UI screenshots.
In the experiment, the subjects were offered about 500 website interface screenshots and asked to label UI elements in them: highlight the element in a box and identify the elements’type. They were using a dedicated software tool, LabelImg (see in Fig. 1). In total, 11 human labelers took part in this activity, after providing informed consent. 2.2.
In addition to the manual labeling of the screenshots, we also performed their automated analysis,
using our dedicated Visual Analyzer (VA) software tool, available at http://va.wuikb.info/ and
described in detail in [
In Table 2 we show the types of UI elements that had been identified in manual and automated
labeling that we performed. As one can seen from it, the main difference between the two methods
lies in determining the types of elements. In the manual labeling, the participants were able to rather
successfully identify 26 different UI types, while for the automated labeling, there were 8 types,
resulting from the pre-trained ML models. In addition, due to high visual diversity of today’s
designs, the accuracy of the automated type detection was the most problematic dimension in the
semantic-spatial analysis that our VA tool performed (see [
From the labeling datasets, the heights and widths of the elements were calculated (also by the elements’ types: text paragraphs, buttons, images, panel,sa,ndettch.)ere was also the data on the height and width of the screenshot. Based on this information, areas of each UI he element was calculated, as well as the share occupied by this element in the total screenshot space (by the elements’ types: texts, images, background, etc.).
Thus, there were 2 groups of factors in the behavior models that we further constructed for complexity, aesthetics and orderliness that were the dependent variables in the study: 1) the number of elements: the number of elements of a certain type located in one screenshot; 2) the proportions: the shares of the total area of the elements’ typetso the total area of the screenshot.
web
For each screenshot, we also had evaluations of complexity, orderliness and aesthetics, provided
by another 137 participants (67 female, 70 male). The majority of them were Russians (89.1%), while
the rest were from Bulgaria, Germany, South Africa, etc. More details on the participants and the
procedure can be found in one of our previous works [
Each of the subjective perception dimensions was assessed on a scale from 1 to 10. Then the average values for each indicator were found using formulas (1)-(3), where i is the number of web pages, nj is the number of participants who provided the evaluations for the i-th website.
∑ (1) where – indicator of the aesthetics of the interface of the i-th web page, the aesthetics of the interface of the i-th web page
∑ where – indicator of the complexity of the interface of the i-th web page, of the complexity of the interface of the i-th web page ∑ – estimation of – estimation (2) (3) where – indicator of the ordering of the interface of the i-th web page, – estimation of the ordering of the interface of the i-th web page.
Complexity is the number of elements on the screen and their arrangement. Orderliness is an
ordered data structure of the web interface that allows the user to easily find the information they
need. Aesthetics is an assessment of the attractiveness of a product by the user. This indicator is
important because the aesthetics of any web interface has a strong impact on the user, even when he
tries to evaluate the functionality of the system [
Using the SPSS statistical analysis software, we build linear regression user behavior models (as representing the most universal ML method) with the factors resulting from the manual and the automated labeling. The dependent variables in the models were the three subjective perception evaluation scores, while the independent variables were the factors resulting from the two labeling processes. The step-by-step method of the regression analysis allowed stepwise inclusion of the factors in the models, thereby discarding those that did not make a significant contribution to explaining the dependent variables.
The results of the regression analysis are presented in Tables 3-5, each corresponding to a different subjective perception dimension. The null hypothesis H0 was as usual in the regression analysis, that the regression equation is not significant. For each of the dependent variable, the models turned out to be significant (p < 0.05), while the significances for the factors selected by the step-by-step method are shown in the respective columns of the tables.
As one can note from Table 3, the perceived complexity of websites was most influenced by the amount of text, the number of radio buttons, the number of buttons and tabs. The subjective orderliness (Table 4) was most influenced by the proportion of radio buttons and buttons. The perception of aesthetics (Table 5) was most influenced by the proportion and amount of text on the page, the number of buttons, labels and images. t
Despite the fact that the constructed models had rather low determination coefficients (R2), it is still possible to draw a general conclusion about which elements affect the assessment of website perception among users, and whether the degree of this dependence is influenced by the labeling method (Table 6).
As one can see from Table 6, the share of text on the page was significant for all the three subjective perception dimensions. Moreover, the presence of the text had the greatest impact on the assessment of complexity and aesthetics. At the same time, the obtained values for the automated and the manual labeling did not differ much (4-10%), but with the automated labeling, the models’ quality indexes, as represented by R2s, were somehow superior.
The study showed that the subjective assessment of the website perception is influenced by the amount of text on the page and the share of images: the more text on the page, the more complex and less aesthetic the website appears. However, this statement is not entirely correct, since in addition to
R2 (Automatic method) 0.071 the share of the text area, the text style (font, size, etc.), and the presence of pictures that dilute the text, and many other factors are also important. The number of images, their size, quality and position on the web page affect the overall website subjective perception, including the assessment of the aesthetics and orderliness. As a general rule, these indicators are interchangeable: the lack of text is often compensated by a variety of graphic elements and images. Therefore, in order for the web UI to comply with usability standards and to be simpler and more understandable for its users, it is necessary not to clutter the interface with a large number of elements and break long texts into smaller parts. At the same time, the number of element types did not significantly affect any of the subjective perception dimensions.
The quality of the user behavior models built on the factors resulting from the automated labeling
was slightly higher than for the manual one, which suggests feasibility of our UI visual analysis tool.
In general, this may indicate that the automation of the labeling process makes sense, but it requires
high costs and the presence of certain knowledge to implement it. After all, our VA software was
initially trained with the data once provided by human labelers [
It should be noted that the constructed linear models have low R2 coefficients, so they should not be used in production. The goal of our current study was merely to compare the two labeling methods, while for real user behavior modeling, more advanced ML methods and architectures should be used.
Our plans for further research include investigation of the effects of web page layouts (the size of the element, its type and occupied area) on the subjective assessment of the perception of the site, but also the colors, fonts, types of buttons, animations, etc.
The reported study was funded by RFBR according to the research project No. 19-29-01017.