The research aims to form a new method for evaluating interfaces, ensuring its multi-criteria nature and eliminating the shortcomings of previous methods. A combination of expert and heuristic approach is proposed, to detect a wide range of UI/UX problems, to ensure assessment competence and to reduce the level of distrust of the expert. In the first experiment, two groups of interfaces with different characteristics were evaluated, with two interfaces in each group. Fifteen heuristics were evaluated: ten general purpose criteria and five specialized criteria. Thirteen experts were involved, for whom weighting coefficients were previously calculated, taking into account their professional competencies and personal qualities influencing the reasonableness of the evaluation. After analyzing the results of the first experiment, it was decided to investigate the influence of the number of experts in the sample on the overall UI score. Therefore, for the second experiment, the optimal number of experts in the group was calculated to ensure the lowest score variance. Applications were evaluated in five groups (the number of heuristics did not change).
UI, UX expert evaluation, heuristic evaluation, evaluation methods, user interface, expert weighting,
In a highly competitive environment, companies are forced to invest huge sums in the development of advertising and information support for business - sites and applications are becoming a necessary component to ensure the success of the enterprise, and thus make a profit.
According to the statistics [
development
80-360 hours 80-400 hours 120-600 hours 120-600 hours 175-760 hours
rub./hour
Maximum price
rub./hour 1500 1200 1800 3400 3200 3800 6000 16400
2021 Copyright for this paper by its authors.
Note that a significant portion of the cost is spent on design and user interaction strategy. This stage also involves evaluating the interface, which can significantly reduce costs by reducing the number of edits and, as a consequence, iterations of redesign.
Based on the foregoing, the goal of the study was determined: the development and testing of a new method for assessing the interface, combining a qualitative and quantitative component.
To do this, it is necessary to perform the following tasks: Analysis of existing methods for assessing interfaces; Development of an evaluation algorithm with the following properties: flexibility based on the functional features, complexity and / or scope of the interface; speed and ease of use; potential for formalization; the possibility of reducing or completely eliminating subjective perception; Selection of the mathematical apparatus; Approbation of the method on various interfaces; An overview of potential opportunities for formalization; Development of recommendations for improving the method.
It is assumed that as a result of using the new method, the customer will be able to obtain both an overall assessment of the interface and individual criteria, which helps to determine the elements that need to be modified in the first place. Additionally, it is possible to develop recommendations based on expert opinion to improve the project.
In a previous study [
At the first stage described in the previous part of the experiment [
The second stage of the experiment included the direct evaluation of UI. As prototypes were used works of 4th year students of NSTU n.a. R.E. Alekseev, studying on 09.03.02 "Information systems and technologies" major in "Information technologies in design" within the study of "Mobile application development" discipline.
In the first experiment, each expert was asked to evaluate 4 interfaces grouped in pairs: Group A browsing and maintaining (creating) content, Group B - training applications and simulators (Figure
A set of heuristics, among which there were 10 general and 5 highly specialized questions, provides a quick experiment and allows us to determine the applicability of the method to mobile interfaces.
The level of consistency of the interface color palette with the target audience (TA);
The heuristics also included questions for a specific application category, such as Group A (viewing
10. The convenience of the search procedure. and maintaining content):
Level of personal satisfaction with the color palette of the interface.
1.
2.
3.
4.
5.
6.
7.
8.
9.
1.
2.
3.
4.
5.
1.
2.
3.
4.
[
Then we calculated the total score by assigning points to a single criterion according to the formula
the range from 0 to 1) score of interface compliance with the allocated criterion from 0 to 10,
is the weight coefficient of the expert, calculated in the first phase of the experiment [
∑ =∑ =1 ∙ , = ̅1̅̅,̅̅̅, (1) and its compliance with the principles of usability.
The resulting score ∙ 100% characterizes the average value of user satisfaction with this criterion If we consider the results of the evaluations of each of the experts as realizations of some random variable, we can apply the methods of mathematical statistics to them. The average value of the estimate for the i-th criterion where n is the number of experts.
The average value ri expresses the collective opinion of the group of experts. The degree of consistency of the experts' opinions is characterized by the value
i reflects the degree of influence of the evaluation of the i-th criterion on the overall assessment of the interface, calculated by the formula: 2).
Thus, the overall degree of satisfaction with the interface in percentage terms is defined as The screenshot of a fragment of the calculation and evaluation table in Excel is as follows (Figure ri j1
L r
ji n r ji i , r n 2 = 1 ∑( − )2, = ∑ ∙ called the variance of the estimates. The smaller the value of the variance, the more confident you can rely on the found values of the ri estimate of the importance of a particular criterion. As a measure of reliability of the cited expertise, we take called variation. The average value of the estimate is used to ri determine the weighting coefficients (2) (3) (4) (5) (6)
For clarity, the normalized average score for each criterion is formatted using color scales. This allows you to see the most (bright green) and the least (red) developed aspect of the interface.
For example, the following results were obtained for the examined interfaces (Figure 3, Figure 4):
Total score Minimum value Maximum value
For the second experiment, it was decided to change the number of experts in the sample.
It is proved that the number of experts must be large enough [
To calculate the number of the group of experts, we used the ratio that is used in calculating the error
of observations [
= 2/2, (7) where N is the number of experts in the group, εl = ε /S – maximum permissible relative error of expert estimation,
tp – is the Student coefficient, which determines the width of the confidence interval and the dependence on the value of the probability estimate P (tp is a tabulated value).
Depending on the given error of expert evaluation and the chosen probability value, the minimum possible number of experts in the group N can be determined (Table 3). εl Probability of estimation P
0,99 0,95 0,90 0,85 0,80 0,75 0,70 0,65 0,5 26 15 11 8 7 5 4 4 0,3 74 43 31 23 19 15 12 10 Empirically, it was found that experts of 13-15 people can be considered a sufficiently representative group to conduct the examination.
This is confirmed by the dependence of the accuracy and reliability of the results of the estimation of the date of occurrence of the event on the number of experts in the group N (Figure 5).
Number of experts N 1
Thus, it was concluded that the optimal solution would be to organize an expert group of 10-12 people. 4. Determination of expert weights that deviate from the main range of sample values
Interquartile range Q3 - Q1 = 0.1125 Determine internal limits 0.3325 + 0.1125 × 1.5 = 0.50125; 0.22 - 0.1125 × 1.5 = 0.05125 In our case, none of the calculated values of the weights exceeds the internal limits. In the case of such a situation, it is necessary to determine whether the number out of the range is a significant outlier.
To do this, determine the outer limits of the data set 0.3325 + 0.1125 × 3 = 0.67; 0.22 - 0.1125 × 3 = -0.1175
The determination of whether an outlier should be excluded from the data set must be based on a set of reasons. An outlier may not necessarily be a measurement error (and should be excluded), but may be related to new information or a trend and should be accounted for in the calculations.
It is also important to assess the degree of influence of the outliers on the median of the data set (its distortion), if the deviation of the median is not significant, then the outlier can be included in the data sample.
To confirm the hypothesis that the evaluation will be performed with greater accuracy and a smaller number of outliers, it was decided to conduct a second experiment with a smaller (11 people) number of experts.
Interquartile range Q3 - Q1 = 0.135
Determine internal boundaries 0.385 + 0.135 × 1.5 = 0.5875; 0.25 - 0.135 × 1.5 = 0.0475 Thus, in our case, none of the calculated weights exceeds the internal limits.
Let's calculate the outer bounds of the data set to determine the weighting thresholds 0.385 + 0.135 × 3 = 0.79; 0.25 - 0.135 × 3 = -0.155
After forming a sample of experts and calculating weighting coefficients (Table 4), it was proposed to evaluate 5 groups of interfaces. The results of the evaluation are presented in Figure 6-Figure 10: 6. Comparative analysis of the developed method with previously studied methods
Let's consider the most well-known methods for assessing interfaces and their applicability (Table 5)
formalize
evaluation
subjectivity of evaluation
people to evaluate
11
sample of experts includes ordinary users)
final iterations)
7-9
sample of experts includes ordinary users)
final iterations)
No
1 No
ease of experiment)
Thus, the developed method in the aggregate is more universal (in terms of the number of considered parameters), easy to implement and formalize (due to the simplicity and clarity of the mathematical apparatus).
Further development of the method presupposes its formalization on the basis of a web application and the creation of a system for developing recommendations for improving the analyzed interfaces. To date, a simulated layout of the service has been implemented using Google-services (https://sites.google.com/view/evalui).
The following patterns were revealed as a result of the experiment: The overall score is higher when there is greater consistency among the experts, i.e., the lowest variance of the estimates The overall score is higher with a smaller degree of difference in the weight coefficients of the experts in the group When the number of experts decreased from 14 to 11, the quality of the expertise increased (the experts' evaluations differed less numerically) The overall heuristic score does not correlate with individual subjective preferences Thus, this evaluation algorithm allows the maximum leveling of distrust of the expert due to the elaborate system of ranking of experts, and the formation of a general assessment of the interface is performed taking into account the degree of importance of this criterion in the overall grading system.
The results of the experiments allow us to draw conclusions about the applicability of the developed method for the evaluation of interfaces. The chosen mathematical apparatus is suitable for calculating the computational characteristics of the expert weights and the evaluation itself. In the future it is necessary to develop heuristics for different categories, also more detailed elaboration of the expert evaluation criteria for more accurate determination of the expert weights is possible.
URL: [14] How to calculate outliers, URL: https://ru.wikihow.com/%D0%B2%D1%8B%D1%87%D0%B8%D1%81%D0%BB%D0%B8% D1%82%D1%8C-%D0%B2%D1%8B%D0%B1%D1%80%D0%BE%D1%81%D1%8B.html