Educational trends have evolved during the last decades, with the implementation of Information and Communication Technologies (ICT's) and the change of approaches in the teaching process, the need has arisen to create tools that can face the new challenges of teaching. Along with this process, the user experience (UX) must be considered in the development process of educational support tools, as a fundamental part of Human-Computer Interaction (HCI). Consequently, in the process of developing tools for this purpose, technologies, approaches or concepts that can result in a pleasant UX should be considered. The objective of this study was to evaluate the UX of a web tool, elaborated with the purpose of detecting learning styles and personality. The tool was developed under the principles of Responsive Web Design (RWD) and using the Material Design Guidelines (MDG). For the evaluation of UX, the User Experience Questionnaire (UEQ) was used. Subsequently, the data obtained were analyzed using a tool developed in MS Excel. The results showed that, in general terms, the UX is good, however, some metrics indicate that the results could vary if the number of participants in the study were increased.
The contexts and methodologies applied to teaching no longer have the same approaches as in years past. The classic approaches, based on classrooms and activities proposed by teachers, have been transformed. Now, they are heading towards teaching based on the learning of knowledge and skills. Taking into account the autonomous activity of the students[1]. Consequently, it is important to have the necessary instruments to detect how learning occurs autonomously. Some authors have made efforts researching, analyzing and developing different instruments that help with the task of thoroughly knowing the way in which a student acquires learning individually [2],[3],[4], [5].
The last decades have been of great importance for the expansion of the Internet, this expansion has made various areas show interest in migrating their information to the cloud, this change has generated the need to create methods of innovation, in the creation of graphical interfaces functional, to present different contents on different types of devices, in all their shapes and sizes [6].
In the area of education and psychology is no exception. Designing interfaces for a correct HumanComputer Interaction (HCI) is a fundamental process. The impact of Information and Communication Technologies (ICT's), on the development and construction of instruments for educational and psychological evaluation, has widely transformed the static and classic models of measurements through instruments made with pencil and paper, leading them to the possibility of applying the instruments in digital formats [7].
As mentioned by Pin
As a result of the problem in the poor adaptation of the UI to the different devices on the market,
international companies have made proposals that reduce the workload of web designers and developers, for
example, the design language proposed by Google
Human-Computer Interaction (HCI) is the discipline in charge of studying the main aspects surrounding the design, implementation and evaluation of computer systems with which the human being can interact [11]. The usability of an interface is the main area in the discipline of HCI, it focuses on the methods used for the evaluation and measurement of the ease with which a user uses and interacts with the interfaces of computer systems [12].
The evaluation of UX in web applications is a fundamental activity in HCI, which allows us to analyze
if an application really fulfills its purpose. There is a great variety of research works, in which the need
and benefits of implementing evaluation techniques to measure the performance of an application, site
or web system can be observed. For example, the work done by Isherwood
Considering the type of device, from where the content of a web application is viewed, can be a point of reference for the design of better UI’s that work on any screen resolution. An example is the use of User Centered Design (UCD) and MDG, to measure the effectiveness and efficiency of web content delivery, in different resolutions, through RWD [8].
In the area of psychology and education, we can find the evaluation of an e-learning platform, through the UX evaluation technique, using the UEQ instrument [15]. Another example is the evaluation of a web system used for thesis management, where techniques such as: heuristic evaluations and the application of UEQ were used [16]. One more work describes the analysis that was made to user reviews, through data mining, to detect strengths, weaknesses and usability gaps. This study was carried out on 106 mental health applications, publicly available in the App Store of the Apple company and Google Play of Android [17]. 3.
The web tool that was used is the product of a research in process, developed within the
Autonomous University of Zacatecas (UAZ), Mexico, whose purpose is to improve the instruments for the
detection of learning styles and personality, using, artificial intelligence (AI) techniques. In this
research, the Myers Briggs Type Indicator (MBTI) instrument was chosen [5]. Myers & Br
Due to the nature of the research project from which this study emerges, the participants were only students from the Autonomous University of Zacatecas (UAZ), Mexico, specifically from the Academic Unit of Electrical Engineering (UAIE). Being the target population, it was necessary to limit the selection of the sample to this specific group. The characteristics of the participants are shown in Table 2. 3.3
The purpose of this study is to measure the user experience of a product developed for a specific purpose, in this case, a web tool to measure learning styles and personality. To fulfill the purpose, a User Experience Questionnaire (UEQ) was implemented [23]. UEQ provides a series of items, each with two opposite meanings (negative and positive). The questionnaire has 6 scales, measured through 26 items. We can group them into pragmatic (goal-directed) quality and hedonic (non-goal-directed) quality [23]. UEQ, measures different aspects of the user experience, the items are presented on a scale from -3 to 3, where -3 represents the most negative response, 0 a neutral response and 3 a positive response [23]. A description of each scale is shown in Table 3.
Can user’s complete tasks effortlessly?
A. Application of User Experience Questionnaire
For the evaluation of UX, the UEQ [23] was applied. The procedure was carried out inside the Laboratory of Interactive Technologies and User Experience (LITUX), located within the Autonomous University of Zacatecas (UAZ), Mexico. The data obtained is a sample of 11 postgraduate students and one undergraduate student (n=12). To count on their participation, they were invited to collaborate and thus come to the laboratory. Within the laboratory, the process was as follows: In a controlled environment and without distractions, each of the participants was asked to complete the tasks of the web tool, subsequently and without any type of pressure or coercion, we asked them to, according to their experience in manipulating and completing the tasks of the web tool, they will answer the UEQ questionnaire. The study participants can be seen in Fig. 7. The procedure for the application of UEQ is detailed
The study was developed with the intention of evaluating UX, of a web tool, developed under the principles of RWD and the use of MDG. In this context, the study was carried out, using different devices. To evaluate in a desktop environment, a single laptop was used, the evaluation in mobile devices, was done in different models and brands, with different screen resolutions. A detailed description of the devices used is shown in Table 4.
GB RAM
2400 x 1080
px 1792 x 828
px 2400 x 1080
px 2960x 1440
px 2400×1080 px
For the interpretation of the data, the "UEQ Data Analysis Tool, Version 10 (DAT)" was used, which is free to use and available on the website (https://www.ueq-online.org/). The tool is a product developed in MS Excel, which was designed for the statistical analysis of UEQ items and scales. You can calculate the reliability of UEQ, with metrics such as Cronbach's Alpha. The Cronbach's Coefficient (Cronbach's Alpha) is a very popular method. It is widely used to measure the consistency of the scores obtained, in the application of a questionnaire. In this study, Cronbach's Alpha was considered to measure the consistency obtained in each UEQ scale. To consider solid consistency, Cronbach's Alpha must be > 0.7 [24]. It is also possible to obtain other statistical measures, for example, the statistical means or the standard deviation.
Another important metric that DAT allows us to obtain are the confidence intervals of each scale. The confidence interval measures the precision of the estimate of the value of the mean, for each of the scales. In this study, an interval of 5% is considered. The range of the confidence interval depends largely on the size of the sample obtained [23].
The tool provides a Benchmark for the evaluation of the answers obtained with UEQ. The Benchmark classifies a product through 5 categories by scale. The categories that are classified by each scale are: excellent; good; above average; below average and bad. To consider the quality of a product, it is necessary to compare it with historical data of other similar products. In this context, the Benchmark provides a set of historical data. In total, 452 products have been evaluated with UEQ and 20,190 participants in all evaluations [23]. For each classification there is a corresponding interpretation. Table 5shows the interpretation for each classification. In the range of 10% of the best results. 10% of the results in the reference set are better than the evaluated product and 75% are worse. 25% of the results in the reference set are better than the evaluated product and 50% are worse. 50% of the results in the reference set are better than the evaluated product and 25% are worse.
The results of the study were obtained thanks to the UEQ analysis tool. With the use of this tool developed in MS Excel, it was possible to obtain relevant data to check the reliability and consistency of the 6 UEQ scales. To measure reliability and consistency, in this study, the Cronbach's Coefficient (Cronbach's Alpha) was considered. Thanks to the reference data set (Benchmark), it was possible to compare the results obtained to measure the quality of the product, through the 6 UEQ scales. Thanks to the MS Excel tool, for data analysis, graphs were obtained, which allow a simpler interpretation. 4.1 Cronbach’s Alpha
The results obtained from the UEQ Data Analysis Tool (DAT) and according to the Cronbach's Alpha values, which indicate that, to consider a solid consistency, the Alpha value must be > 0.7, it is observed that almost all the scales measured in UEQ show a solid consistency. The scales of attraction; perspicuity; efficiency; dependability and stimulation, show a fairly high consistency. However, the novelty scale shows the lowest result. Table 6 indicates the values obtained for Cronbach's Alpha, in each UEQ scale.
For this study, the values obtained from the statistical mean of each of the 6 scales turned out to be outside the confidence interval. However, the results may indicate the need to modify the number of participants and probably narrow the range of the confidence interval and thus obtain a more precise estimate. The confidence intervals, the reliability of the estimate, the means and standard deviations, in each of the scales, are represented in Table 7.
For the interpretation of the results, the standard values are the following: for a neutral evaluation, the mean value for each scale must be a value between -0.8 y 0.8. Values >0.8 represent a positive evaluation, when an evaluation represents a negative user experience, the average values per category are <0.8. In general terms, it can be seen that the UX evaluation of the web tool obtained a positive value, except for the novelty scale, which, according to the value obtained in the statistical mean, the UX evaluation is neutral. Fig. 9 shows us a graphical interpretation of the statistical means and confidence intervals, for each scale.
Having a Benchmark, for the evaluation of UX, is necessary to be able to compare the results obtained when applying UEQ. This task was carried out, thanks to the MS Excel analysis tool with which we are working. The tool has a data set that classifies the UEQ scales in 5 categories; the categories that it evaluates are: excellent; good; above average; below average and bad. When processing the information with the UEQ Data Analysis Tool (DAT), in general terms, it can be seen that the web tool obtained a positive evaluation. For the classification of each of the scales, it is observed that, in the Attractiveness and Perspicuity scale, the product is good. Efficiency is rated as excellent; Dependability and Stimulation are above average. However, on the Novelty scale, we can see that it was classified below average. Fig.10 shows the results of each category, for each scale. The comparison of the data obtained in this study, directly with the Benchmark, of the analysis tool, together with the interpretation of each result, can be found in Table 8. 10% of the results in the reference set are better than the evaluated prod-uct and 75% are worse. 10% of the results in the reference set are better than the evaluated product and75% are worse.
In the range of the top 10% re-sults. 25% of the results in the reference set are better than the evaluatedproduct and 50% are worse. 25% of the results in the reference set are better than the evaluated product and 50% are worse. 50% of the results in the reference set are better than the evaluated product and 25% are worse.
This paper aims to evaluate the user experience (User Experience, UX) in the web tool, which was developed for the detection of learning styles and personality. The selection of the sample was limited only to the students of the Academic Unit of Electrical Engineering of the Autonomous University of Zacatecas, Mexico. This limitation influenced the number of participants in the UX evaluation, through the User Experience Questionnaire (UEQ). The results to measure the reliability of the answers obtained in UEQ, indicated that many of the values of the statistical means, obtained in each of the 6 scales, did not belong to the confidence interval, however, increasing the number of participants in the study, could narrow the confidence intervals and obtain greater precision. On the other hand, the data obtained, to measure the consistency, using the Cronbach's Coefficient (Cronbach's Alpha), indicate that the consistency of each scale is solid, being that, in all the scales, the value obtained is > 0.7. The data obtained, with Cronbach's Alpha, give an encouraging result.
The evaluation of the 6 scales, for the classification of each of them, in the 5 categories, which through the data set (Benchmark), available in the tool to analyze the UEQ results, shows that in general, the evaluated product, in compared with the historical data of other evaluations, it has a good general performance, however, according to the classification obtained, it is necessary to propose solutions, to improve the scale of innovation.
To conclude, it is recommended that, in future work, the sample of participants be larger and another academic profile be included, also that the number of devices be increased, where the web tool is evaluated to obtain better results. Finally, it is proposed to use the data obtained in this study to consider increasing the performance of the evaluated scales, where the result shows a low performance, for example, in the innovation scale.
Thanks to CONACYT, for the scholarship granted. [9] [10]