Usability studies in the web domain are based on various metrics, but the question is how to apply these metrics to evaluate a larger group of people. When we consider that every user has different qualities, skills and experiences, we could expect that the results of testing of same scenarios will be different. We aim our research to show that quantitative studies can provide more accurate results if we work with information about personal characteristics of participants. We have already conducted a preliminary controlled experiment on a small sample of participants, which explores influence of a web literacy.
UX studies help to focus the specific problems, e.g. understanding users in order to improve design or functionality. Some UX studies require the involvement of a large sample of participants, therefore to conduct a quantitative experiment. Unlike from small qualitative studies, we don’t have an opportunity to communicate with participants about the various details, for example to identify their individual characteristics. Usually it is necessary to use a large sample of participants, also in some cases it could be convenient to have a diversified group. Results could be affected by several impacts, some of which have already been identified and are described in next section.
A presence of information technologies in almost every area of
our life requires effort of users to adapt to different environments.
Ability to work with such an environment depends on several
factors, including: the duration of interactions, provided
functionality and also on information architecture. It also relates
to user’s behaviour. This is very individual and differs according
to experiences, knowledge, goals, location and social contexts [
Almost every research article shows areas in which were found relations among data, or shows what attributes to focus when measuring or evaluating for improve results. We consider it selfexplanatory that we want to expand the study on the reliable tests to obtain additional information such as a web literacy level. Effective carrying tasks in the digital space of Web, such as search for or interference of information, learning, communication or use of services significantly affect quality of life. Our attention is directed to the studies that evaluate user interaction with a focus on individual differences of participants in quantitative studies. While the usability, as part of the human-computer interaction is handled for over twenty years, research at present does not answer all of the questions.
Collecting data from implicit or explicit feedback and further
derivation of various properties and characteristics can be greatly
refined using modern equipment. Existing solutions are based on
monitoring of user behavior using peripherals such as mouse and
keyboard [
With the rapid growth of the Web, we could expect the introduction of the concept of web literacy. At the beginning of the millennium there have been established several tools for testing, but they are no longer supported today.
There are several approaches to the term of web literacy. One of
the presented view divides web literacy into three sub-categories:
Web searching skills, Web reading skills and Web evaluating
skills [
Despite the expectations, we did not find a proven and established web literacy test. Existing tests for Information, Computer and even Internet literacy may contain indications of the issues that we would expect in a web literacy.
Individuality, as a set of characteristics in a given establishment,
distinguishing two people from each other, is considered as one of
the important factors with great impact on the results of the
studies [
In a study with test of the university website, we take into account
the profession, the daily number of hours spent on the Internet or
weekly computer usage [
In recent studies, the authors focus on the individuality in domain
knowledge. Another work shows the results, where social
networks could be identified, for example that people open to new
experiences are using virtual communities and Facebook is used
more by extroverts. Author divided individual differences to
physical (sex, age, position, experience) and psychological (based
on the model of the five aspects of personality: extroversion,
agreeableness, openness to new experiences, conscientiousness
and neuroticism) [
Testing gender differences in the interaction with the computer is
relatively easy to make, and used in several studies. A study based
on physical differences showed that the difference in perception
of quality and service or information quality is negligible between
the sexes [
One of the latest research was done in national survey of teens
[
Many studies nowadays analyze Information and Communication
Technology (ICT) Literacy, such as the research about validation
of a student tool for Technology Literacy [
Evaluation of the UX study results of the random groups can be inaccurate or misleading if we do not know the basic characteristics of participants. Interpretation of results also may be insufficient without more knowledge of respondents which we can encounter for example in election polls. Random samples could be sufficient when we do overview studies or surveys, however, when we do evaluation based on the study, we should know what is actually a sample of participants that has taken part in testing. Research objectives in the context of the thesis is very ample. We want to discover and verify the characteristics that divide users by some aspects, for example, those which will be reflected in the speed of problem solving, response correctness and that like. A current specific objective is to assess the web literacy from the view of the impact on user engagement. Web literacy as a subset of Human Computer Interaction is just one of the many needs when participating on today's user studies. Therefore, our specific objective is to determine whether the Web Literacy affects the study. Next, determine in which type of tasks it makes sense to deal with web literacy and finally propose a set of tests, which are able to properly measure it. This is closely tied with the ability to choose the right decision tests for further automatization. It also opens the topic of searching for connection between the use of the web and web literacy.
Besides the internet literacy factor, we would like to find also other significant impacts with usage of eyetracking devices in user experience studies. Not quite arising, but also related to the topic is investigating the impact of web features on the user behavior.
Our vision is to propose an evaluation method suitable for usability studies. As we described in previous sections, our matter is everywhere present individuality. When we consider that every user has different qualities, skills and experiences, we could expect that the results of testing of same scenarios will be different. We mainly want to find out the impact of web literacy and to which extent it could affects experimental results. Currently proposed method consists of a test, which could help to determine web literacy and as a subset also partially detect computer literacy. We are also interested in the relations between different parts of our test. The future way of evaluation will lead to machine learning methods. In next step we plan to prepare a mechanism for event logging, which will collect data from interaction in web environment.
Our main contribution consists of cleaning the data and adjusting results on the basis of personal characteristics, in this case a web literacy. Another benefit lies in defining the user characteristic impacts which enters the evaluation of user studies.
Accurate and extended information can help as for evaluating, as for example, when creating own list of relevant user study participants. Purpose of our approach is to focus on participants and their evaluation. Regular user studies would obviously have an impact on increasing web literacy of participants from the list and periodical verification would be essential. However, this approach does not fall under the primary research, so we may consider it in a future work.
The scope and testing options depend on the scenario of the study itself. So if we use tests on the basis of additional tasks, time requirements which can result in fatigue could be considered as limiting. In this case it will be necessary to assess the benefits and success ratio of our solutions with the current self-reporting tests.
Basic idea of the method is comparing participants with greater literacy and participants with lower web literacy. A prerequisite is to obtain information from the user's behaviour on the Web. We believe that we can discover suitable tasks to estimate web literacy of participants. User experience is verified through the short test, consisting of three parts.
In the first part the user is asked to identify the various areas on a website without specific content. We derived a set of patterns associated with tasks from real websites. It examines web literacy linked with the observation position of the various items, such as "shopping cart" or "search input". In this test, we are able to compare the response of the user and the primal content, which can be used later for designing and creating user-friendly web pages, thanks to the transparency, simplicity, and content placement following the established standards.
The second part examines web literacy by selecting the proper icon in few iterations, such as icons "send email", “refresh” or “show menu”, The user indicates the knowledge he has gained from working with web sites and web applications where he has the opportunity to visually and subconsciously perceive these pictograms typical for web interface.
In the third part, we examined user knowledge through a questionnaire with provided options and open questions. Questionnaire consist of questions about practical usage of Internet, mainly the Web.
We assume that quantitative studies will provide more accurate results with information enriched with personal characteristics. In order to discover the different influences, we want to conduct the experiment on a larger sample of participants. Based on the detected relations, we try to adjust the results of studies. We have proposed an outlines for method of estimating web literacy based on the tracking of user behaviour in a web environment. This paper is dedicated to researching web literacy of ordinary users on the internet, especially on websites. Research is conducted by tests and questionnaire, to more explicitly gather information about the experience and knowledge of website users. At first we wanted to monitor user’s behaviour while searching for the right placing of element in a website layout. This idea was transformed into searching tasks in standard layout, with random texts for making the layout a little bit more abstract. There was assumption that we would get only very short interactions from such a scenario. With content and preserved graphic pattern present, it still appeared very unchallenging, therefore without the need for further thinking.
The duration of the experiment is unlikely to be significant because it can be affected by several factors such as the speed of comprehension, reading speed, and many others.
A pilot experiment was conducted in UX Lab on FIIT STU, eye
tracking device Tobii TX300 was used for data collection. User
sessions are recorded [
Web literacy is estimated by using tests. The questions are not supposed to be answered in a form of self-reporting, but rather by answering the identified questions (contains common and wellknown facts and situations on the web). We want to find basic patterns in searching for areas on the website where we take two groups of participants as a basis - experienced and less experienced. Next, we plan to compare the differences in determining the position of an element, as we further described. We mainly focus on the variance in the response.
Online experiment was conducted on a participant sample (N=32, mean age = 27.34, standard deviation = 12.71). Results of the second part showed that the selected icons are well known or obvious, thus they were marked correctly almost without exception by all participants. In the third test, we identified four questions without importance, because all participants knew the correct answer. It is not clear yet, whether there is a correlation between first and third part of the test.
We have decided to use various website patterns, mostly from frequently visited but not necessarily popular websites. Primal websites were labelled as “electronics brand”, “movie database”, “main page of e-shop”, “main page of auction portal”, “faculty website”, “article from online news” and a “list of flight tickets”. Firstly, we have edited the screen captures of random websites, so the content is not clear, but the pattern structure is maintained by grey regions. Main contours were also preserved. Then experimental environment adds a thin grid of small cells (20px) which helps to guide about clicking among the regions. We were also considering a version of content hiding by blurring all of the content within region borders, but this seems to be very distractive or in some cases partially helpful (i.e. the participant could recognize a specific website).
Every user session includes three steps of examination. Not to distract or to give participant a clue, we have decided to use order of steps as follows: First step should examine the user skill in web environment by asking about placing well known objects like shopping carts, links, searching bars etc. Experimental environment for second step was prepared as a set of 16 wireframe schemes associated with tasks based on previously mentioned websites as on the Figure 1. Instructions are given to the participant before every view of stimuli and then he started to search for the spot to select. Second was the test with icons. We have chosen 15 characteristic web icons (closely related to browser environment or basic web tasks). Test environment was created as a dynamic website. This step was undertaken as a support information beside the third step, which consists of a questionnaire.
Third part was a web literacy questionnaire, containing 14 questions (providing 3 or 4 options for each question or open question) about simple search engine and query tasks, browser tasks and browsing. Questionnaire was set up in Google Forms tool as an online accessible and standalone part.
Order of questions in first and second part was randomly generated for each participant.
We plan to involve more participants for greater statistical sample. By including great sample, we expect to gather larger dataset from interaction and also greater variability of responses. According to complete results we will try to apply our method to user experience studies.
Still, we knew that after the initial tests the work on quantitative studies will require a further examination using extensive data. In next step we are going to prepare a well-known web environment for event logging. Therefore, further examination with the machine learning will take a part.
This work was supported by grants No. VG 1/0646/15, APVV-150508 and it was created within the Research and Development Operational Programme for the project University Science Park of STU Bratislava”, ITMS 26240220084, co-funded by the ERDF. testing. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 1347-1350). ACM.