Computer Literacy is an important predictor for how pro cient a person is in its interaction with computers, which can determine whether a person is motivated and able to use speci c software. Measuring Computer Literacy or its constituent elements (Skills, Attitude, Knowledge and Experience) has traditionally been done using questionnaires. This method has several limitations: it is e ort-intensive for subjects, subject to cognitive biases, and constitutes only a snapshot of a person's actual Computer Literacy. This limits the usefulness of Computer Literacy as a factor in persuasive systems design. In this paper, we describe an experiment to test the design of a system that extracts elements of Computer Literacy based on observation of human-computer interaction. This new method has the potential to enable the use of Computer Literacy in software design by addressing some of the barriers to its use "in the wild", opening up new possibilities for tailored and adaptive systems.
Digital technologies have become pervasive in our society. We now use computers for work, entertainment, administration and a host of other applications in our daily lives. To do so, we need to understand how to interact with computers: how to open documents, nd information on the world wide web, look up directions, and so on. In short, the ability to use software has become a vital skill in our daily lives. As the use of software has moved from an expert skill to a basic one, the spectrum of user types has become wider and more diverse. Persuasive technologies often depend on understanding these user types that they are trying to reach. One of the principles of persuasive systems design listed by OinasKukkonen and Harjumaa[11] is tailoring: the ability to adapt a system to the user's characteristics and preferences.
Many methods exist to gather information for such tailoring (e.g., interviews, surveys, observational studies) on the level of groups of users. On an individual level, systems that display some form of awareness of a user's preferences, such as recommender systems, have become common. But context awareness in software design is a multi-dimensional construct [6], and while some elements of context awareness are now commonplace in software systems, others remain hard to implement. One of these elements in particular is the right way to communicate with users. The way users are approached has an e ect on how the message is received. The Elaboration Likelihood Model (ELM) [12] of Petty and Cacioppo postulates that "Although people want to hold correct attitudes, the amount and nature of issue-relevant elaboration in which people are willing or able to engage or evaluate a message vary with individual and situational factors". When considering the context of using a digital system to change behaviors or attitudes, Digital skills is one such skill-based variable that can be taken into account to personalize a message. For example, in a digital security application such as a rewall, an interface that provides more control and information would be desirable for highly skilled computer users that are capable and willing to engage in a higher level of elaboration, whereas novice users would be overwhelmed. This makes understanding a user's pro ciency with computers crucial to designing the right interface to interact with them. To create a system that adapts to individual users' computer pro ciency, however, would require the system itself to know or measure such pro ciency.
Bawden performed a review of Digital Literacy and related concepts in 2001 [1], examining the relationship between several closely related concepts such as Digital Literacy, Information Literacy and Computer Literacy. We focus our attention on Computer Literacy, which is a concept that dates back to 1972 [2], but remains relevant today. This work builds on a recently completed systematic literature survey [13] which has re ned the concept based on 190 existing articles. The article recognizes four main dimensions within Computer Literacy: Computer Experience, Computer Knowledge, Computer Skills and Computer Attitude. All of these a ect the pro ciency with (and adoption of) software in some way. Computer Attitude a ects computer interaction on multiple levels. Venkatesh [18] indicated a relationship between several elements of Computer Attitude (e.g., Computer Self-E cacy, Computer Anxiety and Computer Interest) and usage acceptance and user behavior as seen through the Technology Acceptance Model [4]. This is a nding that has been rea rmed by other research in the eld [17]. Similarly, Computer Experience and Computer Knowledge have also been found to be a signi cant predictor of technology adoption [7], [16].
Traditionally, Computer Literacy is measured using questionnaires. But questionnaires su er from several problems when used in software. First, they constitute a snapshot of a user's Computer Literacy (attitude, skills or otherwise). Persons' attitudes towards computers can change over time, while experience, skills and knowledge are sure to grow along with their computer use. Second, questionnaires, especially those that deal with a user's perceived skill are subject to perception biases such as the Dunning-Kruger e ect [9], limiting their validity. Finally, questionnaires require active participation of users, which is not desirable because it requires extensive interaction with subjects, presenting a barrier to adoption.
To mitigate these three issues, we propose to design software that can gauge a user's Computer Literacy in real time. While there are speci c tests for Computer Skills such as the European Computer Driving Licence (ECDL) [3], these still require extensive user interaction and su er from the same snapshot issues mentioned previously. To date, no research exists into automated assessments of Computer Literacy. In this paper, we describe an exploratory experiment that was designed explore a way to ll this gap, nding elements of human-computer interaction that can be used as indicators of Computer Literacy or its constituent parts (Attitude, Knowledge, Skills and Experience). In this experiment, participants installed a software tool that let us monitor several aspects of their interaction with the computer. Additionally, these participants lled in the INCOBI-R, an existing validated Computer Literacy questionnaire, to assess their computer knowledge, skills and attitude. Additional questions were added to identify their Computer Experience. The resulting data (software log les and questionnaires) were subjected to a correlation analysis, identifying several possible relations between the questionnaire results and the software logs. This research aims to answer the following research questions: RQ1: What elements of Computer Literacy measured by questionnaires can be observed by logging human-computer interactions? RQ2: Which of these logged computer interactions give the best indication of a person's Computer Literacy? Based on the outcome of our exploratory experiment, we discuss possible implications for the real-time measurement of Computer Literacy and future experiments to be done to corroborate our preliminary results. The rest of this paper is structured as follows: First, we describe the experimental design and encountered issues during implementation. Then, we present the results of the experiment. Finally, we discuss the answers to the research questions above, limitations, implications and future work. 2
The experiment in this paper was designed as an exploratory study to nd computer interaction patterns that might be correlated with dimensions of Computer Literacy. In the experiment, participants were asked to both ll in the INCOBI-R Computer Literacy questionnaire, and install a software tool to collect human-computer interaction data. This interaction data was compared to the questionnaire results. The experiment was performed using ten participants. Participants ranged widely in age (21-63), context (work, entertainment etc.) and skill level (as measured by the INCOBI-R). The experiment was planned to run over the course of a minimum of 2 weeks in order to collect su cient data. Data was collected on private computers in a home setting of the participants, although most participants indicated they also used the computer for work/study purposes. Participants were asked to ll in the questionnaire at the beginning of the experiment. 2.1
To determine which elements of computer interaction could yield meaningful data on a user's Computer Literacy, we examined its constituent
Average time the computer was on per day Average time the computer was in use per day Time the computer spends active vs. idle parts, as de ned in the previously mentioned literature survey [13]. For each of the elements of Computer Literacy, we discussed software metrics that might feasibly be implemented in the tool by the author and the software engineer employed for this project. We constrained ourselves to computers using the Windows operating system due to three factors: a) the di culty in nding a wide range of test subjects for other operating systems, b) the amount of available programming expertise, and c) the known monitoring options available in Windows computers through Python libraries interfacing with the Win32 API. A nal list of what was measured can be found in Table 1. The link between these variables and the elements of Computer Literacy survey can be found in Figure 1.
To log useful data of the participants, observation in a real-life environment for a prolonged period was vital. To do so, a software application was developed for the Windows operating system (see Figure 2) consisting of three parts: a) a logging module consisting of several Python scripts using the Win32 API to log window and input behavior such as typing speed and window focus, b) a Chrome plugin to log the participant's browsing behavior, and c) a boot and syncing script to collect the produced log les, encrypt them, and send them to a central server location. The application was o ered to participants as a standalone installer and installed without supervision.
At install time, the installer generates a 16 hexadecimal character token used to uniquely identify participants. This token is used to link the questionnaires serving as ground truth to the log les. We asked the participants to use the Chrome browser for the duration of the experiment. The application syncs log les to a university server once per day as encrypted archive les. The gathered data was inserted into a MySQL database, after which a Java program was used to extract the variables de ned in Table 1. All software developed for the experiment is available from the authors on request.
The monitoring application has some limitations: For keyboard events, we assume that events log keystrokes of the past ve seconds, which is an approximation. This could potentially introduce a minor bias towards lower or higher typing speeds. If two system events were logged within ve minutes, the computer was considered to be on during this time. This could introduce a systemic bias towards higher daily computer times. This duration was determined by experimenting with di erent threshold values to nd plausible periods of time where the computer was on. Similarly, the computer was considered to be active if any mouse or keyboard events were logged in the ve minutes. Finally, we de ned typing accuracy as the number of backspace key presses per key press. This is not a perfectly accurate representation of accuracy as it does not consider delete or text selection as correction mechanisms, and future experiments will consider a more re ned measurement of typing accuracy. 2.3
To determine a participant's baseline Computer Literacy, it was necessary to measure the computer literacy of a participant using an existing method, by nding or developing a questionnaire or test that covered its constituent elements: Computer Attitude, Computer Experience, Computer Skills and Computer Knowledge. A suite of tests would only cover the Computer Skills dimension, so the participants' Computer Literacy was assessed using a questionnaire. While this means the experiment uses a method with the aforementioned shortcomings as validation method, a correlation between the measurements and the questionnaire can be used as an indicator for how promising speci c measurements are as computer literacy predictors. The development and validation of a new questionnaire was outside the scope of this research, and so an existing questionnaire, the INCOBI-R [15], was used. This German questionnaire by Richter et al., a revised version of the INCOBI [14], is a recent and complete questionnaire which covers theoretical and practical computer knowledge, computer anxiety and several computer attitudes (negative and positive). Several questions speci c to the experiment were appended (e.g., how much of your time do you typically spend on this computer, do other people use this computer). The questionnaire was distributed digitally with the installer, and included a request to provide the unique ID of the monitoring application that was generated at install time. 3
After processing the logs and questionnaire data, we examined whether all the data was usable for analysis. We discarded the following variables: Program Focus: This is a variable that shows what program a user was interacting with at any given time. We planned to log changes in window focus, allowing for a detailed view of what a user was looking at. The Win32 API calls that were used for this implementation, however, also logged many focus changes that were solely performed by the system (e.g., an auto-save function of a text processing program might switch focus for one or two seconds every so often, or a system process might perform some task seemingly in the background). This made the focus logging aspect of the experiment unreliable as a measure of user activity, requiring us to omit it. A future iteration will contain a revised version of this logging method. Installed Programs: Based on the categories of programs installed, we initially surmised that it would perhaps be possible to associate several programs with high or low levels of computer literacy. Due to the variety of programs and small sample size, however, this data was not usable in the current experiment.
Running Programs: Similar to installed programs, the number program categories versus the number of subjects also precluded the use of this variable.
Visited site categories: Also discarded as the variety of categories versus subjects was too great.
The remaining observed variables of Table 1 were used. For the INCOBIR questionnaire, the division of subscales de ned in the questionnaire was initially adhered to: Theoretical Computer Knowledge, Practical Computer Knowledge, Computer Anxiety and Computer Attitude. The Computer Attitude scale is de ned as a 3 level construct consisting of 8 variables: attitudes towards computers on a societal versus a personal level (Personal Experience / Social Implications), work versus entertainment (Work and Education / Entertainment and Communicatoin) and perceived bene t/feelings of control (Perceived Usefulness / Perceived Lack of Control). 3.1
Since it is unknown whether the linearity assumption holds, a Spearman correlation analysis was performed to identify promising variables to measure. Due to the small sample size and exploratory nature of the experiment, Exploratory Factor Analysis is reserved for a future experiment with a larger sample size. The resulting correlation grid can be found in Figure 3. Also due to the small sample size, only very few variables were statistically signi cant (i.e., the time based dimensions such as avg on/day and avg active/day). As such, p-values were omitted. 3.2
The full grid of Figure 3 was re ned to a simpli ed form suitable for extracting promising variables to measure in a future experiment. This was done by merging several variables and omitting others: a possible link between cursor speed and Computer Literacy. However, since speed and accuracy of cursor movement is a motor skill that can be trained like any other, we believe that, similar to Typing Speed, Cursor Speed could prove to be a predictor of Computer Literacy.
Average Time On was, surprisingly, not the uniformly positive predictor of Computer Literacy that we expected, instead showing a positive relation to Perceived Lack of Control. Computer Anxiety, Knowledge and Perceived Usefulness still support this conclusion, however. We cannot immediately explain the link between Perceived Lack of Control and Average Time On, and speculate that this might be because of the relatively small sample size. A future experiment might shed light on whether this relationship holds in larger sample sizes. 4.1
Should Typing Speed, Cursor Speed and Average Time On be predictors for Computer Literacy or any of its parts (Computer Attitudes, Experience, Skills and Knowledge), these elements could be incorporated into persuasive systems design, increasing the persuasiveness of systems through a new level of context awareness. Systems could be made to offer technical advice on a level more likely to be appropriate to a user's technical competency and/or interest level. This would reduce user frustration and improve engagement, helping to o er information, according to Fischer [6], in the 'right' way, to the 'right' person.
Should this prove to be a reliable tool to model users, this method could also be extended to o er valuable and more complex insights into individual users by modeling latent variables and mental constructs such as motivation or security consciousness through the observation of humancomputer interaction. 4.2
We acknowledge several limitations to the experiment and its conclusions. First, the experiment had a sample size that was too small for meaningful statistical analysis. Any conclusion drawn here is intended to guide future work, indicating possible correlations and demonstrating a novel measurement method rather than presenting statistically signi cant conclusions. Second, embedding sensors in a speci c application will result in a partial picture of a user's interaction patterns. To gain the fullest understanding of a user, such sensors would need to be present on all of a user's devices. We believe, however, that a partial view is su cient when it covers the context for which it is intended (i.e., the user interacting with speci c applications). Further work would be needed in order to verify this assumption. Finally, it has not been established whether the observed interaction patterns are su ciently unique to distinguish di erent users. If this is not possible with any degree of accuracy, any application with multiple users would need a mechanism to verify which user is currently operating the device. An obvious extension of the current work would be to repeat the current experiment with a larger sample size. There are, however, several other re nements that can be made to the current experimental setup. First, expanding measurement to include mobile platforms could provide insights into users' mobile literacy and its links to computer literacy. Second, variables that were omitted from this analysis such as program focus should be revisited in a new experiment. Finally, we plan to use this data about users' Computer Literacy to investigate new ways to improve communication towards the user. Our future work includes the development of an application to advise users in the area of information security, adapting the content of the advice based on a user's Computer Literacy. 5
This research is sponsored as part of the PISA project by NWO and
KPN under contract 628.001.001.
9. J. Kruger and D. Dunning. Unskilled and unaware of it: How
di culties in recognizing one's own incompetence lead to in ated
self-assessments. Journal of Personality and Social Psychology,
77(
Zeitschrift fur Padagogische Psychologie, 24(