=Paper=
{{Paper
|id=Vol-2340/06-BCSS2019_paper
|storemode=property
|title=Towards More Individualized Interfaces: Automating the Assessment of Computer Literacy
|pdfUrl=https://ceur-ws.org/Vol-2340/06-BCSS2019_paper.pdf
|volume=Vol-2340
|authors=Roeland H.P. Kegel,Marten van Sinderen,Roel J. Wieringa
|dblpUrl=https://dblp.org/rec/conf/persuasive/KegelSW19
}}
==Towards More Individualized Interfaces: Automating the Assessment of Computer Literacy==
https://ceur-ws.org/Vol-2340/06-BCSS2019_paper.pdf
Towards More Individualized Interfaces:
Automating the Assessment of Computer
Literacy
R.H.P.Kegel, M. van Sinderen, R.J. Wieringa
University of Twente
Abstract. Computer Literacy is an important predictor for how profi-
cient a person is in its interaction with computers, which can determine
whether a person is motivated and able to use specific software. Mea-
suring Computer Literacy or its constituent elements (Skills, Attitude,
Knowledge and Experience) has traditionally been done using question-
naires. This method has several limitations: it is effort-intensive for sub-
jects, subject to cognitive biases, and constitutes only a snapshot of a
person’s actual Computer Literacy. This limits the usefulness of Com-
puter Literacy as a factor in persuasive systems design. In this paper,
we describe an experiment to test the design of a system that extracts
elements of Computer Literacy based on observation of human-computer
interaction. This new method has the potential to enable the use of Com-
puter Literacy in software design by addressing some of the barriers to its
use ”in the wild”, opening up new possibilities for tailored and adaptive
systems.
1 Introduction
Digital technologies have become pervasive in our society. We now use
computers for work, entertainment, administration and a host of other
applications in our daily lives. To do so, we need to understand how to
interact with computers: how to open documents, find information on
the world wide web, look up directions, and so on. In short, the ability
to use software has become a vital skill in our daily lives. As the use of
software has moved from an expert skill to a basic one, the spectrum of
user types has become wider and more diverse. Persuasive technologies
often depend on understanding these user types that they are trying to
reach. One of the principles of persuasive systems design listed by Oinas-
Kukkonen and Harjumaa[11] is tailoring: the ability to adapt a system
to the user’s characteristics and preferences.
Many methods exist to gather information for such tailoring (e.g., in-
terviews, surveys, observational studies) on the level of groups of users.
On an individual level, systems that display some form of awareness of a
user’s preferences, such as recommender systems, have become common.
But context awareness in software design is a multi-dimensional con-
struct [6], and while some elements of context awareness are now com-
monplace in software systems, others remain hard to implement. One
of these elements in particular is the right way to communicate with
users. The way users are approached has an effect on how the message
is received. The Elaboration Likelihood Model (ELM) [12] of Petty and
Cacioppo postulates that ”Although people want to hold correct attitudes,
the amount and nature of issue-relevant elaboration in which people are
willing or able to engage or evaluate a message vary with individual and
situational factors”. When considering the context of using a digital sys-
tem to change behaviors or attitudes, Digital skills is one such skill-based
variable that can be taken into account to personalize a message. For
example, in a digital security application such as a firewall, an interface
that provides more control and information would be desirable for highly
skilled computer users that are capable and willing to engage in a higher
level of elaboration, whereas novice users would be overwhelmed. This
makes understanding a user’s proficiency with computers crucial to de-
signing the right interface to interact with them. To create a system that
adapts to individual users’ computer proficiency, however, would require
the system itself to know or measure such proficiency.
Bawden performed a review of Digital Literacy and related concepts
in 2001 [1], examining the relationship between several closely related
concepts such as Digital Literacy, Information Literacy and Computer
Literacy. We focus our attention on Computer Literacy, which is a con-
cept that dates back to 1972 [2], but remains relevant today. This work
builds on a recently completed systematic literature survey [13] which
has refined the concept based on 190 existing articles. The article recog-
nizes four main dimensions within Computer Literacy: Computer Expe-
rience, Computer Knowledge, Computer Skills and Computer Attitude.
All of these affect the proficiency with (and adoption of) software in
some way. Computer Attitude affects computer interaction on multiple
levels. Venkatesh [18] indicated a relationship between several elements
of Computer Attitude (e.g., Computer Self-Efficacy, Computer Anxiety
and Computer Interest) and usage acceptance and user behavior as seen
through the Technology Acceptance Model [4]. This is a finding that
has been reaffirmed by other research in the field [17]. Similarly, Com-
puter Experience and Computer Knowledge have also been found to be
a significant predictor of technology adoption [7], [16].
Traditionally, Computer Literacy is measured using questionnaires. But
questionnaires suffer from several problems when used in software. First,
they constitute a snapshot of a user’s Computer Literacy (attitude, skills
or otherwise). Persons’ attitudes towards computers can change over
time, while experience, skills and knowledge are sure to grow along with
their computer use. Second, questionnaires, especially those that deal
with a user’s perceived skill are subject to perception biases such as the
Dunning-Kruger effect [9], limiting their validity. Finally, questionnaires
require active participation of users, which is not desirable because it re-
quires extensive interaction with subjects, presenting a barrier to adop-
tion.
To mitigate these three issues, we propose to design software that can
gauge a user’s Computer Literacy in real time. While there are specific
tests for Computer Skills such as the European Computer Driving Li-
cence (ECDL) [3], these still require extensive user interaction and suffer
from the same snapshot issues mentioned previously. To date, no re-
search exists into automated assessments of Computer Literacy. In this
paper, we describe an exploratory experiment that was designed explore
a way to fill this gap, finding elements of human-computer interaction
that can be used as indicators of Computer Literacy or its constituent
parts (Attitude, Knowledge, Skills and Experience). In this experiment,
participants installed a software tool that let us monitor several aspects
of their interaction with the computer. Additionally, these participants
filled in the INCOBI-R, an existing validated Computer Literacy ques-
tionnaire, to assess their computer knowledge, skills and attitude. Addi-
tional questions were added to identify their Computer Experience. The
resulting data (software log files and questionnaires) were subjected to
a correlation analysis, identifying several possible relations between the
questionnaire results and the software logs. This research aims to answer
the following research questions:
RQ1: What elements of Computer Literacy measured by questionnaires
can be observed by logging human-computer interactions?
RQ2: Which of these logged computer interactions give the best indi-
cation of a person’s Computer Literacy?
Based on the outcome of our exploratory experiment, we discuss pos-
sible implications for the real-time measurement of Computer Literacy
and future experiments to be done to corroborate our preliminary re-
sults. The rest of this paper is structured as follows: First, we describe
the experimental design and encountered issues during implementation.
Then, we present the results of the experiment. Finally, we discuss the
answers to the research questions above, limitations, implications and
future work.
2 Method
The experiment in this paper was designed as an exploratory study to
find computer interaction patterns that might be correlated with dimen-
sions of Computer Literacy. In the experiment, participants were asked
to both fill in the INCOBI-R Computer Literacy questionnaire, and in-
stall a software tool to collect human-computer interaction data. This
interaction data was compared to the questionnaire results.
The experiment was performed using ten participants. Participants ranged
widely in age (21-63), context (work, entertainment etc.) and skill level
(as measured by the INCOBI-R). The experiment was planned to run
over the course of a minimum of 2 weeks in order to collect sufficient
data. Data was collected on private computers in a home setting of the
participants, although most participants indicated they also used the
computer for work/study purposes. Participants were asked to fill in the
questionnaire at the beginning of the experiment.
2.1 Determining What To Log
To determine which elements of computer interaction could yield mean-
ingful data on a user’s Computer Literacy, we examined its constituent
Variable Description
Computer Experience
Avg on/day Average time the computer was on per day
Avg active/day Average time the computer was in use per day
Active/idle ratio Time the computer spends active vs. idle
Basic Computer Operation Skills
Cursor speed Average cursor move speed in pixels
Typing speed Average typing speed
Typing accuracy Amount of backspace per character typed
Application Specific Skills
Running programs Running time per program
Focused programs Time a program spent as top window
Time spent per program category Running time per program category
Installed programs Installed programs (measured per day)
Internet Skills
Browsing time Average time spent browsing internet per day
Time per site category Average time per website category per day
Table 1. Information logged by the software tool
Fig. 1. Elements of Computer Literacy and variables measured during the experiment.
Parts in grey are known parts of Computer Literacy. Parts in white are variables that
were measured using the software tool during the experiment.
parts, as defined in the previously mentioned literature survey [13]. For
each of the elements of Computer Literacy, we discussed software metrics
that might feasibly be implemented in the tool by the author and the
software engineer employed for this project. We constrained ourselves to
computers using the Windows operating system due to three factors: a)
the difficulty in finding a wide range of test subjects for other operat-
ing systems, b) the amount of available programming expertise, and c)
the known monitoring options available in Windows computers through
Python libraries interfacing with the Win32 API. A final list of what was
measured can be found in Table 1. The link between these variables and
the elements of Computer Literacy survey can be found in Figure 1.
2.2 Monitoring Software Design
Fig. 2. The software tool used for logging participants’ interactions with the computer.
To log useful data of the participants, observation in a real-life environ-
ment for a prolonged period was vital. To do so, a software application
was developed for the Windows operating system (see Figure 2) con-
sisting of three parts: a) a logging module consisting of several Python
scripts using the Win32 API to log window and input behavior such as
typing speed and window focus, b) a Chrome plugin to log the partici-
pant’s browsing behavior, and c) a boot and syncing script to collect the
produced log files, encrypt them, and send them to a central server loca-
tion. The application was offered to participants as a standalone installer
and installed without supervision.
At install time, the installer generates a 16 hexadecimal character to-
ken used to uniquely identify participants. This token is used to link the
questionnaires serving as ground truth to the log files. We asked the par-
ticipants to use the Chrome browser for the duration of the experiment.
The application syncs log files to a university server once per day as
encrypted archive files. The gathered data was inserted into a MySQL
database, after which a Java program was used to extract the variables
defined in Table 1. All software developed for the experiment is available
from the authors on request.
The monitoring application has some limitations: For keyboard events,
we assume that events log keystrokes of the past five seconds, which is
an approximation. This could potentially introduce a minor bias towards
lower or higher typing speeds. If two system events were logged within five
minutes, the computer was considered to be on during this time. This
could introduce a systemic bias towards higher daily computer times.
This duration was determined by experimenting with different threshold
values to find plausible periods of time where the computer was on.
Similarly, the computer was considered to be active if any mouse or
keyboard events were logged in the five minutes. Finally, we defined
typing accuracy as the number of backspace key presses per key press.
This is not a perfectly accurate representation of accuracy as it does not
consider delete or text selection as correction mechanisms, and future
experiments will consider a more refined measurement of typing accuracy.
2.3 Computer Literacy Questionnaire
To determine a participant’s baseline Computer Literacy, it was neces-
sary to measure the computer literacy of a participant using an existing
method, by finding or developing a questionnaire or test that covered its
constituent elements: Computer Attitude, Computer Experience, Com-
puter Skills and Computer Knowledge. A suite of tests would only cover
the Computer Skills dimension, so the participants’ Computer Literacy
was assessed using a questionnaire. While this means the experiment uses
a method with the aforementioned shortcomings as validation method,
a correlation between the measurements and the questionnaire can be
used as an indicator for how promising specific measurements are as
computer literacy predictors. The development and validation of a new
questionnaire was outside the scope of this research, and so an existing
questionnaire, the INCOBI-R [15], was used. This German questionnaire
by Richter et al., a revised version of the INCOBI [14], is a recent and
complete questionnaire which covers theoretical and practical computer
knowledge, computer anxiety and several computer attitudes (negative
and positive). Several questions specific to the experiment were appended
(e.g., how much of your time do you typically spend on this computer,
do other people use this computer). The questionnaire was distributed
digitally with the installer, and included a request to provide the unique
ID of the monitoring application that was generated at install time.
3 Results
After processing the logs and questionnaire data, we examined whether
all the data was usable for analysis. We discarded the following variables:
Program Focus: This is a variable that shows what program a user
was interacting with at any given time. We planned to log changes
in window focus, allowing for a detailed view of what a user was
looking at. The Win32 API calls that were used for this implemen-
tation, however, also logged many focus changes that were solely
performed by the system (e.g., an auto-save function of a text pro-
cessing program might switch focus for one or two seconds every so
often, or a system process might perform some task seemingly in the
background). This made the focus logging aspect of the experiment
unreliable as a measure of user activity, requiring us to omit it. A
future iteration will contain a revised version of this logging method.
Installed Programs: Based on the categories of programs installed,
we initially surmised that it would perhaps be possible to associate
several programs with high or low levels of computer literacy. Due
to the variety of programs and small sample size, however, this data
was not usable in the current experiment.
Running Programs: Similar to installed programs, the number pro-
gram categories versus the number of subjects also precluded the use
of this variable.
Visited site categories: Also discarded as the variety of categories versus
subjects was too great.
The remaining observed variables of Table 1 were used. For the INCOBI-
R questionnaire, the division of subscales defined in the questionnaire
was initially adhered to: Theoretical Computer Knowledge, Practical
Computer Knowledge, Computer Anxiety and Computer Attitude. The
Computer Attitude scale is defined as a 3 level construct consisting of
8 variables: attitudes towards computers on a societal versus a personal
level (Personal Experience / Social Implications), work versus entertain-
ment (Work and Education / Entertainment and Communicatoin) and
perceived benefit/feelings of control (Perceived Usefulness / Perceived
Lack of Control).
3.1 Full Correlation Grid
Since it is unknown whether the linearity assumption holds, a Spear-
man correlation analysis was performed to identify promising variables
to measure. Due to the small sample size and exploratory nature of the
experiment, Exploratory Factor Analysis is reserved for a future exper-
iment with a larger sample size. The resulting correlation grid can be
found in Figure 3. Also due to the small sample size, only very few vari-
ables were statistically significant (i.e., the time based dimensions such
as avg on/day and avg active/day). As such, p-values were omitted.
3.2 Reduced Correlation Grid
The full grid of Figure 3 was refined to a simplified form suitable for
extracting promising variables to measure in a future experiment. This
was done by merging several variables and omitting others:
Fig. 3. The correlation grid for INCOBI-R variables versus software measurements,
using a Spearman correlation analysis. Computer Attitude is split into Personal Expe-
rience (PE)/Societal Implications (SI), Entertainment and Communication (EC)/Work
and Education (WE), and Perceived Usefulness (PU)/Perceived Lack of Control (PLC).
Fig. 4. A simplified correlation grid for INCOBI-R variables versus software measure-
ments, using a spearman correlation analysis
Attitude: The Computer Attitude model presented the INCOBI-R con-
tains several distinctions which are less relevant to the current ex-
periment. First, Attitude towards Work versus Entertainment are
less relevant to the measurement of one’s ability to use a computer.
Additionally, this distinction is less relevant in context aware sys-
tems design as most applications will function in only one of these
domains. Second, the difference between attitudes for personal and
societal implications are less likely to be correlated with the variables
observed by the software tool. To reflect these two points, Computer
Attitude was simplified to two variables: Perceived Lack of Control
and Perceived Usefulness. This was done by taking the mean of their
constituent parts.
Knowledge: Theoretical and Practical computer knowledge were sep-
arate subscales in the INCOBI-R, but since both of the scales con-
tained questions that were firmly in the Computer Knowledge di-
mension, they were merged.
Time: The three measured times (on, active and browser time) were
identical due to the use of the Spearman method. As such, active
and browser time were dropped in favor of Average Time On.
Typing: Similar to Average Time On, Typing Accuracy and Typing
Speed were highly correlated and so only typing speed was used for
the simplified grid.
The reduced correlation grid can be found in Figure 4.
4 Discussion
Figure 4 seems to suggest that there are several possible connections
that can be made between variables observed by the software (Typing
Speed, Cursor Speed and Average Time On) and concepts traditionally
measured via questionnaires and tests (Computer Anxiety, Computer
Knowledge, Perceived Lack of Control and Perceived Usefulness).
Typing Speed appears positively correlated with Computer Literacy.
In 1986, Morrow et al. [10] found no significant correlation between
Computer Anxiety and Typing Speed, but Evans and Simkin [5]
found Typing Speed to one of the primary indicators of ”Computer
Proficiency”. No recent work has been performed to investigate the
link between Computer Literacy and typing skills, but the results
of this experiment seem to suggest that Typing Speed is a positive
indicator of Computer Literacy, especially for Computer Anxiety and
Perceived Usefulness. Furthermore, previous research [8] has found
support for Computer Self-efficacy as a predictor of Computer Skills,
which closely aligns with Perceived Lack of Control, its opposite.
We therefore maintain that typing skill is a possible predictor for
both Computer Skills and Computer Attitude, in line with previous
research.
Cursor Speed seems to be positively correlated to Computer Literacy,
with a stronger relation than Typing Speed in Computer Knowl-
edge and Perceived Lack of Control, and a weaker one in Computer
Anxiety and Perceived Usefulness. To date, no research exists into
a possible link between cursor speed and Computer Literacy. How-
ever, since speed and accuracy of cursor movement is a motor skill
that can be trained like any other, we believe that, similar to Typ-
ing Speed, Cursor Speed could prove to be a predictor of Computer
Literacy.
Average Time On was, surprisingly, not the uniformly positive pre-
dictor of Computer Literacy that we expected, instead showing a
positive relation to Perceived Lack of Control. Computer Anxiety,
Knowledge and Perceived Usefulness still support this conclusion,
however. We cannot immediately explain the link between Perceived
Lack of Control and Average Time On, and speculate that this might
be because of the relatively small sample size. A future experiment
might shed light on whether this relationship holds in larger sample
sizes.
4.1 Implications For Practice
Should Typing Speed, Cursor Speed and Average Time On be predic-
tors for Computer Literacy or any of its parts (Computer Attitudes, Ex-
perience, Skills and Knowledge), these elements could be incorporated
into persuasive systems design, increasing the persuasiveness of systems
through a new level of context awareness. Systems could be made to of-
fer technical advice on a level more likely to be appropriate to a user’s
technical competency and/or interest level. This would reduce user frus-
tration and improve engagement, helping to offer information, according
to Fischer [6], in the ’right’ way, to the ’right’ person.
Should this prove to be a reliable tool to model users, this method could
also be extended to offer valuable and more complex insights into indi-
vidual users by modeling latent variables and mental constructs such as
motivation or security consciousness through the observation of human-
computer interaction.
4.2 Limitations
We acknowledge several limitations to the experiment and its conclu-
sions. First, the experiment had a sample size that was too small for
meaningful statistical analysis. Any conclusion drawn here is intended
to guide future work, indicating possible correlations and demonstrating
a novel measurement method rather than presenting statistically signifi-
cant conclusions. Second, embedding sensors in a specific application will
result in a partial picture of a user’s interaction patterns. To gain the
fullest understanding of a user, such sensors would need to be present on
all of a user’s devices. We believe, however, that a partial view is sufficient
when it covers the context for which it is intended (i.e., the user interact-
ing with specific applications). Further work would be needed in order
to verify this assumption. Finally, it has not been established whether
the observed interaction patterns are sufficiently unique to distinguish
different users. If this is not possible with any degree of accuracy, any
application with multiple users would need a mechanism to verify which
user is currently operating the device.
4.3 Future Work
An obvious extension of the current work would be to repeat the cur-
rent experiment with a larger sample size. There are, however, several
other refinements that can be made to the current experimental setup.
First, expanding measurement to include mobile platforms could provide
insights into users’ mobile literacy and its links to computer literacy.
Second, variables that were omitted from this analysis such as program
focus should be revisited in a new experiment. Finally, we plan to use
this data about users’ Computer Literacy to investigate new ways to
improve communication towards the user. Our future work includes the
development of an application to advise users in the area of information
security, adapting the content of the advice based on a user’s Computer
Literacy.
5 Acknowledgements
This research is sponsored as part of the PISA project by NWO and
KPN under contract 628.001.001.
References
1. D. Bawden. Information and digital literacies: A review of concepts.
Journal of Documentation, 57(2):218–259, 2001.
2. E.G. Begle, W.F. Atchison, S. Charp, W.S. Dorn, D.C. Johnson,
and J.T. Schwartz. Recommendations regarding computers in high
school education. In Conference Board of the Mathematical Sciences,
Committee on Computer Education, 1972.
3. D. Carpenter, D. Dolan, D. Leahy, and M. Sherwood-Smith.
Ecdl/icdl: A global computer literacy initiative. 16Th IFIP
Congress, ICEUT200, Educational Uses of Information and Com-
munication Technologies, 2000.
4. F.D. Davis. Perceived usefulness, perceived ease of use, and user
acceptance of information technology. MIS Quarterly: Management
Information Systems, 13(3):319–339, 1989.
5. Gerald E. Evans and Mark G. Simkin. What best predicts computer
proficiency? Commun. ACM, 32(11):1322–1327, November 1989.
6. Gerhard Fischer. Context-aware systems: the’right’information,
at the’right’time, in the’right’place, in the’right’way, to
the’right’person. In Proceedings of the international working
conference on advanced visual interfaces, pages 287–294. ACM,
2012.
7. M. Igbaria and J. Iivari. The effects of self-efficacy on computer
usage. Omega, 23(6):587–605, 1995.
8. Richard D Johnson and George M Marakas. The role of behavioral
modeling in computer skills acquisition: Toward refinement of the
model. Information Systems Research, 11(4):402–417, 2000.
9. J. Kruger and D. Dunning. Unskilled and unaware of it: How
difficulties in recognizing one’s own incompetence lead to inflated
self-assessments. Journal of Personality and Social Psychology,
77(6):1121–1134, 1999.
10. Paula C. Morrow, Eric R. Prell, and James C. McElroy. Attitudinal
and behavioral correlates of computer anxiety. Psychological Reports,
59(3):1199–1204, 1986.
11. H. Oinas-Kukkonen and M. Harjumaa. Persuasive systems design:
Key issues, process model, and system features. Communications of
the Association for Information Systems, 24(1):485–500, 2009.
12. R. Petty and J. Cacioppo. The Elaboration Likelihood Model of Per-
suasion, volume 19 of Advances in Experimental Social Psychology,
pages 123–205. Elsevier, 1986.
13. R. Klaassen R.H.P. Kegel, S. Barth and R.J. Wieringa. Computer
literacy systematic literature review method. Ctit technical report
series, University of Twente, 2017.
14. Tobias Richter, Johannes Naumann, and Norbert Groeben. Das
inventar zur computerbildung (incobi): Ein instrument zur erfas-
sung von computer literacy und computerbezogenen einstellungen
bei studierenden der geistes-und sozialwissenschaften. Psychologie
in Erziehung und Unterricht, 48(1):1–13, 2001.
15. Tobias Richter, Johannes Naumann, and Holger Horz. Eine
revidierte fassung des inventars zur computerbildung (incobi-r).
Zeitschrift für Pädagogische Psychologie, 24(1):23–37, 2010.
16. I. Sahin and M. Shelley. Considering students’ perceptions: The dis-
tance education student satisfaction model. Educational Technology
and Society, 11(3):216–223, 2008.
17. T. Teo, C.B. Lee, and C.S. Chai. Understanding pre-service teachers’
computer attitudes: Applying and extending the technology accep-
tance model. Journal of Computer Assisted Learning, 24(2):128–143,
2008.
18. V. Venkatesh. Determinants of perceived ease of use: Integrating
control, intrinsic motivation, and emotion into the technology accep-
tance model. Information Systems Research, 11(4):342–365, 2000.