This contribution presents results from two exploratory studies on technology acceptance and use of widget-based personal learning environments. Methodologically, the investigation carried out applies the unified theory of acceptance and use of technology (UTAUT). With the help of this instrument, the study assesses expert judgments about intentions to use and actual use of the emerging technology of flexibly arranged combinations of use-case-sized mini learning tools. This study aims to explore the applicability of the UTAUT model and questionnaire for widget-based personal learning environments and reports back on the experiences gained with the two studies.
A personal learning environment can be modelled as a
network of people surrounding an individual with the
persons in this network making use of artefacts and tools
while they engage in isolated or collaborative activities
of more or less planful (co-) construction of knowledge
and information
Though the individual tries to structure the environment, she is not fully in control to design it, as interactions of the agents in the network (persons, tools, artefacts) are not working towards a common goal or joint plan. Moreover, affordances and characteristics of its agents moderate performance and behaviour in this fragile ecosystem. Even where parts of this environment are . subjected to user control, for example in selection and use, this is largely influenced by attitudes, norms, expectations, intentions, and the like.
Widget-based personal learning environments provide a
technology for meeting these heterogeneous
requirements better. They challenge the dominant design
of classical managed learning environments offered by
institutions and open up environments for flexible
recombination of their elements
The determinants of acceptance and use have been
studied in several models – the unified theory of
technology acceptance and use of technology (UTAUT)
being one of the most elaborate (see Venkathesh et al.,
2003). UTAUT has been elaborated from a set of eight
prominent models for information technology acceptance
research and has been found to outperform these
precursors with respect to the ability to explain user
intention to use information technology
The determinants identified in the unified theory relate to
individual reactions to technology such as expressed
expectations, assessed social pressure, and other types of
statements about influencing factors, that are known to
drive the intention to use and – ultimately – actual use
behaviour (see Figure 2). Together, the variables of the
model have been found to explain about 70% of the
variance in user intention to use particular technologies
For the first study, a hands-on session was prepared for participants of a workshop held at the Joint European Summer School in Technology-Enhanced Learning (JTEL‟10). The session focused on constructing a personal learning environment in form of a paper prototype. The participating 13 doctoral candidates and mentors were first briefed on the widget approach as such and with the help of selected widgets from the language technology for lifelong learning (LTfLL) project on typical use-cases of individual widgets. Each group was then provided with empty flipchart paper (representing an empty widget container) and with printed and blank widget cards, which they could use to populate their own widget space. They were instructed to discuss and create a personal learning environment with the help of these materials. The group session lasted for about 45 minutes and finished with a group presentation of the PLE created back to the plenum. Afterwards, the participants were asked to fill in the technology-acceptance questionnaire.
The second study took place at the University of Bukarest, with 25 computer science students participating. The students were working for one day with an elgg-based implementation of a personal learning environment (Wild et al., 2010) to achieve certain given tasks (see snapshot of the system in Figure 11). Afterwards, they filled in the questionnaire. The questionnaire deployed consisted of a set of items, which were minimally adjusted from the original questionnaire of Venkatesh et al. to fit to the scenario of widget-based PLEs. Besides the core constructs mentioned above, additional questions were included to collect data on moderating variables.
The items of the questionnaire are grouped into five sets (see Table 1), supported by questions on moderating variables such as gender, age, highest level of education, employment, and generic questions about computer and internet usage skills. These five 1 The system can be accessed at http://augur.wu.ac.at/elgg/; an openID is required for the full functionality to work. constructs cluster together items on expectations on performance gains (PE) and efforts to be invested (EE), statements assessing whether there is social pressure pushing forward the use of widget-based PLEs (SI), availability of support and resources necessary (FC), and – finally – intentions to use (BI).
Within this section, results of the two studies will be reported. The section will start with an overview in form of descriptive statistics on the grouped items as proposed in the unified theory of acceptance and use of technology. In a second step, the item-item reliability of the constructs used is measured with Cronbach‟s α to gain insight into whether the questionnaire items of the model chosen in fact converge in the groups proposed. Since this was not the case, we calculated a factor analysis after exclusion of unreliable items to see if the groups predicted by theory are reflected in the empirical data gathered in the two studies. The results indicate that the grouping as proposed in the underlying model can be justified, though alternative ways of clustering would be possible. A correlation analysis rounds up the section. For all items of the questionnaire, basic descriptive statistics were calculated as listed in Table 2 and 3, thereby taking into account the average of the items for each construct. As visible from Table 2, the users rated the expected benefit for performance using widget-based PLEs with moderate 3.33 in the first study. The effort expected is rated with 3.88, which means that the users think that this approach makes it moderately easy to achieve their goals. The social influence has the lowest average with 2.98: users slightly tend to agree to being socially influenced by others to use this approach. The facilitating conditions are rated moderate high, which could express that users have the resources and the knowledge to use the system, but additional improvements of support are possible. The intention to use the system in the next 12 months is moderately high. The second study shows similar means compared to the first one. One notable exception can be found in the items aggregated under behavioural intention to use. While in the first study the mean was slightly higher than the average (3.43) in the second study the mean is lower (2.79). To investigate the quality of the questionnaire in this context of widget-based PLEs, the inter-item reliability was calculated using Cronbach‟s α to detect whether the items correlated high amongst each other in each given construct. If inter-item reliability is found to be high, this would express that the items of each construct are in line with the theoretical model proposed in the UTAUT. In the first study, „performance expectancy‟ consists of the four items U6, RA1, RA5, and OE7 – and Cronbach‟s α for these four items is .76. While three items have a high inter-item correlation, the correlation of OE7 is weak for all other items. If OE7 is excluded Cronbach α rises to .95. The item “If I use the widgetbased PLE, I will increase my chances of getting a raise” seems not to fit the other three items, which target the usefulness of the system for the job, to accomplish tasks, and to increase the productivity. Since the target groups investigated were early career and more advanced researchers in this first data set, this finding is not very surprising: other performance will rather less directly impact on salaries in an academic setting than in a business.
Analyzing the items of the „effort expectancy‟ (items EOU3, EOU5, EOU6, and EU4) finds a Cronbach‟s α of .83: the inter-item correlation matrix shows low correlations of the item EOU3 with the other items. Although all four items are directed towards ease of use and easiness to understand the system, the item “My interaction with the widget-based PLE would be clear and understandable” (EOU3) seemed to be not properly formulated. Even though Cronbach‟s α rises to only .88, EOU3 will be excluded from the further analysis as for its low correlation with the other items.
The factor „social influence‟ consists of the four items SN1, SN2, SF2, and SF4. Removing item SF4 would raise Cronbach‟s α only from .80 to .86 and thus the item will not be excluded from the further analysis. Analyzing the items for the factor „facilitating conditions‟ (PBC2, PBC3, PBC5, and FC3), Cronbach‟s α loads with .29 rather low. After the exclusion of FC3 and PBC5, which both correlated low with all other items of this factor, Cronbach‟s α rises to .79. While PBC2 and PBC3 ask about resources and knowledge to use widget-based PLEs and are positive formulated, the item PBC5 “The widget-based PLE is not compatible with other systems I use” is negative formulated”, which could be the reason for its low correlation with the other items. The item FC3 asks if assistance is available for using the system. While the first two items could be seen more as in control of the individual, the last item contains a social component, which could have led to the low correlation with the other items.
The items of the factor „behavioural intention‟ have a high Cronbach‟s α of .96.
In the second study, the items for „performance expectancy‟ (U6, RA1, RA5, OE7) have a high interitem reliability (Cronbach‟s α = .84). While in the first study we excluded the item OE7 for the further analysis, we will keep it for the second study.
The items for „effort expectancy‟ (EOU3, EOU5, EOU6) have a Cronbach‟s α of .89 (.92 if EOU3 deleted). While we excluded EOU3 from the first study, we will include it for the following analysis, due to the only small gain of the Cronbach‟s α, when removed. This could indicate that the item EOU3 should be reformulated in further studies.
Amongst the items for „social influence‟, Cronbach‟s α of SN1, SN2, SF2 and SF4 is .76. This is in line with the results of the first study.
Cronbach‟s α for the „facilitating conditions‟ (PBC2, PBC3, PBC5, FC3) is again rather low (.28). After the exclusion of PBC5, it rises to .49 (and with FC3 excluded to .93). This is similar to the first study and could be seen as a hint to reformulate or to drop these items in future studies.
The „behavioural intention‟ items (BI1, BI2, BI3) have a high Cronbach‟s α of .91.
Except for the items EOU3 and OE7 that will be kept for this second data set, we could repeat the results of the first study regarding the inter-item reliability: both studies identify a problem for two items in the facilitating conditions; these two items PBC5 and FC3 should be dropped or reformulated in future studies. In the next step, we apply a factor analysis to detect if the constructs as grouped by the UTAUT model are also reflected in factors for our data sets. Therefore, we first tested the statistical requirements for normal distribution, which is a precondition for the conduction of an exploratory factor analysis. The Shapiro-Wilk tests indicate that normal distribution is only given for the items RA1, RA5, SN1, SN2, SF2, PBC2, and BI3 of the first study. The Shapiro-Wilk tests for the second data set indicate that normal distribution is only given for the items OE7, BI2 and BI3, compared to RA1, RA5, SN1, SN2, SF2, PBC2, and BI3 for the first study. This has to be taken into account for the interpretation of the following factor analysis, which should be only applied if all items are normal distributed. However, since the goal of this study is to gain experience with the UTAUT model and to further develop the questionnaire, the results are still considered relevant, but have to be interpreted with precaution.
According to the UTAUT model, all factors (= groups of items) should be more or less independent from each other. To test this assumption on our data, a factor analysis with varimax rotation was calculated, providing means to investigate whether the items load on factors as suggested by their theoretical underpinnings.
The pre-analysis of the first study resulted in a nonpositive correlation matrix, which normally indicates the need of a bigger sample size. The scree plot would suggest a two- or three-factor solution. To investigate, however, the closeness to the theoretically postulated clustering, the rotated factor analysis calculated with the five factors (as indicated by the UTAUT model) shows the results presented in Table 4.
The three items for performance expectancy (component 1) as well as for effort expectancy (component 2) and social influence (component 3) load high on factors, see Table 4. This can also be found for two out of the three variables for behavioural intention (see component 4) and for one variable of the facilitating conditions (see component 5). According to the rotated factor analysis, however, PBC3 loads high on the factor of effort expectancy, and BI1 high on the factor of the performance expectancy items. Still, the general picture is that the items of our first study load on factors similar to the factors predicted by UTAUT.
Based on these findings of the factor analysis, the items with high inter-item correlations and high level of independence as suggested by the factor analysis will be used for the final next step of the analysis: the calculation the correlations of the UTAUT factors. For the first study, performance expectancy consists of the items U6, RA1, and RA5. Effort Expectancy consists of the items EOU5, EOU6, EU4 and social influence of the items SN1, SN2 and SF2. Only the item PBC2 of the facilitating conditions remains, and the items of the behavioural intention to use are BI2, and BI3. The pre-analysis of the second study revealed that the Kaiser-Meyer-Olkin of the partial correlation coefficients is relatively low with 0.4 (values higher than .5 are seen a condition for calculating a factor analysis). However, the Chi-Square value of Bartlett‟s test is high (288,45; df = 136) and the probability of an error is low. As in the first study, the requirements for a factor analysis are not satisfied. As the goal of the study is to find hints for the construction of the next questionnaire, the factor analysis was calculated as it could help to determine if certain items should be assigned to another construct of UTAUT or not.
Based on these results we calculated a factor analysis with five fixed components with varimax rotation. The result is presented in Table 5.
The results of the rotated component matrix are less conclusive as in the first study, but can be interpreted when having the factors of the UTAUT model in mind. The items RA1, RA5 of the performance expectancy load high on component 2, while the items U6, RA5 and OE7 load high on component 4. As the items PBC2 and PBC3 of the Facilitating Conditions load high on component 2 as well, we will take into account for the further analysis the items U6, RA5 and OE7 of component 4.
The items of the effort expectancy (EOU5, EOU6, and EU4) load high on component 1, while EOU3 loads high on component 5. The items of the effort expectancy and the behavioural intention to use load high on the same component 1.
Only the items SF2 and SF4 of the social influence variable load high on component 3, whereas SN1 loads high on component 5 and SN2 on component 2. Based on the results of the inter-item reliability and factor analysis, the items RA1, EOU3, SN1, SN2, PBC2 and PBC3 were excluded.
After the application of the inter-item reliability and the factor analysis, we calculated again the descriptive statistics. This time it takes into account the findings from the above-mentioned analysis steps and thus represents a cleaner model of the data. For the first study, the items of each construct were aggregated again and basic descriptive statistics were calculated (see Table 6). The results of the descriptive statistics, using the refined set of items, show slightly higher values as compared to the first descriptive statistics. Especially the effort expectancy and the behavioural intention to use the system with a mean of 4.0 and relatively low standard deviations are indicators that the users of the scenario would use the system and they perceive it as easy to use. For the second study, the results of the descriptive statistics show a slightly different picture than in the first study. The facilitating conditions with a mean of 4.5 are more than one point higher than in the first study (3.42). And the behavioural intention to use was high in study 1 (mean of 4.0) it is lower in the second study (2.8). The other constructs have a similar mean in both studies.
In a further analysis step, we calculated the correlations between the constructs as proposed in UTAUT. First, we examined the normal distribution as a precursor for the Pearson test.
The Shapiro-Wilk test for normal distribution indicates normal distribution for each of the aggregated components of the first study. With normal distribution given, the Pearson correlation (one tailed) was calculated for each of the aggregated components. The results are the following. The correlation between Performance Expectancy and the Behavioural Intention are low (r = .14; not significant). The correlation between Effort Expectancy and Behavioural Intention is medium (r = .54*). There is a high correlation between the Social Influence and the Behavioural Intention (r = .76**). A Structural Equation Model was calculated using AMOS, but did not lead to statistically satisfying results, although tested with a variety of models. This can be attributed to the relatively small sample size. Regarding the second study, except from the facilitating conditions, the Shapiro Wilk test indicated normal distribution, which leads to the decision of using the Pearson Correlation (one-tailed).
The correlation between effort expectancy and the behavioural intention to use was the only significant one with r = .60; all other correlations were not significant. This value is similar to the one in the first study (r = .54). The significant correlation between social influence and intention to use could not be replicated.
A Structural Equation Model was tested with AMOS, taking into account the reduced set of items (refined with the insights from the inter-item reliability analysis and the factor analysis). The model, however, was not admissible. The AMOS model calculated with all items produced output, but was not admissible. This can be attributed to the small number of participants in the studies. A follow up study would shed further light on this.
The paper presents results about the applicability of the technology acceptance model as proposed in UTAUT – adapted to the context of widget-based Personal Learning Environments. The UTAUT questionnaire can be seen as an instrument to assess whether users are highly likely to actually use a widget-based PLE. The acceptance model predicts a high probability of use if the construct behavioural intention and the facilitating conditions are high. In two studies, we applied this method with the goal to gain experiences with this instrument and to tailor the questionnaire to the context of widget-based PLEs. Both studies found high and moderately high values for the facilitating conditions (study one: 3.42, study two: 4.50, see Tables 6 and 7). With regards to the behavioural intention to use, the two studies differed: whereas study one found with 4.0 moderately high values, study two was 2.79 rather average. As the data sets were relatively small, these findings cannot be generalised and must be handled with precaution.
The results have been encouraging, but it also became clear, that the model (and questionnaire) couldn‟t be mapped directly to the domain of PLEs. Both studies show in their inter-item reliability and factor analysis, that the components of the original UTAUT model can be more or less confirmed. These methods, however, also revealed potential to improve the model and questionnaire when applied to study acceptance of PLEs. The reason why the structural equation model was not admissible in both studies seems to lie in their relatively small number of participants. However, further research is needed to gain experience about a practical sample size. This is especially important for the validation of an acceptance model for PLE scenarios.
Although technology acceptance studies are widely used, studies from one domain cannot be compared with the domain of investigation without limitations. To build up a strong argument about the explanatory power of this study, a baseline from a similar study setup would be required.
Furthermore, as Al-Qeisi (2009) summarises, the results are limited in so far as they base on self-reports of users, but not on their actual use. In other words, further tests to check validity against the criterion actual usage would be helpful.
Additionally, another limitation can be found in the selection of participants for this study: one important moderator effect we have to consider is, that both samples consisted of technically skilled persons. They can be seen as early-adopters or innovators of new technology. Yet, this group of people does not necessarily represent the larger group of people who are less technology affine. It is hard to predict how these results will change, when turning to people with other backgrounds.
As the goal of the study was to test if the technology acceptance model is applicable for the domain of PLEs, as such the results of the first two studies can be seen as promising for further work to refine the method. The results, however, should not be mistaken as statements about the general usefulness of PLEs according to the UTAUT model. These statements would be misleading in this early research stage of the validation of the technology acceptance model and its instrument for PLEs.
This work has been co-funded by the European Union in the projects Stellar and LTfLL under the Information and Communication Technologies (ICT) theme of the 7th Framework Programme. Traian Rebedea and Bernhard Hoisl helped organising the workshop for the first study and Traian conducted the validation workshop of the second study. Fridolin Wild and Thomas Ullmann conducted the statistical analysis and wrote up the paper in discussion with Peter Scott.