Design Characteristics of Virtual Learning Environments: An Ex- pert Study Daniel Mueller, Saarland University, Germany d.mueller@mis.uni-saarland.de Stefan Strohmeier, Saarland University, Germany s.strohmeier@mis.uni-saarland.de Abstract. Virtual Learning Environments (VLE) constitute the current Information Systems (IS) category for electronically supported corporate training and development. Frequently supposed advantages of using VLE refer, for instance, to the efficiency, individuality, ubiquity, timeliness, and task orientation of learning. However, a crucial precondition of realizing such advantages is an appropriate systems design. Hence, the question which specific design characteristics actually characterize successful VLE is of specific interest for training and development practice. The current paper therefore addresses design characteristics by conducting an expert study which is based on a general theory of IS success and previous insights of the literature. As a result, a set of relevant, well-defined design characteristics is presented and discussed while implications for research and practice are derived. Keywords: Virtual Learning Environments, Design Characteristics, Expert Study. 1 Introduction For decades, electronic learning systems constitute the basic enablers of corporate e- learning. Though designations as categorizations of such learning systems are rather heterogeneous and also change over time, current systems can be pooled under the rubric of Virtual Learning Environments (VLE), which can be understood as electronic Information Systems (IS) for the administrative and didactical support of learning processes in vocational settings by systematically providing corporate learners adequate learning materials as well as corresponding collaboration facilities so as to develop intended qualifications [e.g. 8, 42, 49]. The usage of such systems in corporate training and development is commonly justified based on diverse advantages such as efficiency, individuality, ubiquity, convenience, timeliness, cost efficiency and task orientation of VLE-based learning [e.g. 15, 20, 41]. Such advantages may also explain the ever increasing adoption of VLE in corporate training and development [e.g. 15, 19, 48]. However, the actual realization of such advantages crucially depends on several preconditions, while the specific characteristics of the used VLE constitute a prominent Strohmeier, S.; Diederichsen, A. (Eds.), Evidence-Based e-HRM? On the way to rigorous and relevant research, Proceedings of the Third European Academic Workshop on electronic Human Resource Management, Bamberg, Germany, May 20-21, 2010, CEUR-WS.org, ISSN 1613-0073, Vol. 570, online: CEUR-WS.org/Vol-570/ , pp. 167-185. © 2010 for the individual papers by the papers´ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. aspect. It is evident that only adequately designed VLE will offer the promising potential for success, while ill designed systems may even cause harmful disadvantages. This directly focuses on design characteristics of VLE as a crucial aspect of learning success. Technically [e.g. 18] as managerially oriented literature [e.g.6, 7, 45] congruently understands design characteristics as the set of those inherent information system properties, which determine IS success (while IS success is differently conceptualized as net benefits, user acceptance, or actual usage, among others). Though termed "design" characteristics, such properties critical to the success of VLE gain practical importance for the entire process of developing or else procuring, implementing and applying VLE in organizations. It is not surprising that design characteristics firstly are relevant for developing new VLE. Here design characteristics offer a framework of requirements which mandatorily must be met by the future system to assure its quality. Given that corporate VLE are getting purchased more and more from external vendors, design characteristic also are relevant for the systems procurement, since they offer a valuable set of selection criteria. Beyond development and procurement, design characteristics may also instruct the technical implementation process by defining technical implementation goals. Finally, design criteria offer suitable evaluation criteria for already applied VLE, and hence support the inspection and improvement of existing systems. Given the wide-spread and still increasing usage of VLE, design characteristics of VLE hence are of relevance for a broader group of technical and managerial decision makers in corporate training and development. The current paper therefore aims at elaborating general VLE design characteristics. An expert study is conducted for this purpose. As a general foundation for the study theoretical bases are discussed first. In order to contribute to cumulative research and to integrate the expert study with previous findings subsequently a review of previous research is conducted. Based on this, the method of the expert study is exposed and the results are presented and discussed. Finally, implications for practice and research are derived. 2 Expert Study 2.1 Foundation As a clear explorative empirical method, expert studies usually are employed to gain insights in topical domains which are theoretically not or at least not well developed and hence, are not open to confirmative research. In certain respects, this applies also to research into design characteristics of VLE. At least, there is no completely developed theory of VLE design which would allow for a direct elicitation of the desired design characteristics. However, alternative foundations may be found in more general theories of – given the subject of the study – in the area of general IS design or general IS success. In the recently flourishing area of IS design the necessity of a general theory of IS design is well recognized [e.g. 14]. However, so far rather procedural models of design research have been offered [e.g. 14, 35], while an explicit theory of IS design, which directly unfolds design characteristics or at least allows to derivate design characteristics, is missing at present. Conversely, in the area of general IS success there are some recognized theories [e.g. 7, 45]. Since explaining success of IS such theories mandatorily present a set of success predictors. As long as such success predictors constitute or at least refer to IS characteristics, these theories can also be used to found design characteristic research. In view of this possibility, in particular the IS success model (ISSM) [6, 7, 40] presents general success relevant IS characteristics and, 168 additionally, is repeatedly validated. Basically, ISSM offers three groups of success predictors, namely, systems quality, information quality and service quality [6, 7]. Visibly, systems quality understood as a set of features which refers to the system as such, and information quality, understood as a set of features which refers to the content of the system, constitute system-related characteristics and hence, are appropriate for design characteristics research. Service quality, however, understood as a set of features which refers to the user support, does visibly not constitute a characteristic of the system itself, and hence is not appropriable. Transferred to VLE design characteristic, the ISSM hence clarifies that system-related (features of the VLE as such) and information-related (learning content of the VLE) constitute essential groups of design characteristics. Being a general theory, ISSM however is not able to provide more detailed information about VLE design characteristics. It is hence the task of the expert study to ascertain systems as information-related design characteristics of VLE empirically. 2.2 Review In order to add to cumulative research and integrate the expert study with previous findings a comprehensive review of previous research in design characteristics of VLE was conducted. The review considered studies which directly deal with design characteristic of VLE in an empirical or conceptual way. Extensive searches of electronic databases (EBSCO, ScienceDirect, and Scopus) as well as of selected journals and conference websites were carried out to identify appropriate studies. However, in order to assure the quality of results only outlets with double blind peer- review were taken into account. To map existing studies comprehensively, a time frame of 20 years (1989-2009) was analyzed. Based on this procedure 25 relevant studies could be identified (these studies are marked with an asterisk in the references section and are summarized in Appendix 1). The analysis of VLE design characteristics identified within these studies yielded several interesting results. Firstly and unexpectedly, a plethora of over thirty different design characteristics could be identified (for details see Appendix 1). Though this may be judged as ample results of previous research, this abundance also represents a certain problem like an increasing number of design characteristics detracted from there applicability and usefulness. Hence, future research should strive for a limited set of major design characteristics rather than amassing a maximum of design characteristics. Secondly, as predicted by the ISSM, all identified design characteristics could by classified as either system- related or information-related, while quite frequently systems quality and information quality were presented as design characteristics. Whereas this constitutes a consent concerning the general design characteristics, there is dissent concerning more concrete design characteristics within these groups. This heterogeneity adds to the problem of the mere number, since it is still unclear which concrete design characteristics actually are relevant for success. Hence, it is necessary to validate design characteristics to attain a set of resilient characteristics. Thirdly, the design characteristics found are of rather different granularities, understood as the grade of operativeness and detailedness of design characteristics. Basically, very general, coarse-granular characteristics such as the mentioned "systems quality" or "information quality" and rather medium-granular characteristics such as "personalization" or "clear terminology" can be differentiated, while fine-granular, detailed, i.e. very specific design characteristics could not be detected. Granularity of design characteristics evidently is of major importance since expressiveness and usability increase with granularity (for instance, ―develop/select/use 169 personalized VLE" constitutes a more expressive and usable statement than ―develop/select/use VLE with good systems quality‖). In view of this, at first glance one may claim maximal granularity from the expert study, however increased specificity commonly is aligned with a decreasing range of validity. Hence, to warrant general validity the expert study may have to get by with a medium granularity. Fourthly, there is a prevalent lack of explicit definitions of design characteristics (while there are some exceptions). Since the design characteristic presented, such as ―perceived flexibility", represent rather complex constructs which can be understood in quite different ways, the lack of definition aggravates the understanding of design characteristics as well as their further usage. It also complicates the detection of possible redundancies of characteristics found in different studies such as ―personalization‖ and ―user adaptation‖. Hence, the expert study mandatorily has to elaborate thorough and explicit definitions of design characteristics. In summary, previous research suggests a set of design characteristics which is copious, of limited congruence, of different granularity, and frequently unclear in meaning. This clearly justifies the necessity of the expert study. However, instead of just adding a further unconnected study, the current state of knowledge is to be used as a base to contrast but also enrich the expert study and thereby integrate it with previous work. 2.3 Method To ascertain success relevant system- as well as information-related characteristics of VLE with an expert study systematically, the Delphi method was considered as promising approach [e.g. 11, 12, 13, 22]. Besides supporting practical forecasting and practical decisions, the Delphi method is also appropriate for systematically analyzing complex and multifaceted scientific topics that are not directly and easily accessible via quantitative research approaches [e.g. 11]. To ascertain relevant design characteristics systematically, a two-phased approach was performed. Phase I was aimed at a general inquiry and categorization of generally imaginable design characteristics. As a starting point of phase I, a group of appropriate experts was to be appointed. Participants were selected based on different criteria, while it was intended to arrange an international group of experts with extensive knowledge in the design of VLE which is of diverse disciplinary provenience (computer science, management, pedagogy, and psychology), and diverse institutional affiliation (universities and private companies). The resulting group consisted of 13 international experts with different affiliation and backgrounds (see Appendix). Subsequently, an online-questionnaire was developed. Beside the provision of a general introduction into the questionnaire, relevant terms such as VLE or design characteristic were thoroughly defined in order to assure a consistent understanding of constructs and questions. In so doing, the questions referred to the creation of a comprehensive list of design characteristics in general, and to subsequently match this list to a categorization of system- and information-related design characteristics of VLE. Balancing the trade-off between specificity and validity it was asked for characteristics which were specific, but generally valid. To avoid mere adjective lists with undefined and hence unclear constructs, experts were explicitly encouraged to explain the stated design characteristics in detail. The questionnaire was pre-tested and slightly modified based on in-depth interviews with two experts. The online survey was carried out in autumn 2009, while all 13 experts participated. A monitoring team of five independent researchers individually evaluated the results obtained in phase I. In particular, based on the construct explanations synonymous 170 design characteristics were identified and adjusted, the adjusted set of design characteristics was summarized respectively aggregated according to the principles of ―summarizing content analysis‖ [31], and finally, general definitions for the summarized design characteristics were derived from the expert explanations. In a subsequent group discussion, individual results of the monitoring team members were mutually adjusted, while there was an initial high degree of inter-coder reliability [31] within the monitoring team anyway. During the preparation of phase II the result list of design characteristics of phase I was compared with the results of the review of previous work. As there was substantial agreement concerning several design characteristics, the result list from phase I missed some of the design characteristics that proved to be significant for success. To be more concrete, experts did not mention ―multimodal‖ [30, 36, 38], ―accessible‖ [30], ―appealing‖ [3, 4, 16, 27, 38], ―reliable‖ [27, 30, 47], ―secure‖ [30] and ―structured‖ [3, 36]. To test also the relevance of these characteristics, they were added to the results of phase I. The resulting list of adjusted, aggregated, enriched, categorized and defined design characteristics constituted the base of the second online-questionnaire. The 13 experts this time were asked to rank the presented system- and information-related design characteristics of VLE from highest (rank 1) to lowest (rank n) priority for success. The resulting priority lists were summarized by calculating means and standard deviations of the respective rank positions. 2.4 Results Interim results of phase I firstly revealed an unadjusted list of 55 design characteristics (31 system-related, 24 information-related). This list was successively reduced by adjustment of synonyms to 31 design characteristic (13 system-related, 16 information- related) and the summarizing of design characteristics to 16 design characteristics (10 system-related, 6 information-related). VLE Design Characteristic A. System-Related Reliable A1. 3.08 (1.44) Secure A2. 4.38 (3.52) Learning-Process-Supportive A3. 4.46 (3.13) Interactive A4. 4.77 (3.11) Appealing A5. 5.08 (2.25) Transparent A6. 5.15 (2.79) Structured A7. 5.92 (2.22) 171 Standard-Supportive A8. 6.46 (2.79) Accessible A9. 6.85 (2.15) Platform-Independent A10. 7.62 (2.90) B. Information-Related Understandable B1. 2.23 (1.48) Consistent B2. 2.92 (1.66) Credible B3. 3.23 (1.30) Challenging B4. 3.54 (1.51) Multimodal B5. 4.00 (1.78) Enjoyable B6. 4.58 (1.44) Table 1: Means and Standard Deviations of VLE Design Characteristics Ranks. As depicted this list was enriched with 7 literature-based characteristics (6 system- related, 1 information-related). Final results are rendered in Table 1 and 2. Table 1 firstly depicts the results of the prioritization process in phase II by presenting the mean values and the standard deviations (in brackets). The derived definitions of these characteristics are presented in Table 2, while each definition is illustrated with selected statements of the literature review and/or experts to make their origin more transparent. Design Definition Source Exemplary Statement Characteristic A. System-Related VLE are reliable, if their end-users/learners can ―Whenever I use the e- literature Reliable apply it without learning tool, it always review technology owed works correctly.‖ [30] disturbances. 172 VLE are secure, if the system itself as well as unauthorized users cannot modify or delete the learners' personal profile literature ―I trust the system security.‖ Secure data, respectively their review [30] learning history, progress (i.e. learning outcomes), and corresponding resources. VLE are learning-process- supportive if they support the provision of (further) learning activities and/or materials with their Learning- inherent information (e.g. expert ―Workflow-management Process- activity description and/or study component‖ Supportive instruction, etc.) according to the learners‘ current status in the unit of learning, and help the learners to coordinate audit dates, group meetings, etc. VLE are interactive if they ―[…] key to the learning allow for learner-system- process are the interactions (e.g. taking self-tests, among students themselves, uploading assignments, literature the interactions between Interactive etc.), learner-learner-, review faculty and students, and the and/or learner-teacher- collaboration in learning communication and/or that results from these collaboration (e.g. via interactions.‖ [34, 36] 173 audio/video conference, ―The core of learning blackboard, chat, forum, remains a relationship expert etc.). between a learner and a study tutor. VLE must keep this crucial factor in the loop.‖ VLE are appealing, if their ―Screen design is the way literature Appealing graphical user interface information is presented on review has a pleasant appearance. the screen.‖ [16, 23, 28] ―The e-learning system VLE are transparent, if literature allows the user to control they allow the learners to review his/her improvement.‖ [30] keep an eye on their own ―The system enables users and/or other learners‘ to trace why and how learning history (i.e. Transparent certain recommendations completed and/or passed expert are made, how much learning activities of a unit study personal data one allows the of learning) and current system to data mine status in the learning implicitly/explicitly to process. produce a user profile.‖ VLE are structured, if learners can quickly detect the allocated information (e.g. learning resources such as learning materials, collaboration services, ―[…] the ease with which literature Structured assessment items, system- users can move around the review generated information such system.‖ [23] as user guidance, feedback, etc.) in, respectively can easily navigate the graphical user interface. 174 VLE are standard- supportive, if they facilitate learning materials which are compiled based on approved eLearning standards such as IMS Standard- expert ―Interoperability and Learning Design [17], or Supportive study standards compliance‖ SCORM [1] as these eLearning standards enable learning materials to be widely shared across VLE which also support these standards. VLE are accessible, if ―The e-learning tool is learners can access it literature Accessible accessible according to my according to their own review own possibilities.‖ [30] possibilities. VLE are platform- Platform- independent, if they run on expert ―VLE should be Web- Independent a wide range of operating study based, not standalone.‖ systems. B. Information-Related The information provided ―Terminology refers to the by VLE is understandable, literature words, sentences, and if the words, sentences, review abbreviations used by a and abbreviations applied system.‖ [23, 28] within the learning Understandable materials are clear in meaning (e.g. by use of expert ―Understandability vs. definitions), easy to study complexity.‖ comprehend and easy to read. 175 The information provided ―The use of terms literature by VLE is consistent, if the throughout the (E-library) is review learning materials consistent.‖ [16] Consistent themselves are without ―Sequencing of learning contradictions, coherent expert objects, tasks, and and presented in a logical study assessments.‖ order. The information provided by VLE is credible, if they ―[…] how much one trust originate from a expert the credibility of the Credible trustworthy source (e.g. study material (i.e. it does not teacher, certified and/or convey wrong concepts)‖ reputable organizations, etc.). The information provided by VLE is challenging, if the learning materials ―For ambitious learners, expert Challenging contain difficult but focusing on learning study interesting tasks which objectives.‖ stimulate learners‘ curiosity to solve them. The information provided ―The Web-based learning by VLE is multimodal, if system offers multimedia the learning materials are literature Multimodal (audio, video, and text) presented in different review types of (course) content.‖ media formats such as text, [36] audio, and video. The information provided ―Positive user experience, by VLE is enjoyable, if the expert associated with pleasure, Enjoyable learning materials study fun, playability, and provided do so in their enjoyment.‖ own right aside from their 176 textual value, and consequently make the learning experience more pleasant. Table 2: Definitions, Sources and Exemplary Statements of VLE Design Characteristics. 2.5 Discussion The present expert study provides a systematic set of well-defined, specific but generally valid system- and information-related design characteristics based on the ISSM and compatible to previous research results. Hence, the general objective of the expert study could be satisfactorily achieved. Findings concerning the system-related design characteristics show that ―reliable‖ was unambiguously rated as the most important system-related design characteristic, followed by ―secure‖, ―learning-process-supportive‖, ―interactive‖, ―appealing‖, ―transparent‖, ―structured‖, ―standard-supportive‖, ―accessible‖, and ―platform- independent‖. It should be noted, ―interactive‖, ―appealing‖, and ―transparent‖ show almost the same mean values, that may be a consequence of the prioritization procedure within phase II as study participants were ―forced‖ to rank the given design characteristics even though they may have preferred similar priorities of different design characteristics. However, the salient disagreement amongst study participants concerning the relative importance of ―secure‖ (SD: 3.52) as the second important system-related design characteristic of VLE in particular might be engendered by its diverse understanding (―[…] the system itself as well as unauthorized users cannot modify or delete the learners' personal profile data […]‖, etc.). The same may count for ―learning-process-supportive‖ (SD: 3.13), ―interactive‖ (SD: 3.11) as well as ―transparent‖ (SD: 2.79) and ―standard-supportive‖ (SD: 2.79). Once again, this result may originate in the way the prioritization procedure was conducted. It is noticeable, amongst the five system-related design characteristics considered to be the most important ones, rank number one (―reliable‖), two (―secure‖), and five (―appealing‖) are design characteristics from the literature review which were added subsequently. This shows that even though study participants did not even mention these system-related design characteristics within the first survey wave, they considered them as highly- relevant system-related design characteristics of VLE. Thus, the prioritization of preceding expert statements and theoretical-founded design characteristics proved to be a feasible and promising approach. Hence, the set of system-related design characteristics presented should always be under consideration when designing, and evaluating VLE. Regarding information-related design characteristics, findings show that ―understandable‖, is considered to be the most important design characteristic, followed by ―consistent‖, ―credible‖, ―challenging‖, multimodal as well as ―enjoyable‖. It should be pointed out that not similar to their system-related counterparts, all information- related design characteristics show high levels of agreement amongst study participants regarding their relevance for VLE (SD spectrum: 1.30 - 1.78). Hence, when designing and evaluating VLE one should consider the set of information-related design characteristics presented. 177 To conclude, the results of the expert study presents a comprehensive set of VLE specific information- and system-related design characteristics, which should be considered when developing, purchasing, implementing or evaluating VLE. 3 Implications The above-mentioned results should generally provide a basic starting point for future research as design endeavors, while there are some implications for research as well as practice. Concerning research implications, firstly, some further theoretical deliberations may improve future research. The used ISSM is able to roughly categorize relevant design characteristics, but however does not allow to deduce directly specific design characteristics. This likely applies to further imaginable theoretical foundations, in particular to the prominent TAM-approach, what could be proved within the frame of the literature review (see e.g. the TAM-based studies of [3, 38]). Again, the basic TAM does rarely directly propose concrete design characteristics. In order to overcome this theoretical gap, more recent theoretical developments that are orientated towards design and intervention (e.g. the TAM 3 offered by [45]) may offer deeper foundations. In addition, also amalgamations of such approaches with the ISSM may be worth of a trial (see the example in [32]). Furthermore, given that expressiveness and usability of design characteristics increase with growing specificity, future research should aim at increasing specificity of design characteristics, however without losing general validity. One imaginable way is to work out different facets of the design characteristics by constituting sub-characteristics. For instance, based on the definitions elaborated certain sub-characteristics of ―flexible‖, ―learning-process-supportive‖ or ―transparent‖ could be established. As an important aspect considered by one previous study [30] and confirmed in the expert study, possible interdependencies of design characteristics should be taken into account. Basically, design characteristics may not be arbitrarily combinable for logical and/or technical reasons [10], hence, future research should also strive for (in-)compabilities of design characteristics found. This also entails a question that has not been tackled till now, whether different system- and information-related design characteristics contribute rather individually and independently to VLE success, or whether whole bundles or entire configurations of design characteristic are triggering success. Moreover, given the benefits of an experimental design, such as controlling relevant while excluding confounding variables, ensuring direct relevant experiences of respondents, and, particularly enabling the manipulation of specific design characteristics [21], experimental designs seem to be a promising approach to ascertain and evaluate relevant design characteristics empirically (cf. the pioneering work of [36] who conducted an offline experiment to determine relevant design characteristics). Finally, given the costs and duration of developing prototypes, and, all the more, full versions of a VLE, it would be highly beneficial if relevant design characteristics could be ascertained as early as possible, in order to avoid misconceptions and failure [5]. Hence, the usage of simple prototypical models (paper prototypes, video mockups, etc.) of the system planned may allow ascertaining relevant characteristics in very early phases of the corresponding software development process [32]. Additionally, the results of the study yield some implications for practice. Managerial and technical decision-makers in the process of developing new, selecting pre-packaged VLE-software, or evaluating and improving already adopted VLE are offered a valuable general (check-)list of criteria relevant for success. Beyond, with a particular view to information-related design characteristics, learning designers and teaching staff may 178 profit from their application while preparing their learning materials. Hereby, information-related design characteristics could also be understood as a checklist in how far their learning materials fulfill the proposed requirements (e.g. understandable, consistent, and credible learning materials). Refining and customizing this (check-)list towards individual corporate settings and subsequently considering the list may lead to practical VLE design- and selection- processes which minimize learner resistance, increase learner satisfaction, and support overall learning success. 4 Conclusions Within this paper a comprehensive literature review and an initial expert study were carried out yielding a systematic list of well-defined system- and information-related design characteristics of VLE. This hopefully will stimulate future research, especially quantitative studies which evaluate and deepen the insights offered, but may also instruct future practical development, selection and evaluation projects, while both streams may finally contribute to improved VLE which support better corporate training and development endeavors. References [1] ADL (2004), SCORM 4th Edition - Version 1.1 (Documentation), available at: http://www.adlnet.gov/Technologies/scorm/SCORMSDocuments/2004%204th%20 Edition/Documentation.aspx, accessed on the 11th of April 2010. [2] *Arbaugh, J. B. (2000). Virtual Classroom Characteristics and Student Satisfaction with Internet-based MBA Courses, Journal of Management Education, 24 (1), 32- 54. [3] *Chang, S.-C. & Tung, F.-C. (2008). An Empirical Investigation of Students‘ Behavioural Intentions to Use the Online Learning Course Websites, British Journal of Educational Technology, 39 (1), 71–83. [4] *Chiu, C.-M., Hsu, M.-H., Sun, S.-Y., Lin, T.-C. & Sun, P.-C. (2005). Usability, Quality, Value and E-Learning Continuance Decisions, Computers & Education, 45, 399-416. [5] Davis, F. D. & Venkatesh, V. (2004). Toward Preprototype User Acceptance Testing of New Information Systems: Implications for Software Project Management, IEEE Transactions on Engineering Management, 51 (1), 31-46. [6] DeLone, W. H. & McLean, E. R. (1992). Information Systems Success: The Quest for the Dependent Variable, Information Systems Research, 3 (1), 60-95. [7] DeLone, W. H. & McLean, E. R. (2003). The DeLone and McLean Model of Information Systems Success: A Ten-Year Update, Journal of Management Information Systems, 19 (4), 9–30. [8] Fry, H., Ketteridge, S. & Marshall, S. (2009): A Handbook for Teaching and Learning in Higher Education: Enhancing Academic Practice, 3rd edition. Routledge. 179 [9] *Fu, F.-L., Chou, H.-G. & Yu, S.-C. (2007). Activate Interaction Relationships between Students Acceptance Behavior and E-Learning, in G. Dong, X. Lin, W. Wang, Y. Yang, & J. Xu Yu (eds.): Joint 9th Asia-Pacific Web Conference (APWeb 2007) and 8th International Conference on Web-Age Information Management (WAIM 2007), LNCS 4505, 670–677. [10] Galletta, D. F. & Lederer, A. L. (1989). Some Cautions on the Measurement of User Information Satisfaction, Decision Sciences, 20 (3), 419–439. [11] Grisham, T. (2009). The Delphi Technique: A Method for Testing Complex and Multifaceted Topics, International Journal of Managing Projects in Business, 2 (1), 112-130. [12] Haeder, M. (2002): Delphi-Befragungen: Ein Arbeitsbuch. VS Verlag für Sozialwissenschaften. [13] Haeder, M. & Haeder, S. (2000): Die Delphitechnik in den Sozialwissenschaften: Mehtodische Forschungen und innovative Anregungen. Westdeutscher Verlag. [14] Hevner, A. R., March, S. T., Park, J. & Ram, S. (2004). Design Science in Information Systems Research, MIS Quarterly, 28 (1), 75–105. [15] *Holsapple, C. W. & Lee-Post, A. (2006). Defining, Assessing, and Promoting E- Learning Success: An Information Systems Perspective, Decision Sciences Journal of Innovative Education, 4 (1), 67-85. [16] *Hong, W., Thong, J. Y. L., Wong, W.-M. & TAM, K.-Y. (2001-2002). Determinants of User Acceptance of Digital Libraries: An Empirical Examination of Individual Differences and System Characteristics, Journal of Management Information Systems, 18 (3), 97-124. [17] IMS (2003), ‗IMS Learning Design Information Model - Version 1.0 (Final Specification)‘, available at: http://www.imsglobal.org/learningdesign/ldv1p0/ imsld_infov1p0.html, accessed on the 11th of April 2010. [18] ISO/IEC (2005), ‗ISO/IEC 25000: Software Product Quality Requirements and Evaluation (SQuaRE)‘, available at: http://www.iso.org/iso/iso_catalogue/ catalogue_tc/catalogue_detail.htm?csnumber=35683, accessed on the 11th of April 2010. [19] Johnson, R. D., Gueutal, H. & Falbe, C. M. (2009). Technolgy, Trainees, Metacognitive Activity and E-Learning Effectiveness, Journal of Managerial Psychology, 24 (6), 545—566. [20] Kiser, K. (1999). 10 Things We Know so far about Online Training, Training, 36 (11), 66-74. [21] Koenigstorfer, J. (2008): Akzeptanz von technologischen Innovationen: Nutzungsentscheidungen von Konsumenten dargestellt am Beispiel von mobilen Internetdiensten. Gabler. [22] Landeta, J. (2006). Current Validity of the Delphi Method in Social Sciences, Technological Forecasting and Social Change, 73 (5), 467-482. [23] *Lee, G. T., Dahlan, N., Ramayah, T., Karia, N. & Hasmi Abu Hassan Asaari, M. (2005). Impact of Interface Characteristics on Digital Libraries Usage, Malaysian Online Journal of Instructional Technology, 2 (1), 1-9. 180 [24] *Lee, Y.-C. (2006). An Empirical Investigation into Factors Influencing the Adoption of an E-Learning System, Online Information Review, 30 (5), 517-541. [25] *Liaw, S.-S., Chang, W.-C., Hung, W.-H. & Huang, H.-M. (2006). Attitudes toward Search Engines as a Learning Assisted Tool: Approach of Liaw and Huang‘s Research Model, Computers in Human Behavior, 22, 177–190. [26] *Liaw, S.-S. & Huang, H.-M. (2003). An Investigation of User Attitudes toward Search Engines as an Information Retrieval Tool, Computers in Human Behavior, 19, 751–765. [27] *Lin, H.-F. (2007). Measuring Online Learning Systems Success: Applying the Updated DeLone and McLean Model, Cyber Psychology & Behavior, 10 (6), 817- 820. [28] Lindgaard, G. (1994): Usability Testing and System Evaluation: A Guide for Designing Useful Computer Systems. Chapman & Hall. [29] *Liu, S.-H., Liao, H.-L. & Peng, C.-J. (2005). Applying the Technology Acceptance Model and Flow Theory to Online E-Learning Users‘ Acceptance Behavior, Issues in Information Systems, 6 (2), 175-181. [30] *Martínez-Torres, M. R., Toral Marín, S. L., Barrero Garciá, F., Gallardo Váquez, S., Arias Oliva, M. & Torres, T. (2008). A Technological Acceptance of E- Learning Tools Used in Practical and Laboratory Teaching, according to the European Higher Education Area, Behaviour & Information Technology, 27 (6), 495-505. [31] Mayring, P. (2003): Qualitative Inhaltsanalyse: Grundlagen und Techniken, Vol. 8. Beltz. [32] *Mueller, D. & Zimmermann, V. (2009). A Learner-Centered Design, Implementation, and Evaluation Approach of Learning Environments to Foster Acceptance, International Journal of Advanced Corporate Learning, 2 (3), 50-57. [33] *Nov, O. & Ye, C. (2008). Users‘ Personality and Perceived Ease of Use of Digital Libraries: The Case for Resistance to Change, Journal of the America Society for Information Science and Technology, 59 (5), 845-851. [34] Palloff, R. M. & Pratt, K. (1999): Building Learning Communities in Cyberspace: Effective Strategies for the Online Classroom. Jossey-Bass. [35] Peffers, K., Tuunanen, T., Rothenberger, M.A. & Chatterjee, S. (2007). A Design Science Research Methodology for Information Systems Research, Journal of Management Information Systems, 24 (3), 45–77. [36] *Pituch, K. A. & Lee, Y.-K. (2006). The Influence of System Characteristics on E- Learning Use, Computers & Education, 47, 222-244. [37] *Poelmans, S., Wessa, P., Mills, K., Bloemen, E. & Doom, C. (2008). Usability and Acceptance of E-Learning in Statistics Education, based on the Compendium Platform, International Conference of Education, Research and Innovation (ICERI 2008), 1-10. [38] *Roca, J. C., Chiu, C.-M. & Martínez, F. J. (2006). Understanding E-Learning Continuance Intention: An Extension of the Technology Acceptance Model, International Journal of Human-Computer Studies, 64, 683-696. 181 [39] *Sahin, I. & Shelley, M. (2008). Considering Students‘ Perceptions: The Distance Education Student Satisfaction Model, Educational Technology & Society, 11 (3), 216–223. [40] Seddon, P. B. (1997). A Respecification and Extension of the DeLone and McLean Model of IS Success‘, Information Systems Research, 8 (3), 240–253. [41] Sitzmann, T., Kraiger, K., Stewart, D. & Wisher, R. (2006). The Comparative Effectiveness of Web-based and Classroom Instruction: A Meta-Analysis, Personnel Psychology, 59, 623-64. [42] Strohmeier, S. (2008): Informationssysteme im Personalmanagement: Architektur – Funktionalität – Anwendung. Vieweg+Teubner. [43] *Sun, P.-C., Tsai, R. J., Finger, G., Chen, Y.-Y. & Yeh, D. (2008). What Drives a Successful E-Learning? An Empirical Investigation of the Critical Factors Influencing Learner Satisfaction‘, Computers & Education, 50, 1183-1202. [44] *Tobing, V., Hamzah, M., Sura, S. & Amin, H. (2008). Assessing the Acceptability of Adaptive E-Learning System, 5th International Conference on eLearning for Knowledge-Based Society, 1-10. [45] Venkatesh, V. & Bala, H. (2008). Technology Acceptance Model 3 and a Research Agenda on Interventions, Decision Sciences, 39 (2), 273-315. [46] *Wang, W.-T. & Wang, C.-C. (2009). An Empirical Study of Instructor Adoption of Web-based Learning Systems, Computers & Education, 53, 761-774. [47] *Wang, Y.-S. (2003). Assessment of Learner Satisfaction with Asynchronous Electronic Learning Systems, Information & Management, 41, 75–86. [48] *Wang, Y.-S., Wang, H.-Y. & Shee, D. Y. (2007). Measuring E-Learning Systems Success in an Organizational Context: Scale Development and Validation, Computers in Human Behavior, 23, 1792–1808. [49] Weller, M. (2007): Virtual Learning Environments: Using, Choosing and Developing Your VLE. Routledge. [50] *Yeung, P. & Jordan, E. (2007). The Continued Usage of Business E-Learning Courses in Hong Kong Corporations, Education and Information Technologies, 12 (3), 175–188. 182 Appendices: 1) Literature Review – Results Overview Study Theory Design Characteristic Construct Method Definition System-Related TAM ISSM … Information-Related 1. [26] System Quality Offline X X - Survey Perceived Usability, 2. [4] Perceived System Quality) Survey with X X Application Information Quality 3. [25] System Quality Offline X X - Survey System Quality Online 4. [38] X X X X Survey with Information Quality Application - Online 5. [24] Perceived X Survey with Content Quality, Application Course Attributes System Quality Survey with 6. [15] X Prototype and Information Quality Application System Quality Offline 7. [27] X X Survey with Information Quality Application System Quality Offline 8. [50] X X X Survey with Information Quality Application System Quality Evaluation of 9. [48] an Information Quality Applicatoin System Quality Offline 10. [37] X X X Survey with Information Quality Application 11. [3] Perceived System Quality Survey with X X - Application System Quality 12. [32] X X Conceptual Information Quality 183 13. [46] System Quality X X X Online Survey Information Quality System Adaptability Offline 14. [44] Experiment X X - with Application Perceived Course Offline and 15. [2] Interaction, Online Survey X Perceived Flexibility with (time, location, methods) Application Screen Design Online 16. [16] X Relevance, X Interview with Application Terminology Learner Interface Learning Community Interview and 17. [47] X Survey with Personalization Application Content - 18. [29] Survey with X X eLearning Materials Application Presentation Types: 1. Text-Audio, 2. Audio-Video, 3. Text-Audio-Video Screen Design, Evaluation of 19. [23] X Navigation X an Application Terminology System Functionality, Offline System Interactivity, 20. [36] Experiment X x System Response with Application - Functionality, Online Survey 21. [9] X Interface Design with Application - e-Learning Course Flexibility Online Survey 22. [43] (time, location, methods) X X X with Course Quality Application 184 Accessibility, Communicativeness, Feedback, Interactivity and Control, 23. [30] Reliability, Survey with X User adaptation, Application User tools Format Flexibility Online 24. [39] X Survey with - Application Screen Design Offline 25. [33] X Survey with Relevance Application 2) Expert List Name Affiliation Background Anh Vu, N.-N. University of Leicester, UK Computer Scientist Christina, H. IMC, Germany Pedagogue Dominique, V. OUNL, the Netherlands Pedagogue Effie L. University of Leicester, UK Psychologist Elisabetta, P. Giunti Labs, Italy Computer Scientist Jad, N. WUW, Austria Computer Scientist Kai, H. TU Darmstadt, Germany Computer Scientist Luis, de la F. UC3M, Spain Computer Scientist Marvin, S. DFKI, Germany Computer Scientist Milos, K. OUNL, the Netherlands Computer Scientist Patrick, P. IMC, Germany Computer Scientist Susanne, N. University of Vienna, Austria Pedagogue Volker, Z. IMC, Germany Management, and Business Informatics Specialist 185