Two different approaches can increase the accessibility of digital educational materials: content that has been built with the widest possible set of users in mind (universal design), or content that has been designed in such a way that it can be personalised to individual user needs and preferences (personalised design). This paper outlines a number of approaches that could be used to evaluate the provision of learning materials that have been adapted to or chosen for individual learners. A number of different perspectives are considered in this paper: a learner's perspective, the perspective of the tutor or teacher, and an institutional perspective. A number of complementary methodologies are presented. It is argued that the evaluation of a system that provides personalised learning content is a challenging activity that necessitates the application of multiple methods to effectively understand the underlying costs and benefits of providing personalised learning materials.
The emergence of tools such as virtual learning environments has enabled institutions to create digital services that can be used to augment face-to-face teaching. Learners can increasingly access a wealth of digital learning resources that can either help students become familiar with the subjects that are going to be discussed before a lecture or class, or allow learners to consolidate concepts that were taught during a lecture.
The development of these new technologies can, to a varying degree, be considered to be especially beneficial to learners with disabilities. Those students who are unable to attend a class may be able to use a virtual learning environment to make a contribution by participating within on-line activities. The accessibility of learning technologies ultimately depends upon the accessibility of the tools that are used to present learning materials (such as a VLE) as well as the learning material it contains.
Digital learning materials (or content) can be presented in a multitude of different formats. Digital content can be in the form of simple web pages, audio pod casts, fragments of video, or even interactive demonstrations such as simulations. The choice of what format to use may depend upon a wide range of factors, including the learning objectives that the educator aims to convey the availability of appropriate digital resources from a third party, the amount of time that an educator or learning technologist could spend creating those resources, and levels of internal expertise.
Although a range of digital resources can be created and presented through a virtual learning environment, there is a risk that some of the content may be inaccessible for certain groups. A visually impaired user may not be able to benefit from the provision of a video resource if it does not contain additional audio descriptions or other complementary materials. Similarly, a learner who has an auditory impairment may not be able to take advantage of a pod cast if the pod cast does not have an accompanying transcript.
The accessibility of learning material for learners who have disabilities rests upon representatives from an educational institution making practical decisions to ensure that no students are disadvantaged in terms of either being able to ‘access’ the materials through a preferred modality, or being disadvantaged through the adoption or use of learning objectives that makes their academic achievement difficult or even impossible.
The next section of this paper introduces the notion of content personalisation and adaptation as a way to enhance the accessibility of digital learning resources. This is contrasted with the competing notion of universal design: the ideal that all products, resources or systems should be accessible and usable by all people.
Within an educational institution, many different people have a collective responsibility regarding the provision of accessible learning. A number of groups or people who are acknowledged to have a responsibility are highlighted in the stakeholders section.
The focus of this paper lies with understanding how to evaluate whether it may be
possible to deploy or develop a successful content personalisation approach to deliver
the best possible educational experience for the widest possible group of learners. The
EU4ALL project aims to build a practical framework to demonstrate the operation and
potential benefits of content personalisation and other processes that can enhance the
provision of accessibility [
There are two fundamental approaches for the development of accessible resources. The first is the development of a resource that is universal, i.e. a digital resource that all learners can use, regardless of their disability or sensory impairment. A video may be designed in such a way so that all the themes and principles it presents can be explained through the audio track with any accompanying visual descriptions merely emphasising the points that are being made. For the video to be accessible for people with hearing impairments, a set of subtitles may simultaneously be presented. This means, that the learning resource could be useful to people who have either visual or auditory impairments. In this way, such a video may be considered a product of universal design.
One argument against universal design is that learning resources that can be used by all people may not be optimal for everyone: each learner may have their own precise learning needs and requirements since each learner may have a unique combination of skills and disabilities. An alternative to the ideal of a universal resource is the notion of personalised or adaptable resources. A digital resource could be designed in such a way so it could be adapted (or customised) to match the needs and preferences of individuals. Some learners may prefer to listen to spoken versions of learning content due to a learning style preference. Some learners, on the other hand, may require subtitles of a particular size, colour, font or speed. Other learners may find that a transcript of a video might be more useful, since it can be more readily edited or manipulated.
The EU4ALL and TILE [
EU4ALL project has created a framework that can facilitate discussion about how the accessibility of virtual learning environments and e-learning systems in general might be improved, developed and enhanced. The EU4ALL framework can also be practically implemented through its proposed adoption of a service-oriented design and use of a new generation of learning technology standards. As a result, the framework has the potential to be used by a range of different VLE systems that can eventually deliver personalised e-learning to different users.
The EU4ALL framework comprises of a number of distinct components. The VLE system is used to store digital learning resources. Information about the accessibility of the resources is held within a component known as the Metadata Repository (MR). The accessibility information is represented in terms of whether a particular resource is appropriate for a particular modality. Information about the user is held within the User Model (UM). The choice of the content that is appropriate to the user is performed by the content personalisation module (CP). The EU4ALL framework offers other components that are discussed elsewhere, a Device Model (DM), a Recommending System (RS) and an E-Services Server (ESS) which provides accessibility provision administrative support in the form of workflow services.
It should be stated that there are some parallels that can be drawn between
EU4ALL and the ADAPT2 architecture [
The learner interface to the system is facilitated through the link to the Virtual Learning Environment (VLE). Other external information systems can be connected to the EU4ALL framework, hence the link to an external ‘Staff Intranet’, where administrative personnel can gain access to other components, such as performance statistics that are produced by the ESS.
Different components of the EU4ALL framework have been implemented by different project partners. The User Modeling (UM) and Recommending System (RS) has been implemented by the aDeNu (Adaptive Dynamic online Educational system based oN User modeling) research group which are situated at the Universidad Nacional de Educación a Distancia (UNED) of Spain. The metadata repository has been developed by a commercial partner, ATOS Origin, and the content personalisation component has been implemented by an organisation called Indra. A substantial challenge lies with ensuring not only that all the individual components work together, but also to ensure that the end result from the entire system is of benefit to learners and different institutional stakeholders.
The use, availability and presentation of accessible digital resources requires cooperation between different stakeholders. Whilst a single teacher or educator may be able to upload their own notes or presentations to a VLE which may be accessible in their own right, the availability and accessibility of the VLE is dependent upon a number of other people. The following table describes a number of important stakeholders who guide the development and provision of accessible e-learning. The names of the stakeholders are designed to be ‘high level’ groupings that can be used to guide discussion about the various roles and responsibilities. The notion of an academic manager, for example, can be represented by either a dean (a head of a faculty), or a head of a department.
Provides tutor, lecturer and other technical staff guidance about how to best make use of different formats and learning technologies. Can have training responsibilities to ensure that staff are familiar with the operation of new technologies. System Developer System Administrator Disabilities Advisor Academic Manager Principle
A software specialist who is responsible for developing and enhancing the operation of one or more learning technologies. The developer needs to have an awareness of the importance of accessibility and is likely to liaise with the learning technologist and system administrator.
Responsible for deploying and ensuring continual operation of learning technologies that have been selected by the learning technologist and/or lecturing staff. Liaises with other technology personnel to ensure continual network operation and service. Also provides security support and backup services.
Offers institutional guidance to individual learners. In some institutions this stakeholder may be split amongst a number of roles, including needs assessments and sourcing of appropriate assistive technologies that are to be supplied to tutors.
Line manager for individual lecturers or tutors. In terms of lecturers, the academic manager may be a head of a department or a dean, for example.
For tutors (within the Open University), the line manager would be called a staff tutor. The academic manager may need to be aware of support issues and be able to make available resources to facilitate the provision of accessible learning.
Individual or group that is responsible for the operation of an institution.
Principle has responsibilities for adhering to national and international legislation and developing organisational structures that permit the delivery of accessible learning.
This table is by no means complete and its precise constituents will vary depending upon the differences between institutions. All of these stakeholders have a role to play regarding the delivery of accessible learning experience.
Given a similar list of stakeholders, a substantial question that should be asked is: will these stakeholders accept the development or the delivery of a system that presents learning materials that are personalised to the needs of individual learners? The following section aims to consider what evaluation methodologies could be used to uncover what issues or barriers may prevent the acceptance or development of the a personalised approach to accessibility.
To understand the complexity of introducing a system that personalises learning material requires the application of a number of different evaluation methods. The choice of the method depends upon the question that is to be assessed, and this varies between the perspectives held by each of the stakeholders. A learner will hold a different perspective than the tutor, or the principle, for example.
This section presents a brief description of the different methods that could be used to evaluate the EU4ALL framework (outlined in figure 1) and its content personalisation functionality.
A software inspection, also known as a code review, is where the internals of a software system are shared between a number of different developers who then debate its internal design and quality, drawing upon prior experience of other systems. Issues discovered as a result of applying this method are likely to be discussed and any design flaws may be either addressed or recorded for further consideration. It is possible that this approach could be modified to facilitate the inspection or internal evaluation of an accessible adaptable resource.
Automated checking tools can be used to provide guidance about whether digital
resources conform to a number of well known accessibility guidance. Tools, such as
the Imergo web compliance manager [
A heuristic evaluation is a recognised human-computer interaction technique that aims
to quickly identify usability problems with an interactive device by asking usability
experts to assess an interface using a number of known usability principles [
Predictive evaluations represent a range of techniques that are designed to predict the
performance or the effect of a design change. A well known predictive technique is
called GOMS [
A difficulty with predictive evaluation is that they do not take into account differences between users. A difficulty of heuristic evaluations is that experts may fail to find all usability problems, or alternatively find usability problems that do not exist (known as a false positive). Evaluating a system with real users allows researchers to gain an indepth understanding of real difficulties that are faced, given a particular task or activity. End-user evaluations are often carried out after a series of heuristic evaluations have been completed since they are considered to be both expensive and time consuming. This said, end user evaluations are considered essential to uncover accessibility problems.
Whilst end-user evaluations are usually carried out within a laboratory, field
evaluations are carried out in the situation where a product or system is likely to be
used [
Pedagogic evaluations aim to assess whether a system (or a pedagogic practice) can
facilitate learning. Assessment of learning is usually carried out through a test or task
that has been designed to explore whether certain principles or knowledge has been
retained. The effectiveness or performance of a learning tool, system or practice may
be demonstrated by offering pre and post ‘learning task’ tests to participants. Control
experiments can be used to explore the effect or power of a particular system. Whilst
test scores can provide a quantitative assessment of learning, a complementary
qualitative approach is to explore the attitudes or perceptions learners hold regarding a
system. Learners could be asked if they felt that one system was better than another in
terms of usability or accessibility. More detailed questions, presented in terms of
usability and user experience goals, [
A substantial evaluation consideration should be whether or not a new system is likely to be cost effective either in terms of how much time a system takes to operate, or how much money it could cost to implement and maintain. These financial dimensions can be implicitly seen within some of the other evaluation approaches. The predictive evaluation method aims to proportion time against elements of an interface. The act of conducting a user evaluation (with either the tutor or a learner) may indicate clearly that certain tasks may be difficult to understand. In yielding such a response, it may be possible to conclude that a system may be costly to use, and increase the risk of it not being used or accepted. A thorough economic evaluation in terms of whether any new system can be connected to an existing information technology infrastructure is also necessary. If, from a maintenance perspective, rework or redevelopment of existing systems is necessary or the purchasing (and operating) of additional hardware, the management personnel within an organisation need a clear picture about its underlying costs and benefits.
It is important to take into account the attitudes that the various stakeholders may hold towards the proposed system since this is likely to influence whether it is likely to be accepted, regardless of whether or not the system is likely to improve the learner experience. Perception evaluation, as it is called here, can be carried out by carrying out a series of stakeholder interviews. If the new system requires the mobilization of additional resources to ensure that the system can be effectively deployed, this technique will help the attitudes relating to such issues to be explored. It may be possible to mitigate against challenging attitudes by presentation of end user experiences, current legislation and potential benefits to the institution as well as to end users.
The comprehensive evaluation of a new system (or framework) like EU4ALL that aims to deliver learning materials and services that are customised to an individual’s needs and preferences is a considerably challenging task. Not only are there a number of different methods and approaches that could be used, there are also a wide and varied number of stakeholders whose views must all be taken into consideration.
The end users perspective is essential when it comes to understanding the difficulties that learners face when interacting with a system. The difficulty of using a system is likely to go hand in hand with the task that a system is used for. Whilst consuming content that has been personalised for a learner may be an activity that could be straightforward, there are significant challenges in understanding how lecturers and tutors might be encouraged to create materials that can be ‘personalised’ when the task of a lecturer is not to create personalised content, but to lecture or to teach. As a result, any system that allows personalised content to be authored must be as easy to use as possible. One of the challenges of evaluation is to find an approach that enables some of the key difficulties to be identified. When issues are identified, their usability or accessibility could be further developed, or tasks could be reduced in complexity.
Another key issue that must be remembered when conducting an evaluation of any system is the issue of ethics. This issue is particularly significant when we begin to consider the issue of pedagogic evaluations. Consider the example of constructing an experiment where there are two systems: one system that provides learning materials that are universally accessible, another system that may be personalised to an individual’s needs and preferences. If both systems were deployed in a formal educational setting, it may be possible to argue that one student may have an advantage over the other if the system that is the subject of the evaluation is considered to be ultimately successful. The key, of course, is to always ensure that the participant is always considered to be the most important element of an evaluation. The technology that may be the ultimate focus of a series of studies should always be of secondary importance. It should be unambiguously stated that any evaluation should not be connected with a formal course or qualification. Should a pedagogic evaluation be considered to be required, it may be necessary to create a set of learning materials that are representative of those that may be found within a real course. One of the challenges is to create materials that are interesting enough to persuade participants to become involved with the evaluation.
A further challenge inherent when performing the evaluation of any recommendation or content personalisation system is to assess how the system may be deployed or used within an existing system. To build a complete picture and to learn what must be done in order for real institutions to adopt the proposed approach, consultation with administrative and managerial stakeholders will be necessary. Since the structures of organisations differ, it is suggested that stakeholders from a number of different institutions are consulted. One approach to efficiently gather information relating to the subject of deployment is to carry out a series of focus groups or workshops to assess the political, economic, social and technological (PEST) barriers for acceptance.
The process of evaluation is likely to point towards the ways in which the framework may be applied within a large organisations, such as the Open University. From one perspective, EU4ALL can be seen as technical framework that can guide the practical implementation of new learning technologies and enhancements to existing VLE systems. From a different perspective, it can be used as a tool to uncover the way that information technology can enhance the provision of services to people with disabilities.
To summarise, a multi-method approach is necessary to comprehensively evaluate the acceptance of a new accessibility framework that contains a content personalisation component. It is necessary to evaluate the perspectives of both the learner and those who are responsible for creating new adaptable (or customisable) digital resources. It is also necessary to assess the extent to which a system may be integrated with and connected to an existing infrastructure. A number of different evaluation approaches can be drawn upon. The following practical activities are suggested: 1. 2. 3. 4.
Conducting heuristic evaluations of initial interfaces to assess the efficiency of proposed interfaces.
Complement heuristic evaluations with automated testing of digital resources, taking account of end user profiles.
Complement heuristic evaluations with a series of usability tests.
Design a series of qualitative pedagogic evaluations and liaise with internal pedagogic evaluation experts to assess the effectiveness of their design. Carry out a series of workshops for senior stakeholders that aim to uncover the complexities inherent in supporting the delivery of accessible material and services.
By conducting a combination of these approaches, it is hoped that a rich understanding of the complexities inherent in developing and deploying a framework that contains a content personalisation system will emerge. The resulting lessons can then be used to offer feedback into further designs and be used to inform how most effectively offer mechanisms that can support the provision of accessibility services to further and higher education institutions through the application of information technology.