OpenCourseWare platforms for educational resources have the potential to open new horizons for knowledge sharing and e-learning by reaching learners beyond the constraints of traditional learning systems. SlideWiki is a crowd-sourcing platform that aims to rethink the creation and sharing of knowledge by providing an environment where authors can collaborate, reuse, adapt and share slide contents for educational purposes. As an OpenCourseWare platform, SlideWiki intends to make Open Educational Resources more accessible for all users, including those with disabilities, within formal and informal learning settings. Moreover, the platform offers collaborative tools that enable authors and contributors to translate the slide content. To address the implementation, scalability, usability, and adoption of the platform, it has been designed and deployed in many different learning settings with large-scale trials across Europe. At the time of writing, 56 trials have taken place in different geographical and cultural regions, organizational units, and institutions, covering various teaching and learning scenarios. The experiences and feedback from the trials have influenced the redesign of SlideWiki in terms of accessibility and openness. This paper discusses the findings of the large-scale trials and how they influenced the technical redesign of the platform. It also shows how incorporating user feedback into the technical development process can improve accessibility and collaboration.
A major obstacle to increase the efficiency, effectiveness and quality of education in Europe and developing countries is the lack of widely available, accessible, multilingual, timely, engaging and high-quality educational material which can be adapted to suit the needs of local educators and learners. OpenCourseWare (OCW) systems provide educational materials in an openly licensed, web-based platform that hosts Open Educational Resources (OER). The OER are also openly licensed in a manner that provides users with free and perpetual permission with rights to Retain, Reuse, Revise, Remix and Redistribute content, known as the 5R principles of OER.7.
Improving the availability and adoption of OER is seen as an important step in the
Education 2030 plan of the UNESCO [
Research by Jung et al. [
SlideWiki is an open and accessible OpenCourseWare authoring platform that aims to foster the creation and sharing of qualitative, rich and engaging educational content following the 5R principles of OER. The platform allows educators to create, edit, translate and reuse HTML-based slide presentations complemented with comments, links to sources and supporting materials as well as questions to help learners. SlideWiki uses an open-source code base9 to encourage others to contribute to the project as well as contributing back to the open source community.
The initial version of the platform (SlideWiki 1.0) was first launched in 2012 [
In 2016, an EU H2020 grant was awarded to an enhanced SlideWiki team, to fund
the redevelopment and trial of SlideWiki 2.0 as a collaborative OpenCourseWare
authoring platform. This provided an opportunity to recreate the SlideWiki platform as a
7 http://opencontent.org/definition/
8 https://www.merlot.org
9 https://slidewiki.github.io
user-centered project as well as develop, collate, and trial open educational resources
covering a wide range of topics and educational levels to boost adoption of the
platform. It was important to address the critical factors identified by Vahdati et al. [
By adopting a user-centered design approach where “development proceeds with
the user as the center of focus” [
ties to support children and adults with physical, sensory and cognitive disabilities, lifelong learning or vocational training.
In this paper, we discuss how these strategies, particularly the large-scale trials, have contributed to the re-development of SlideWiki, and whether these approaches can enrich the development of open learning technologies. The remainder of this paper is organized into four sections: Section 3 introduces the SlideWiki platform, Section 4 describes the large-scale trials and how they contribute to the project and Section 5 presents results from the evaluations of the platform by trail participants. Finally, Section 6 discusses the impact of the large-scale trials on the SlideWiki project and draws conclusions that may aid future and related projects. 3
The SlideWiki platform uses the format of slide decks to represent OER, as slide presentations provide a comprehensive means for demonstrating knowledge in a short, concise, and illustrative form. Slides are grouped together into a deck that represents an OER. Authors can import existing slide presentations provided in PowerPoint or Open presentation formats. They can also attach slides from decks they or other authors have created.
The features of the platform are targeting three types of users: (1) authors who create and edit content; (2) educators who reuse and remix content; and (3) learners who consume and interact with content. Figure 1 illustrates the main features of the platform. Decks can be viewed by any user without signing-in; users can navigate between slides (as illustrated in Figure 2), or they can display slides in presentation mode, which can also be made available for offline usage. Users can like and share decks on social media, and download slides as PDF or other formats.
The SlideWiki platform uses a component-based architecture for implementing its functional features. The following list explains the main components of the system: – Slide authoring interface. SlideWiki employs an inline HTML5-based WYSIWYG (What-You-See-Is-What-You-Get) editor for authoring the slides. This is accessible to assistive technology users and creates accessibility-compliant content. – Change management controls. SlideWiki supports versioning of slides and decks to ensure that every authors personal revisions of slides and decks are always preserved and they can track the history and changes of the content. Decks can also be “forked”. This allows authors to create their own copy of a deck to enable repurposing. – Search and browsing interfaces. All the content on SlideWiki, such as decks, slides and users, is indexed to provide efficient search and browsing. Decks can also be grouped using tags and topics. – Social interaction. SlideWiki supports social activities where users can share or comment on slides/decks. Users can receive recommendations of content that may be of interest based on their history and preferences. – Import and export of slide decks. Slide decks can be imported from PowerPoint and OpenOffice formats and exported to a number of formats, including PDF, HTML, SCORM and ePub3. – Self-Assessment tools?. Authors can add questions and quizzes to decks in order to encourage learners to interact with the learning content to a greater depth. – Translation of educational resources?. Authors can translate existing decks into other languages. – Linked Data interface?. The platform provides an RDF-based version of the slide content supported by a Linked Data interface which enables accessing and querying data in a machine-readable format. – Licensing. All content is licensed under the Creative Commons CC-BY-SA license by default. Through the change management controls, contributions and authors are tracked to allow for accurate attribution.
Enhancing the accessibility and inclusiveness of the SlideWiki platform was one
of the main objectives of the project. Therefore, accessibility considerations have been
included at all stages of design and development. The following design decisions were
made:
1. Selecting code libraries by the extend to which they were already created with
accessibility in mind whiles ensuring that all new features were regularly checked
against the Web Accessibility Content Guidelines (WCAG) 2.1 [
The project organized large-scale trials in four educational sectors: (1) professional and vocational training, (2) higher education, (3) secondary education, and (4) open community education. Forty internal three-year trials were organized by 16 project partners and a further 16 external trials have been commissioned for the final year of the project.
The internal trials were planned in order to increase the number of slides and the audience incrementally during the three-year project. Initially, the trials started with a limited number of authors who were usually directly involved in the project before widening out the trials to include a larger set of authors, educators and learners across their organizations and professional networks. In order to establish the trial domains, identify possible gaps, and create clusters of trials, a categorization of the internal trials was undertaken. This was performed during a collaborative session with trial leaders, which resulted in 14 dimensions for the categorization of the trials as shown in Table 1. ? These features were still under development and were not available to the trials.
During the second year of the trials, leaders were asked to analyze the activities they had undertaken based on this categorization. It was found that 87% of the internal trials were crowd-sourcing learning content and reusing existing materials, but at the time only 56% of the trials involved participants of the SlideWiki platform. This categorization has helped focus the priorities for the external trials.
The development team aimed to release updates to the SlideWiki platform every 4-6 weeks using agile development techniques, so there were ample opportunities for feedback from the trials to influence the development of the platform. This included commenting on the design, requirements and priorities as well as highlighting bugs and problems. To support this, the project put in place a variety of formal and informal feedback mechanisms to facilitate communication between the developers and trials: Informal Feedback Mechanisms A feedback button was added to the SlideWiki platform and trial partners were encouraged to provide information on their experiences when using the platform. These messages went directly into the development team’s ticketing system, and could therefore be incorporated immediately into planning design sessions.
In addition, trial leaders received a monthly online demonstration of new features
with the opportunity to discuss these with the development team. This process was
supplemented by reports, focus groups and further meetings with trial leaders, educators
and learners. Through these channels their expectations, experiences and requirements
when using the SlideWiki platform were discussed with a wider audience.
Formal Feedback Mechanisms In order to gather regular, structured evaluation on the
progress of the trials and the platform to fulfill the project’s goals, an online survey was
distributed. This anonymous survey consisted of 23 questions. The survey was approved
by the ethics committee of the university that oversaw the gathering of the feedback, and
participants had the opportunity for informed consent prior to completing the survey.
The survey was made available online in English and where require, translated into
local languages. It included:
– five background questions, including prior experience of SlideWiki and web
applications.
– eight questions related to experiences of using SlideWiki. This section asked how
participants used the platform and also included three quantitative questions to
evaluate the effectiveness of undertaking tasks on SlideWiki (based on questions from
the NASA task load index [
Trial leaders were encouraged to request learners to complete the survey once they had performed activities that involved interacting with the platform (not solely viewing slides). These activities could include commenting on slides, creating or editing presentations, using the questions feature, and solving problems and exercises using the platform. Authors and educators were also asked to complete the survey from the perspective of how they had used the platform to support teaching and learning activities. Survey results were analyzed at the end of each year of the project. Due to the anonymous nature of the survey, it was not possible to analyze responses from individuals with disabilities or individual needs separately. 5
In the first year, the platform was deployed as a beta version, and some of the minimum requirements for teaching while using the platform were not available until the end of that year. The survey responses reflected that the platform was mostly used for authoring by project participants, and that training was only undertaken with a small number of learners. In the second year, many more features were available and the platform was launched for public access in the middle of that year. This resulted in a growing number of trial participants. As of April 2018 the trials comprised 195 contributors who had created 547 decks and 12; 363 slides. In addition, the trials had reached 100 Educators and 1; 947 Trainees.
The feedback mechanism built into the platform proved popular with the trial partners. It allowed them to log issues, ask questions and make suggestions while they worked with the platform. Over an 18 month period, 777 issues were submitted. It was found that tickets received for the same issue or suggestion, helped the development team with the prioritization of future work. However, one limitation of the feedback mechanism was the lack of a means to communicate back to users due to its anonymous nature. 5.2 During the second year, 291 respondents completed the survey and provided 614 free text comments. Table 2 summarizes survey results for the first and second years of the project. It shows the average SUS Usability Standardized Score, the Difficulty of Task Standardized Score, and a score related to the Usefulness of the platform. The results were aggregated for each educational sector.
Educational Sector Higher education Open Community Secondary Education Professional and Vocational Professional and vocational sector In the first year of the project the professional and vocational sector reported a SUS usability score of 57.8 which increased to 66.8 in the second year. The difficulty of the task and the usefulness scores also increased between the first and second year of the trials. The trial leaders identified barriers particularly related to the import process in the Beta version of the platform and the preservation of the formatting when importing presentations. There were positive comments when viewing presentations pointing to the easiness of the task.
Higher Education For the higher education, the SUS scores for year 1 were slightly higher than the scores in other sectors at 62.5. The difficulty of the platform use was considered neutral, but usefulness was also higher (66.7). During the second year, feedback of learners was gathered more systematically and the average SUS score rating remained similar, which indicates that the authors and consumers of slides rate the system equally well, although there is still room for improvement.
While generally the higher education trials found the experience of delivering content was good, many echoed the comments of other trials about the need to improve the slide import and editing tools. This remains a high priority for the development team due to this feedback. The higher education trials also identified a requirement to group decks together which had a similar topic or were used in the same course. This feedback has led directly to the development of a Playlist feature for curating lists of decks. Secondary Schools In year 1 the SUS score for the trials involving authors, educators and learners in secondary schools was 53.5, although the standard deviation values indicated a wide range of responses. For secondary education, usefulness and difficulty scored low and comments by respondents indicated that they encountered many errors or found features that were not yet available. Feedback from the teachers involved in these trials indicated that when they attempted to upload an existing presentation to the platform, it took a considerable amount of time to replicate the exact layout of their existing materials on the platform.
In the second year of the trials, the average SUS score decreased to 35.9 suggesting that the usability of the system decreased. However, the range of responses remained very wide, which indicated that perceptions of the usability of the platform were very varied. Further investigation identified that the scores from one secondary school trial in year 2 with particularly low scores. The leaders of the trial reported that the teachers had limited time to get to know the platform and that at the time the interface had not been translated into their native language.
The school-based evaluations highlighted the difficultly with designing a complex web-application for users who may have limited experience of using online platforms. Such feedback has enabled the development team to identify where features and interfaces need to be redesigned to make them simpler to learn. However, the wide variation in scores also demonstrated how perceptions of usability and usefulness were affected by how the trials were conducted.
Open community sector The open community sector incorporated a wide range of informal learning settings from workshops to MOOCs and online courses. Due the nature of this approach to learning, trials encountered difficultly with gathering survey responses from all the learners that used the content. In year 1, the authors from this sector reported an average SUS score of 53.8 which increased to 63.0 in year 2, in line with other sectors. In year 1, the authors reported that it was very difficult to create content with a difficulty task score of 27.1. By year 2 the task difficulty task score had increased to 60.9, demonstrating that the authors found the platform less difficult to use. However, some trials had trouble with the performance of the SlideWiki platform which made it difficult for large number of students to access the slides at the same time. This was reported to the development team who made performance improvements a priority for the latter part of year 2. 5.3
Two trials specifically addressed the needs of learners with disabilities: one was
designed for learners with visual impairment and the other for learners with cognitive
difficulties (i.e., intellectual and neurodevelopmental disabilities). These trials were
evaluated using face-to-face meetings with the learners to analyze their interactions and
requirements. The trial with visually impaired learners faced some challenges when
accessing the parts of the platform with screen-readers The trial with learners with
cognitive disabilities identified that they required an easy-to-read description of the platform
functions, as well as a simplified means to navigate to a group of decks. . At the end
of these trial meetings, all the challenges and requirements were collected and
prioritized for implementation. While some of these requirements were easy to implement
and were considered useful to the functionality of other trials (for example collating
similar decks using the Playlist feature), other requirements were specific to the
individual needs. Some of the requirements were also conflicting. For example, visually
impaired learners preferred extensive descriptive texts for a given resource while those
with cognitive impairment preferred a visually illustrative material (e.g.images) with
less emphasis on text. This raised the need to consider learner characteristics,
personalization and preferences as part of a leaner profile in order to guide them to the most
appropriate resources and more convenient learning experiences as proposed by [
The development of the SlideWiki platform in parallel with large-scale trials across many different educational sectors has enabled the project to deploy numerous approaches for capturing and utilizing user feedback.
During the second year, a stable version of the platform has been released, the editing interface and functionality had improved and other features have been added to the SlideWiki platform based on feedback from the earlier trials. The feedback from the large-scale trials has also allowed the project plan to be adjusted to meet the needs of the users.
The trial evaluations were also supported by an assessment of the Quality in Use of
the platform at the end of year 2. This was based on the Quality in Use model proposed
by Fogli & Guida [
The SUS scores reported by the trials were lower than are expected from websites
that have been considered user-friendly. It was recognized that as website interfaces
become more complex and offer more functions, users perceive the sites to be less
usable [
The large-scale nature of the trials provided an exceptional opportunity for gathering a wide range of user-centered feedback but also created challenges. While the survey data proved useful for evaluating the areas of the Quality in Use model for Usability and Impact, it was difficult to rely on the quantitative date and SUS scores as responses were closely linked to quality and appropriateness of the learning content that was being created by the authors, as well as how the trial was conducted. Another difficulty was that data tended to be collected over an extended time fame as usually a trial would 10 https://www.w3.org/WAI/eval/conformance be part of a 3-6 month course. As the project was employing an Agile development methodology, the platform was updated approximately every six weeks. This meant that data from surveys and trials often referred to earlier releases of the platform and issues that may have already been corrected. Moreover, it was also clear from the trial responses that it was difficult to compare the raw scores between years when both the platform and the trials became larger and more complex over time.
It appears that the project mainly benefited from using the SUS scores and quantitative data for highlighting areas where further development work should be prioritized, rather than as a benchmark for the usability of the platform. Cross-referencing of survey responses with comments has been necessary to understand what features were influencing the quantitative scores. In addition, it has been necessary to employ direct user research approaches, such as focus groups, interviews and collaborative design sessions to refine the approach to issues raised by the trials and to ensure all accessibility requirements have been met.
The large-scale trials have contributed to the continued growth of the platform user base and have developed a body of OER. Figure 3 illustrates the growing number of users from the beginning of the project 2015 until April, 2018. By April 2018 the platform had 4,820 registered users and 25,261 decks. Of these decks, 1,356 have been forked and 3,512 have been edited by more than one author.
This means that approximately 14% of the decks have been edited by more than one author and 5% of the decks have been forked for reuse. This percentage is not as high as indicated by the trial leaders plans but it is expected that reuse of content will increase towards the end of the project, as trials continue to build on existing content. In addition, these figures do not take into account educators reusing and downloading decks without editing the content which has currently not been tracked.
The trial evaluations highlighted the fact that it can be difficult for authors to become familiar with version control mechanisms, such as forking, that are required for reusing content. This was particularly the case for those authors who are less familiar with online content platforms. This can, in part, be addressed by ensuring the interface is user-friendly. However, authors would also benefit from being guided through the process through training materials and guides.
One limitation of the effectiveness of the large-scale trials has been the objectives and technical priorities that were agreed prior to the project initiation. This is common in externally-funded projects but places constraints on how fully a developed project can adopt user-centered design and agile development processes. Despite this, the largescale evaluations made possible through the trials, combined with the continuous informal feedback tickets, has resulted in the tailoring of the project to reflect users comments and more closely align the platform to meet the needs of those who wish to collaborate and create open presentation decks.
Acknowledgments This research has been supported by the EU project SlideWiki (grant no. 688095) and by the BMBF project InclusiveOCW (grant no. 01PE17004D).