This paper describes and reflects upon taking a 'Research 2.0' approach to developing a 'vision and strategy statement' for a network of researchers involved in researching Technology Enhanced Learning (TEL). It relates how the statement was developed first by collecting content from colleagues within the network through face to face meetings and contributions to a wiki and then by creating a coherent linear text document which further developed the content on the wiki. It discusses the risks inherent in the approach and outlines the strategies taken to address the risks. It suggests that, although the approach taken was successful, the success was limited owing to factors including a) limited engagement by the community with other people's contributions, b) a reluctance to amend other people's contributions and c) the difficulty of aggregating the multiple voices within the community while retaining faithfulness to the philosophies underpinning a 'Research 2.0' approach.
This paper describes, and reflects on, the approach taken to developing a research ‘vision and strategy’ statement for the European Network of Excellence, STELLAR. The statement needed to reflect the views of a diverse community of researchers in Technology Enhanced Learning (TEL), represented by individuals from a variety of backgrounds such as computer science, engineering, education and psychology. The representatives of the community work in sixteen different labs in nine different countries in Europe and work within a wide range of research and cultural traditions. Given the diversity of backgrounds of the individuals within the community, producing a joint vision and strategy was a significant challenge.
This paper reflects on how the deliverable was produced using a ‘Research 2.0’ approach, critically examining the process and the products. The paper develops a use case scenario, discusses the influence of Research 2.0 on the scientific practice of developing the statement and evaluates the use of Research 2.0 tools. The paper describes the novel approach adopted and the successes and failures of the endeavour.
STELLAR is a multi-disciplinary consortium (Network of Excellence) which aims to bring together the different research traditions and disciplines within TEL. The cornerstone of the work of STELLAR is the Description of Work (DoW), which was developed by drawing on knowledge and expertise of members of previous Networks of Excellence, Kaleidoscope and Pro-Learn.
The DoW identified three themes (called ‘Grand Challenges’) intended to be a starting point for providing a framework to identify and formalise the visons and strategies for TEL: 1) Connecting learners 2) Orchestrating learning 3) Contextualizing virtual learning environments and instrumentalising learning contexts. For each theme, the DoW also posed a number of related research questions.
One of the early deliverables for the consortium was to produce a document outlining the vision and strategy of the whole STELLAR consortium, by developing the themes in the DoW. One partner of STELLAR (University of Bristol) had ultimate responsibility for the document, but considered the vision and strategy for the consortium to be the responsibility of all partners, and wanted to find a way to for the whole consortium to contribute to the joint vision and strategy. As such, the enterprise could be seen as successful if all partners were actively engaged of in the construction of the vision and strategy.
As a Network, STELLAR subscribes to the idea of ‘Science 2.0’ as a way of
working; this approach draws on ‘Web 2.0’ and can broadly be described as being
underpinned by the democratic principle in which members of a community have the
opportunity to contribute to a collaborative project and the contributions of all
individuals are valued and become aggregated to represent the ‘wisdom of the
crowds’ [
‘… Web 2.0 has been ushered in by what might be a thought of as rhetoric of 'democratisation'. This is defined by stories and images of 'the people' reclaiming the Internet and taking control of its content; a kind of 'people's internet' … This, we are led to believe, has led to a new collaborative, participatory or open culture, where anyone can get involved, and everyone has the potential to be seen or heard.’ [2] ‘ The Internet is enabling an unprecedented number and variety of individuals to contribute knowledge, by authoring content individually or collaboratively and by helping one another directly in online forums. [3]
We argue that, because the research approach parallels the ‘Web 2.0’ approach, it could be called ‘Research 2.0’. Research 2.0 uses tools and technologies as appropriate for the tasks involved in the research process, and these may include Web 2.0 tools, such as wikis, blogs, micro-blogs, podcasts, reference management and sharing (e.g. Delicious and Mendeley), photograph sharing (e.g. Flick*r) and social networks. (For example, see [4] and [5]). However, we argue that Research 2.0 can also use more traditional non-digital research tools to generate content such as face to face discussion, focus groups and interviews. Our key concern was knowledge creation using appropriate methods and tools.
We suggest that we have much to learn about knowledge creation within a 2.0
approach from the use of Web 2.0 tools and hence we draw on literature relating to
Web 2.0, and in particular wikis, to inform us. We focus on the literature concerning
wikis for two reasons: first because Wikipedia is generally agreed to be a successful
example of knowledge creation (e.g. see [
Processes of collaborating: Producing knowledge collaboratively using Web 2.0 technologies (wikis) is still relatively new and the concern of much literature in the area is about ‘what works’. We argue that understanding online collaboration is at the heart of the ‘what works’ question. Coleman and Levin, 2008, put forward their view on collaboration:
Collaboration is, we believe, primarily about people, about trust, and about the
willingness to share information and work in a coordinated manner to achieve a
common goal [
We agree; collaboration is between people, who coordinate to achieve a common goal; in the context of this paper, this coordinated working involves sharing knowledge and building knowledge together. Those concerned are willing to share knowledge and want to share knowledge.
Contributors’ motivations seem to be critical for sustaining Wikipedia and other
collaborative user-generated content outlets. [
As Coleman and Levine (ibid) point out, it is important to establish trust between
the collaborators. This seems to be particularly important in online collaboration:
Web 2.0 is built upon Trust, whether that be trust placed in individuals, in assertions, or
in the uses and reuses of data. [
A further point made by Coleman and Levine is that in successful collaboration the
goal is shared and that members of the collaboration have the same (or similar) end
point in mind. This point was also made by Wagner and Majchrzak [
A factor that is sometimes reported in the literature as contributing to successful
online collaboration concerns explicit rules related to contributing content. Wikipedia
includes a page of ‘rules’ and ‘guidelines’ which are described as a ‘policy, a widely
accepted standard that all editors should normally follow. Changes made to it should
reflect consensus.’ (see http://en.wikipedia.org/wiki/What_Wikipedia_is_not).
Wagner and Majchrzak (ibid) suggest that these guidelines ensure quality:
Wikipedia has strong editing guidelines that are motivated by the refactoring rules of
software development and principles of objectivity. This ensures that articles, which
might have suffered in readability from the disjointed work of multiple contributors and
commentator, ultimately becomes very readable again. [
However, while there are some who consider that rules encourage contribution to
the wiki, such as Wagner and Majchrzak (ibid), others have found that the presence of
rules makes little difference, (e.g. [
Finally, it seems that constructive engagement could encouraged by allowing
different levels of participation; ‘lurking’, commenting on others’ contributions,
making original contributions, editing and asking for explanations of others’ ideas and
organisation of content for better structure. [
Quality of knowledge: Wikis can be successful tools for collecting and aggregating knowledge. As pointed out above, WikiPedia, probably the best known wiki, is generally seen as a success. At the time of writing this paper (July 2010) it had over 3 million articles in the English version, and it is in the top ten web sites accessed anywhere. This demonstrates that it is possible to create a wiki that ‘works’ in terms of community engagement. There is debate, however, about the quality of the knowledge on wikis.
Whereas wikis sometimes have rules of engagement, the knowledge produced on
wikis is usually not subject to editorial control which leads to concerns over the
provenance of information posted. Concerns relate to various aspects of knowledge,
largely to do with the accuracy of knowledge. For example, Don Fallis (2008)
suggests that:
serious concerns have been raised about the quality (e.g., accuracy, completeness,
comprehensibility,etc.) of the information on Wikipedia [
Fallis’ article suggests that Wikipedia has been dismissed by much of the library and information science communities because it is seen as unreliable. He presents a thorough analysis of potential different types of inaccurate information in terms of factual accuracy, completeness, currency and comprehensibility and he demonstrates that Wikipedia fails rigorous tests of accuracy in these respects. However, he continues by arguing that Wikipedia is ‘quite reliable’ and ‘quite verifiable’ and that it contains ‘quite a lot of high-quality accurate information’ (p 1669). He makes the point that ‘it is probably epistemically better … that people have access to this information source’. (p 1669). He argues that there are ways in which the reliability of information on Wikipedia can be improved, but points out that the cost of this would undermine some of the values on which the project is based, such as the number of contributions and the speed with which entries are added and updated. His key point is that ultimately it is the responsibility of readers ‘to decide whether to believe what they read on Wikipedia’ (p 1671) and he concludes by suggesting ways in which to help readers in this respect (e.g. signaling evidence of the quality of articles, directing readers to further reading, flagging omissions).
Concerns over the accuracy of information on wikis and Wikipedia in particular frequently relate to factual content (and this is to be expected in the case of Wikipedia which collects ‘facts’). However, there are other concerns which relate to the quality of knowledge built using online collaboration. For example, Anderson [5] argues that the ‘Web of Content’ (WoC) discourages ‘a deep level of critical thinking’ because development of content is influenced by a ‘powerful zeitgeist’. The computer scientist, Jaron Lanier, in an essay about the dangers of elevating collectivism above merit and thus lowering standards, describes a similar concern:
What I've seen is a loss of insight and subtlety, a disregard for the nuances of
considered opinions, and an increased tendency to enshrine the official or normative
beliefs of an organization. [
This section has outlined some of the key issues relating to the collaborative production of knowledge within an online environment, with a focus on the use of wikis. It demonstrates the keys risks associated with using a wiki in terms of the amount of knowledge produced and the quality of the knowledge. In terms of the former, the main risk seems to be non-participation in the process of knowledge building and we recognised within our project that we may need to take steps to encourage our colleagues in STELLAR to contribute to the wiki. In terms of the latter, the risk for us was less clear. Our project was not essentially about collecting facts, as Wikipedia is, and we did not consider that we risked inaccurate contributions. Our project was more about developing arguments, debate, insight and vision and did, perhaps, run the risks described by Anderson and Lanier above. These risks were less clear to us at the beginning of the project but as it developed we put strategies in place to encourage high quality debate.
The text from the DoW was used as a starting point to create a ‘Grand Challenges’ wiki. The text was pasted into three main pages, one for each of the three Grand Challenge themes. At the same time, the wider STELLAR community was asked to recommend reading related to producing a TEL vision and strategy statement. The recommended readings and were put together and distributed to the STELLAR network and posted onto the STELLAR web site. Members of STELLAR were asked to engage with the readings prior to the face to face meeting described below.
A day-long face-to-face meeting was set up in Bristol in May 2009 (month 4 of STELLAR). 33 members of STELLAR participated and worked in three groups, each with a chair and a note-taker. The groups were constructed to include individuals who represented the diverse research interests and perspectives within STELLAR.
In the morning there were two discussion sessions. Participants remained in the same groups for both these sessions although the chairs and note takers were different.
In the first session groups discussed questions relating to the Grand Challenge theme ‘connecting learners’. Each group was given one of three questions to discuss: • What are key enabling and success factors for learner networks? • What impact could web 2.0 technologies have on learning in educational institutions and what are the implications for a) professional development b) design and organisation of learning spaces c) policy makers? • What are the changing demands for workplace knowledge and skills and what are the implications for a) leaders and managers and b) the workforce? In the second session groups discussed questions relating to the Grand Challenge theme ‘orchestrating learning’: • What is the role of the teacher/more knowledgeable other in orchestrating learning and how does this relate to collaboration and the knowledge of students? • What is the role of assessment and evaluation in learning and how can technology play a role? • From the point of view of the learner what is the relationship between higher-order skills and learning of a particular knowledge domain and what is the role of technology in this respect? For the third session (which took place in the afternoon), participants were put into new groups. These groups discussed questions relating to the Grand Challenge theme ‘Contextualising virtual learning environments and instrumentalising learning contexts’: • How can new forms of technology-enhanced learning enable novel experiences for learners and for development of human competences and capabilities? • How can the mobility of the learner in distributed and multi environment learning settings be supported, to include the transition between a) real and virtual contexts b) informal and formal learning contexts? • Which standards are needed to achieve interoperability and reusability of learning resources in this field? How can we harmonise the existing learning standards? The main purpose of the meeting was to expand the collective understanding of the community concerning the three research themes, through knowledge contributed by experts within the community and discussion and development of related research questions. The meeting was set up using an adaptation of the ‘knowledge café’ methodology (Firestone and McElroy, 2005). Within this methodology discussion is not driven by an agenda, and this is seen to encourage groups to develop discussion in line with the expertise and interests of the individuals in the group.
Note-takers were told that the notes would be added to the wiki but otherwise were not given any specific instructions or guidelines. They adopted different approaches but generally attempted to capture as many of the points being made as possible, not attempting to organise the points into coherent prose. The examples below are taken from discussion starting from the questions ‘What are key enabling and success factors for learner networks?’ and ‘What is the role of the teacher/more knowledgeable other in orchestrating learning and how does this relate to collaboration and the knowledge of students?’ The examples demonstrate different approaches taken to note taking.
This is the first set of aspects created in the first grand challenge vision workshop on May 20th, 2009 in Bristol: • Connections with people with whom you interact • Merging of Formal & informal, Lifelong, Self-organised / self-constructed, • One holistic network per person, not a private one, professional one… • Medium used for communication is fundamental; Software can support maintenance and building of network • Challenge: Integrate networks with learning processes • Most prominently: Social network; but not only people: Networks of people, artefacts (e.g. paper), and tools (distributed cognition, actor-network theory) • Sense of being in control essential (when to use, how to use, …) / responsibility
What does a more knowledgeable other offer? A frame of reference/organised state of mind, knowledgeable other takes a scaffolding role - metalevel role - from research on expertise. Not just content knowledge - pedagogy as a whole - mediating content children in school unlikely to have pedagogical expertise, but just more content knowledge. Teacher required to facilitate knowledge transfer/representation. Maybe there is a changing role of teacher within 21st century - but not necessarily to do with technology.
In one group, the notes were entered directly into the wiki and in the others they were written in a word-processed document and pasted into the wiki. These notes were seen as the starting point for extending the community’s understanding of the Grand Challenges and the plan was to develop them into a more coherent whole over a period of weeks to form a substantial part of the vision statement. Importantly they were faithful to the spirit of the Research 2.0 approach in that contributions from all individuals were valued and the notes represent the collective responses of the community to the nine Grand Challenge questions chosen as the starting point.
After the Bristol meeting STELLAR partners were invited to join a small team to coordinate the ongoing contributions to the wiki (to be called the D1.1 team). Apart from the Bristol team (UB), five partners volunteered: Istituto Tecnologie Didattiche in Italy (ITD), Ludwig-Maximilians-Universität München in Germany (LMU), Centre for Social Innovation in Austria (ZSI), Know Centre in Austria (KC) and Université Joseph Fourier in France (UJF). UB took a leadership role, with other team members taking responsibility for provoking STELLAR members to contribute to a particular subsection of the wiki related to: • connecting learners (ITD and ZSI) • orchestrating learning (LMU and KC) • contextualising virtual learning environments and instrumentalising learning contexts (UJF and UB)
In the first half of June 2009, the D1.1 team met once online (using FlashMeeting, see http://flashmeeting.open.ac.uk/home.html) to discuss how to proceed. Following this, UB put together a written plan which outlined a tight time-frame for the development of the wiki: • 22/6/09 to 6/7/09 – intensive work by all D1.1 team to get contributions from the whole STELLAR community. • 6/7/09 to 30/7/09 UB will take responsibility for developing the wiki into a deliverable. Other D1.1 team members will be asked to contribute by a) writing sections b) reviewing sections and c) clarifying sections where necessary UB also suggested strategies for the D1.1 team to use to provoke colleagues to contribute to the wiki. For example, written suggestions included:
For example, there might be a part of the wiki which you think requires further development; you could use this as a basis to develop a question for people to answer. You might make a sub page with this one question and invite people you know have expertise in the area to contribute a paragraph.
You might find that two people are making similar points, or two people are disagreeing, it might be worthwhile pointing out the synergies and encouraging further debate. However it could be important to find a way of keeping the ‘disagreements’ in the document.
The team met online again in the third week of June to discuss progress and to kick-start the phase during which the D1.1 worked intensively with colleagues to encourage them to contribute. Towards the end of this phase, one member of the UJF team came to work intensively on the wiki with the UB team for three days in the final week of July 2009.
This section has described the ways in which the online collaboration was organised. The next two sections reflect on the results of the online collaborations in terms of a) the extent of engagement of the STELLAR community and b) the nature of the contributions.
The wiki includes functionality to record the editing history of pages; an example covering the editing history of one page over the period of eight days is provided below:
This information allows us to analyse the extent of engagement. Overall about 20 people from STELLAR contributed to the wiki in the period of development from 22nd June to 6th July 2009. However, sometimes a contribution under one name represented a collation of several contributions from an institution so it could be argued that there were more contributors. The majority of the contributions were made by a small number of people, usually within a short time frame. For example, the three main pages: ‘Connecting Learners’, ‘Orchestrating Learning’ and ‘Contextualising Virtual Learning Environments and instrumentalising learning contexts’ pages had the following contributions: Contextualising Virtual Learning Environments and instrumentalising learning contexts
17 July (1) 21 July (1) 17 July (2)
When individuals were asked to contribute by adding content, explanation or examples, generally they were very willing to do so. For example, when UB approached the Open University of the Netherlands (OUNL) asking for a clarification of what is meant by ‘interoperability’, the response was immediate and detailed.
Most people who contributed used the ‘Edit’ function to enter text directly into the wiki, either by adding in new text or amending text already present. A few used the ‘Discussion’ function.
The D1.1 team made concerted efforts to encourage contributions, but as their comments suggest, this was not always easy:
‘We have done really our best to obtain inputs and feedback, but it has been a hard task’ (email communication).
They went on to suggest that it had been difficult because people were not motivated to contribute because they did not understand the origins of the wiki and did not know what its purpose was.
Authorship was also seen as an issue for a number of reasons. There were conflicting ideas about whether or not to acknowledge individual contributions,
I am working on the wiki this week (until Friday). Although everything will appear under my name, I am integrating contributions from different people of my group. Thus I would like to let you know that VL and JP should also be mentioned in case there is a list of authors in the end (email communication).
Others were concerned about the extent to which it was appropriate to edit/modify/add to/ delete the contributions of other people. There seemed to be a tension between valuing and respecting other people’s contributions (and not vandalising the wiki) but at the same time building the best possible document. As one contributor suggested, he was happy as an academic to use a word processor and the ‘track changes’ tool to write collaboratively. He suggested that using track changes can be seen as a way of checking with the original author that changes are acceptable; in other words track changes points out the suggested changes (which can then be accepted or rejected). In a wiki, however, the changes are not so obvious and anyone interested in the changes made would have to make a small effort to access the trail of devlopment.
Many of those who did make changes seemed to need to check the changes they had made with the original authors. For example: ‘I have done a bit of re-organisation, tell me if I am barking up the wrong tree’ (email communication).
There was some debate about writing IN the wiki as opposed to writing in a word processor. There were some who thought that it was much easier to do the latter, but others who argued that this meant that the full authoring trail would be lost. Again, there was some debate about the authoring trail and about how important it is to retain the trail. On a similar note, there was a comment that sometimes people try to be the ‘last author’ in a wiki that is going to be frozen at a given time, because then their voice will be heard.
Finally, a possible barrier to contributing to the wiki may have been the technical difficulty of logging in to the wiki. We do not consider it to be very difficult, but it seems that some people found it confusing. For example, one STELLAR emailed to say: ‘Unfortunately, it appears that I can't log in to edit it despite I can http://www.stellarnet.eu/’. (email communication) log in
The contributions varied in style and length. In general, they tended to take the form of paragraphs setting out the perspective of an individual. The first example, below, takes the form of an explanation about the meaning of ‘interoperability’, provided in response to a direct request from the D1.1 team (mentioned above). This response was sent by email. Essentially this is about sharing resources and tools and system spanning. Within the community several specifications/standards are used. Basically there are several standards of content exchange that allow for exchange of learning content between different platforms. Furthermore interoperability is an important topic that considers more the functional integration of different learning services.
The D1.1 team found this sort of explanation to be very helpful as a starting point but found that contributions were seldom expanded, by either the original authors or other colleagues, with arguments, examples or references.
The second example below starts from ‘taken as read’ assumptions (contexts are more fluid) to suggest a change in focus for educational theory. It goes on to wrap up the paragraph by arguing against polarisation of educational theories.
When the context was relatively stable (in the case of fixed classrooms) educational theory tended to focus on content. However now that contexts are more fluid there is a shift from a focus on ‘content’ to a focus on ‘context’. However such a polarisation of ‘content’ and ‘context’ might be unhelpful in terms of understanding issues related to learning and knowledge construction.
The D1.1 team found this paragraph helpful and interesting, but again noticed that there were no further contributions to the paragraph.
In general, the D1.1 team found that the contributions on the wiki were individually valuable but that the levels of engagement with other people’s contributions was disappointing. There was little evidence of individuals challenging other people’s contributions or questioning what they had said, but typically were more concerned with phrasing and style. This is demonstrated by the example below, in Figure 2, which was taken from the editing history of the page on Orchestrating Learning. The text on the left is the earlier version, and the text on the right is an edited version. In order to produce the final document – a linear text document – the text was copied from the wiki into a word processor document. A UB team of two took responsibility for editing it. This involved forming it into a coherent narrative, removing repetition, adding references, examples and explanations and amending text to achieve consistency in language and style.
A draft final document was completed. Once again, the UB team felt that it was important, even at this late stage, to work within a Research 2.0 approach and so the document was distributed to the whole STELLAR community with a request for feedback. In particular, the community was asked to check that any contributions they had made had been represented in the way they wanted.
Two members of the community were asked to provide internal peer reviews and a final version was produced, taking into account the feedback from the community and from the internal peer reviewers.
The aim of the project described in this paper was to use a Research 2.0 approach to develop a vision and strategy statement for the STELLAR network. This paper described the processes and reported on the outcomes. This concluding section reflects on the project and ends with some recommendations.
We claim that the project was successful in many respects; members of the community did make contributions and the D1.1 editors were able to produce a deliverable based on the contents of the wiki. We suggest that the success of this way of gathering the views of the community can be explained by the existing ‘preconditions’ for a successful online collaborative venture, as outlined in the ‘Quantity and quality of knowledge produced in wikis’ section above. In particular the members of the community were willing and able to share knowledge and had, by the end of the Bristol meeting, developed a level of trust. On the whole, we could claim also that the community had a common goal, although – as reported above – perhaps this was not clear to all colleagues.
However, we were slightly disappointed that the D1.1 team had to work so hard to encourage the community to engage more deeply with the wiki and that many of the contributions were less well developed than we had hoped. As described above, the D1.1 team realised, as the project unfolded, that there was a risk that contributions may be less well formed and debated than hoped for, and made efforts to encourage deeper engagement.
Finally, we reflect on the Research 2.0 approach we took. This approach aimed to draw on the wisdom of the crowds (in this case STELLAR) and to aggregate the multiple voices of the individuals in the community in order to develop a coherent and unified vision and strategy for the community. However, the crowd had many voices and the spirit of 2.0 suggests that each should be valued and heard; the problem for us was that we could not aggregate all the voices while remaining faithful to the Research 2.0 philosophy underpinning our project. It may be that listening to the multiple voices of the crowds is at odds with forming an aggregation and it may be that we have to re-think how we conceptualise an ‘aggregation’ (particularly an aggregation of visions).
As pointed out above, the use of the wiki was perhaps not as successful as we hoped. We suggest that this was the case despite the will and technical ability of the community to contribute. We do not fully understand why we were not as successful as we hoped, but we have some speculative suggestions:
1) Although it seemed that a good level of trust was present at the beginning of the project, STELLAR was a very new community and relationships within the community were still at an early stage. People did not know one another well and may have felt timid about making contributions. This paper has been written almost a year since the D1.1 project came to an end and in the intervening months the community has developed and grown, and (crucially) may be more willing to take the risk of publicly contributing to a growing wiki because of developing trust.
2) The construction of the wiki meant that it was difficult to engage with. There was too much text on each page, often well crafted, which did not seem to encourage discussion.
3) Members of the community did not seem to be clear about the goals of the wiki and how it would contribute to the vision and strategy of STELLAR. They therefore did not know what they should and should not be posting onto the wiki. Importantly, the project was not a research project; it was something different and therefore difficult to engage with.
4) Individuals were reluctant to change text that others had posted and others were reluctant to have their text changed.
In further work on developing STELLAR’s vision and strategy, we intend to continue with the approach we used to produce this deliverable, and to experiment in the following ways: • reduce the amount of text on each page and include prompts to encourage discussion • make the hopes and intentions of the wiki (and the project) clear • encourage the use of the ‘Discussion’ feature of the wiki to overcome the reluctance to change other people’s entries • make it clear that the wiki is a collaborative effort which is based on a Research 2.0 approach and is therefore about building knowledge together in a way that combines the voices of all the community.