This paper reports on a comparative case study of 3 ongoing research and development (R&D) projects, each conducted virtually across multiple worksites, involving varying degrees of task uncertainty at differing stages on an innovation continuum. This NSF-funded study applied Pava's methodology of sociotechnical systems (STS) analysis to assess the influence of virtuality and task uncertainty on the quality of deliberations. Building on theory of organizations as information processing systems, different technical and social coordination mechanisms were then studied for their impact in mitigating knowledge development barriers at differing levels of task uncertainty. Technical elements, many based in information systems (IS), appeared to be most significant for coordination where task uncertainty and ambiguity were low. On the other hand, in the context of high task uncertainty, the most significant mechanisms were closely tied to the formal and informal social systems of virtual organizations. Using these findings, a trial application of a 4-step 'STS' methodology for design and use of IS and other coordination mechanisms has now been successfully completed in support of virtual knowledge work at a prominent North American research laboratory. In summary, these findings put into perspective the value of cross-organizational information systems, as a valuable part of the solution of virtual organization for innovation, but only within a larger sociotechnical systems framework that is the basis for a robust 'STS' collaboration platform.
1
Cross-industry, cross-discipline, network-based organization has become central to
the emerging practice of science and engineering [
In global software development, coordination has been described as “the major
challenge” [
It is within this context, that this comparative study of ongoing research and development (R&D) conducted in virtual, geographically dispersed organizations aims to shed new light on the coordination of knowledge work and innovation across time and space. The organizations and projects studied here represent different stages in an innovation process continuum ranging from basic research to scale-up and commercial development. Using sociotechnical systems (STS) analysis as a methodological approach, the research has focused on understanding the influence of virtuality on deliberations and knowledge development at various stages of the innovation continuum. Then, the research question has been about how coordination enables actual achievement of innovation in such distributed, multi-organizational collaborations. 2
Three ongoing virtual R&D projects have been included in this study; each project is
in a different industry and each deals with different challenges based on the type of
virtual work being done. R&D has been characterized as an intrinsic learning system
[
Each project in this study is located at a different stage on the continuum of the
innovation process, and each displays a different level of uncertainty in the project
work. The “Orchid Project” was a pure research project (R1) on the innovation
continuum; The “Uniform Data Set Project” was initially studied in the early
development stage (D1), and more substantially at the advanced development stage (D2) on
the continuum; and the “Large Video Game (LVG) Project” was primarily positioned
in the scale-up stage (D4), although the systems engineering aspects of this project
more closely aligned with the start-up stage (D3) of development.
1 Carolyn Ordowich (personal communication, March 26, 2009) outlined an Innovation
Continuum adapted from a research portfolio model developed and used at Bell Laboratories [
Pure% Research%
Work% DON’T&KNOW& %WHAT%% we&are& looking&for& DON’T&KNOW&&
HOW%% to&carry&out& the&research&
R 2 Applied% Research%
Work%% DON’T&KNOW&&
WHAT% (i.e.&end&state& or&objec?ve)&
KNOW& %HOW%% to&carry&out& the&research&
Exploratory% Development%
Work%
Advanced% Development%
Work% D 1 KNOW& %WHAT%% DON’T&KNOW&
&HOW% &to&achieve&it&
D 2 &KNOW& WHAT%
D 3 StartBUp%%% (pilot&plants,&& beta&tes?ng)& Development%
Work&! KNOW&&
WHAT% DON’T&KNOW&&
HOW%IN%
DETAIL%% to&achieve&it&
KNOW&&
HOW% CONCEPTUALLY% to&achieve&it&&
D 4 ScaleBUp% (volume&&&
costs)& Development%
Work%& KNOW&& WHAT% KNOW&&
HOW% OPERATIONALLY% to&achieve&it& HIGH%Uncertainty% LOWER%Uncertainty% ‘Orchid’'Project! ‘Uniform'Data'Set’'Project! ‘Large'Video'Game’'Project!
Each project in this study is located at a different stage on the continuum of the innovation process, and each displays a different level of uncertainty in the project work. The “Orchid Project” was a pure research project (R1) on the innovation continuum; The “Uniform Data Set Project” was initially studied in the early development stage (D1), and more substantially at the advanced development stage (D2) on the continuum; and the “Large Video Game (LVG) Project” was primarily positioned in the scale-up stage (D4), although the systems engineering aspects of this project more closely aligned with the start-up stage (D3) of development.
In addition to being clearly identified as R&D projects, each of the projects has
been conducted in its own virtual organizational setting. In each case, the work is
comprised of interdependent knowledge-based tasks conducted by participants who
are dispersed across space and time and are unable to collaborate face-to-face all or
most of the time. Thus, each case exemplifies the primary characteristics identified in
prior studies of “virtuality” in work processes [
The “Orchid” Project represents the field of fundamental, basic research and appears at position R1 on the innovation continuum; it is a collaborative project among theoretical and experimental physicists from research universities around the world. The project, funded by the US Defense Advanced Research Projects Agency (DARPA), is led by scientists from Caltech and includes physicists from universities in the U.S.A., Canada, Austria, and Germany. It is a pure research study in which the researchers don’t know what they are going to find and therefore, don’t know how to design a research project that will actually be effective. The degree of virtuality is quite high in the patterns of interaction between faculty and students or post-doc staff.
The “Uniform Data Set” Project (UDS) is a joint project of the National Institute of Health and 29 Alzheimer's Disease Centers across the United States. At the outset, in the development of the “minimal data set” the project was positioned at D1 on the innovation continuum – the parties knew what their goal was but didn’t know how to accomplish it. Based on this experience, this has evolved to a mature development project (D2) that is expanding its investigation based on earlier accomplishments. The chief participants have worked together for a number of years under overall guidance of the National Alzheimer’s Coordinating Center. In addition, there are substantial professional ties within and across the centers as the membership consists of a majority of the world’s experts in Alzheimer’s Disease treatment.
The “Large Video Game” Project (LVG) involved some Start-Up Development (D3) and mostly Scale-Up Development (D4) activities; it incorporates art asset production, website and systems engineering, and testing activities shared among the game developer and vendors around the world. Clarity of purpose and outcome is crucial in the D4 positioning of LVG, and though uncertainty about the ‘what’ and, to a somewhat lesser extent, the ‘how’ of the process is low, there is a high degree of virtuality and relatively low face-to-face collaboration in this project. 2.2
In virtual organizations that involve innovation, work is non-linear and knowledge
based. This means much of the work is conducted through discussions and
choicemaking interactions that are often not face-to-face; these are referred to as
deliberations in sociotechnical systems theory. Deliberations are “patterns of exchange and
communication…to reduce the equivocality of a problematic issue” [
The value of deliberation analysis to identify sources of failure and delays in new
product development has been demonstrated [
Building on Pava’s work, others have identified the source of such information and
knowledge “gaps”. In two product development projects co-located within one major
consumer products company, Purser et al. [
Now, for this comparative case study, concepts of ‘deliberations’ and knowledge development ‘barriers’ have been extended to the analysis of knowledge work including exploratory development and fundamental research where “equivocality” and task uncertainty are greater than in most product development, and potentially more so in the context of virtual organization.
Therefore, to help frame the focal questions of our study, an extensive review was conducted of the literature on virtual organization. Then, scoping interviews were conducted in each organization to gain understanding of the projects and teams involved in virtual innovation work. Through structured interviews and observation, key deliberations were identified and tracked in each worksite to gather core data about the emergence of barriers and the extent to which they were addressed in each realworld innovation process. Finally, follow-up interviews and documentation verified the project outcomes. Indeed, STS analysis provided a powerful lens through which to view knowledge generation and sharing, eventually yielding insight into both social and technical forms of coordination in these virtual work environments. 2.3.
From an organizational studies’ perspective, coordination mechanisms are developed
or emerge because of the need for “managing dependencies between activities” of
distributed actors [
A connection between coordination mechanisms and the possibility of mitigating
knowledge development barriers is based upon theory of organizational information
processing [
Specific mechanisms to permit coordination have been proposed using an
information processing view of organization design. However, more specific to global
software projects, and most relevant for our study of R&D, Sabherwal [
Coordination through “standards” relies upon pre-specification of rules, routines, techniques, and targets. Coordination through “plans” is another approach that is mostly impersonal in nature once implemented. Both of these forms of coordination are often built into the structure of information systems. By contrast, in both forms of “mutual adjustment”, coordination is made possible through interpersonal communication, feedback and interaction. In formal mutual adjustment, coordination is “more structured” in design review meetings, supervisory or liaison roles versus informal mechanisms of impromptu or face-to-face communication.
In addition to defining key modes of coordination, theory and empirical research
[
In summary, there is considerable prior literature suggesting that task uncertainty is an important factor influencing coordination mechanisms. The intent of this comparative case study has been to take a ‘grounded theory’ approach to extend these findings to a virtual context, and to encompass the full range of the innovation continuum. 3 3.1
The Large Video Game project is a critically time-bound commercial product development process based in the USA with a virtual organization of contractors dispersed across the globe. There is limited economic viability for face-to-face interaction among members of the virtual project teams. Production includes 3D animation art assets, systems engineering, website design, and quality assurance. In addition to LVG home-based staff, the virtual organization includes external art asset vendors as well as engineering and website development vendors.
Key deliberations at LVG often occur at the front end of the production process involving ‘choice points’ such as vendor selection. Examples of other key deliberations are defining and estimating outsourced project work and specifying documentation and production requirements.
During the period of this case study, it appeared that knowledge sharing and development barriers were less prevalent in virtual art production than for virtual organization of software engineering and web systems development where barriers included unclear expectations, unrealistic timeframes, and lack of documentation. Delayed data transfer resulted sometimes from incompatible IT systems and/or security issues. Intellectual property issues could also prevent LVG core operations from sharing vital source code with vendors.
In the relatively routine and mature work processes of virtual art production for LVG, information systems have provided vital support for clear expectations about task deliverables. Agreements on acceptable output are coordinated using screen shots, visual targets, emails, extensive digital documentation, and in some cases, webbased project management software.
For engineering and web/online game development, however, a key factor limiting the clarity of expectations is that LVG staff will most often not know the fine details of ‘how’ the outputs are to be achieved. In-house staff may do preliminary design of new website features but detailed technical design is outsourced to a vendor. However, quick feedback that is possible in-house, standing over each other’s computers and making ‘live’ corrections to any misunderstandings has generally been unavailable with engineering vendors in this virtual organization. Another disruptive but unintended factor was that the otherwise very successful ‘agile’ development process used by LVG staff caused expectations to change mid-course several times for the work of at least one major website contractor. Both of these factors resulted in delay and cost overruns particularly for the first product version of game development observed in this study.
Fortunately, in the time period between the two product development runs, LVG staff made important changes in their coordination mechanisms. Engineering projects are now “chunked” into phases, and vendors must provide schedules for specific deliverables. And, supplementing all of the regular project management tools and systems, LVG made a structural role change to designate a single “product owner” contact person to resolve issues with each vendor for a specific engineering assignment. New technical arrangements have also helped overcome the intellectual property issues that previously constrained the sharing of game source code--a “cloud-based desktop” solution provides vendors access to source code and the ability to integrate new code, while preserving LVG proprietary control. And, selection of any vendor is now dependent upon verification of IT compatibility and an on-site security check. To close yet another gap in knowledge coordination, quality assurance staff in a remote test center can now videoconference into production meetings and ‘scrums’ at LVG core operations, and thereby increase their tacit knowledge of game architecture. The overall effect of such changes was that the second product run was completed ontime, on-spec, with few quality issues, and on-budget. 3.2
The Uniform Data Set (UDS) is a longitudinal database of clinical and neuropathological information gathered from Alzheimer’s patients in the United States. From 1984 to 1999, the initial development of this database (D1) was the Minimum Data Set that suffered a missing data rate of 20-30%. By 1999, the sponsor agency, the National Institute of Aging (NIA) recognized a need for a reliable, more robust data set as a resource for Alzheimer’s research, and established a National Alzheimer’s Coordinating Center (NACC) at the University of Washington-Seattle. The Center’s mandate was to support more effective collaboration among 29 Alzheimer’s Disease Centers across the United States in development (D2) and utilization of a Uniform Data Set. Since then, the NACC has worked with clinical task forces of Alzheimer’s Disease Center directors and clinical core directors to develop and update the standardized content of the UDS.
Key deliberations in this project (conducted via videoconferences, teleconferences, email, and sometimes, in person) have selected the 725 data points to include in the data set, an important issue because it determines what longitudinal information will be available for researchers. Another key deliberation has revolved around how to collect the UDS data: as many as 18 standardized forms developed by clinical task forces are now used to collect patient data on socio-demographics, family history, dementia history, neurological exam findings, functional status, neuropsychological test results, clinical diagnosis, and imaging tests. Data managers at each of 29 Centers monitor the quality of the local data before submitting it electronically to the NACC each month, creating a reliable, large-scale pool of data for scientists to analyze.
The move to the UDS from the original data set raised a number of issues: initially, many of the Alzheimer’s Disease Centers resisted the concept of a “coordinating center” and viewed the requirement to use standardized data collection systems as an imposition on being able to collect data best suited to their particular research interests. This created major barriers to knowledge sharing in the early deliberations about what elements to include in the UDS. Other barriers arose from the different frames of reference associated with researchers’ diverse disciplines.
The NACC was a purposefully designed coordination mechanism to address the
barriers. It has provided an infrastructure, a neutral “referent organization” [
Furthermore, on an ongoing basis, the NACC coordinates bi-annual face-to-face
meetings of the ADC directors and staff. Although infrequent, these face-to-face
meetings are one key part of a dense set of relationships among participants in the
ADC network. This collaborative ‘spirit’ has been further strengthened by the larger
shared ‘mission’ to reduce or solve Alzheimer’s disease. Overall, the outcome has
been that NACC is now instrumental in Alzheimer’s research and the UDS has
received acclaim as an exemplar of research collaboration [
The Orchid project was an international multi-university collaboration by a team of 20 physicists and graduate students led by faculty at the California Institute of Technology (Caltech) who partnered with scientists at universities in Europe and North America. The project involved experimental scientists and theoretical physicists, many of them physically dispersed. The distributed collaboration most closely studied by our research involved one Caltech lab that fabricated devices for experiments run both on its own equipment and also on different equipment in an Austrian laboratory. There was thus strong interdependence between these laboratories. However, until the Orchid project, staff from these two scientific groups had never collaborated. It was their brief meetings at international conferences that brought them together with a mission to achieve a “scientific breakthrough” in a new field of science, opto-mechanics (i.e. use of light to manipulate mechanical devices at nano-scale).
Key deliberations within this project focused on the selection of experiments to run, design of the actual experimentation, and interpretation and refinement of data gathered. Knowledge barriers associated with these deliberations were significant. Varied disciplinary roots of the research groups led them to use different language to describe the same data, and each group had its own unique problem-solving approach. A significant challenge was the wide geographic dispersion combined with the high degree of reciprocal and team interdependence between their laboratory facilities. There was a constant threat of failure to utilize knowledge if the diversity of scientific perspectives could not be accessed and integrated for creative problem solving in the experimental process. Another major barrier to the acquisition of knowledge resulted from some incompatibility in the equipment used by the different laboratories.
For coordination, Orchid project scientists made extensive use of shared databases and annotated document repositories. Whenever experiments picked up intensity, digital communication such as skype conversations, sometimes with screen-sharing, or use of electronic whiteboards, texting and email could occur almost constantly during a long, multi time zone work day.
However, the project’s greatest collaboration challenges were overcome quite serendipitously. The need to invent a methodology so that devices created at Caltech could run on different experimental equipment in Europe required a detailed understanding by each party of the other’s technical capabilities and limitations. The mechanism in this virtual organization that most helped bridge the different frames of reference was what the scientists came to refer to as the role of an “embedded researcher”. A European graduate student came to Caltech for a short visit by chance and was able to see differences in methods and technology between the two experimental groups and facilitated solutions to merge their approaches. Another graduate student, from the theoretical school, was also unexpectedly sent to Caltech—he was able to give real-time suggestions to help interpret data for the experimentalists. This liaison or “straddler” role was an ongoing help to coordinate knowledge exchange between project theorists and experimentalists.
Both of these temporary roles proved to be vital coordination mechanisms for this
project that over four years yielded a series of internationally recognized publications
[
All the virtual R&D projects in this comparative case study encountered substantial knowledge development barriers, and utilized coordination mechanisms to overcome barriers. Of the four main categories of coordination mechanisms (standards, plans, formal mutual adjustment, informal mutual adjustment), all types were utilized to some degree in specific examples developed in each project. However, the type of mechanisms that project participants indicated were most significant in mitigating knowledge barriers varied noticeably according to the project task (see Figure 2).
One theme in these findings is an apparent correlation between most impactful
types of coordination mechanisms and differing levels of innovation task uncertainty.
For those activities and projects with the lower degree of uncertainty, the more
impactful mechanisms were technical, impersonal, relying on an a priori specification of
action or targets-- for example, the screen shots, visual targets, and project
management software that provided ‘standards’ and ‘plans’ to coordinate expectations
between LVG and its art production vendors. However, the social, or mutual adjustment
mechanisms that are more indeterminate and rely on extensive ad hoc human
interaction had more impact in mitigating barriers in those activities and projects where there
was higher uncertainty about outcomes and process: for example, the “embedded
researchers” who contributed vital liaison across disciplines and institutions in the
Orchid project, and functioned much like the “straddler” role described as a conduit
for “transfer of tacit knowledge” in global software engineering projects [
R1)
‘UDS’) ‘LVG’) D2)))))D3ID4) D3))))))))))D4) +)
In sociotechnical terms, the social or mutual adjustment mechanisms are based
primarily in the social sub-system of a work organization, while standards and plans
are based primarily in the technical sub-system (see Figure 3). For example, a project
status review meeting (as a formal mutual adjustment mechanism) may rely upon a
teleconferencing technology application, but the primary contributions to the review
meeting as a coordination mechanism are located in the leadership and other roles,
mutual task expectations, and relationships within the virtual team or groups
performing the work [
Indeed, the ‘technical’ and ‘socio’ dimensions of coordination appear to complement one another. Neither is entirely sufficient for overall coordination, but each tends to be more impactful, depending upon the stage of innovation or nature of knowledge work. How different types of coordination mechanisms are complementary is exemplified by the experience in the LVG project for systems engineering and website development. At this (D3) stage of innovation, effective coordination required a combination of important ‘technical’ elements of web-based project management software and short time frame “chunking” of project plans, along with a formal mutual adjustment mechanism in the form of a new “product owner” role within the social system of relations between LVG and its vendors. Conversely, what made the informal mutual adjustment mechanism of the Orchid project scientists’ infrequent face-to-face discussions most effective were detailed plans and data-sharing done prior to their meetings.
Another form of interaction between the ‘socio’ and ‘technical’ dimensions of
coordination is the significance of how these mechanisms are used, quite aside from the
process of their design or selection. For example, as suggested by prior studies [
Such use of data repositories enabled dispersed scientists to experience a form of
collaboration awareness: “an understanding of the activities of others, which provides
a context for your own activity” [
The case of the UDS project with its challenges of effective data collection
highlights another critical aspect of coordination mechanisms, namely, “malleability” [
Taking further this notion of ‘fit’ between a coordination mechanism and its ‘field
of work’, the case of quality assurance testing in our study of the LVG project
provides some indication of the importance to distinguish “different modes (i.e. alpha
levels) in which a protocol-based system…can support [coordination]…from the
more constraining mode to the less constraining”, in terms of whether or nor there is
affordance for users to skip or defer any action within the intended process [
In summary, many of these effects occur in co-located work as well as in virtual
organizations. However, participants in this study reported that, compared to their
experience of co-located work, barriers to the development of knowledge (e.g.
intellectual property issues, divergent priorities) were more difficult to manage in the
virtual context of innovation. And, although scientists and their graduate students used
virtual workspace IT tools for task coordination [
Even though there is “a common notion that collaboration technology and
bandwidth will allow a virtual team to perform as if co-located, evidence shows this notion
to be a naïve myth” [
As an indication, a recent trial application of these research findings in a major North American research laboratory was viewed very favorably by scientists and staff challenged with coordination of teamwork across time, space, and changing environments in the laboratory and its network of related universities, and private sector stakeholders. The work of these scientific teams covered a wide variety of topics, at differing stages across the innovation continuum.
A series of workshops were held periodically over several months at the laboratory to share the findings of this research study. During and between workshops, scientists and their fellow team members applied the concepts to analyze the process of their teamwork, and then, select or develop and evaluate new coordination mechanisms, using a four-step STS design methodology.
• First, locate the project or specific knowledge work on the Innovation
Continuum (see Figure 1). Awareness of the positioning of a team’s work on the
continuum, (and this positioning may well move during the life of a project),
helps anticipate the types of ‘technical’ and/or ‘socio’ mechanisms that are
likely to be most significant in mitigating knowledge development barriers
(see Figures 2 and 3).
• Secondly, identify the key deliberations or ‘choice points’ that are essential
to move the team’s work forward. Deliberations are defined by a topic (e.g.
what experiment to run, what software feature to develop), and they require
specific information and knowledge, with the involvement of specific
participants with differing perspectives and interests.
• Thirdly, analyze the most significant knowledge development barriers that
potentially or actually impede the quality of these key deliberations. To help
maintain alertness to such barriers, utilize the typology of (1) knowledge
sharing and planning barriers, (2) cognitive frame of reference barriers, (3)
knowledge retention and procedural barriers, and (4) knowledge acquisition
barriers.
• Fourthly, select, design, and/or utilize appropriately the specific coordination
mechanism(s) that seem most capable of mitigating the identified knowledge
development barriers. This aspect of “designing” [
At the conclusion of the trial application, over 90% of the scientists and staff reported in a feedback survey that these concepts and methodology “will improve how we work together” and “address [distributed teamwork] issues we were trying to solve”. The coordination mechanisms developed by the scientific teams included a combination of new standards and procedures, new systems for information sharing and storage, and redesigned team roles.
The findings of the research reported here and the recent application experience put
into perspective the value of ‘technical’ elements of cross-organizational information
systems (IS) and web-based collaboration technology. They are a valuable part of the
solution for coordination of “virtuality in teams” [