Distributed troubleshooting often requires experts to support non-experts at a distance. The resulting separation between collaborators can impoverish their understanding of each other's task contexts, impeding their collaboration. Consequently, this paper argues that developing effective troubleshooting support systems requires determining (1) what contextual information the participants require to perform their tasks, and (2) how support systems can be designed to help provide that contextual information. Based on an ethnographic study of real-world remote diagnosis of electronic devices by ad hoc teams, we have identi ed 12 types of contextual information affecting the remote interaction of expert and non-expert troubleshooters, and we argue that software tools for remote cooperative troubleshooting should be explicitly designed to bridge the gap between participant contexts by capturing and transmitting these types of information. As a means for performing this task, we propose a support framework based on conversational case-based reasoning. The paper illustrates the application of this approach in an implemented testbed system.
In on-site cooperative troubleshooting, team members have direct access to a rich shared
context
This project has been conducted by the Indiana University Knowledge Acquisition and Projection Laboratory (KAPLab), for the domain of troubleshooting complex shipboard electronic systems. In the current U.S. Navy troubleshooting process, shipboard sailors often with limited expertiseare paired with shore-based experts on an ad-hoc basis, to assist them in dealing with problems that they cannot diagnose. Communication options are limited and often asynchronous, with e-mail messages a primary means of communication. At any time, the expert may be responsible for a number of open cases. To perform effectively, the sailor must maintain a picture of the contextual factors constraining the troubleshooting situation and process, and communicate them to the expert. Likewise, the expert must construct and maintain an understanding of the sailor's context for each open case.
The paper is divided into three parts. The rst presents results from a naturalistic study
of real-world cooperative distributed troubleshooting conducted over a nine-month period
However, when cooperative problem-solving is conducted at a distance, providing
context is problematic.
The naturalistic inquiry into the virtual organization of naval technical communities
included nine months (December, 2002 - August, 2003) of data collection from interviews
with naval and civilian personnel, technical documents, and on-site observations at three
locations in the continental U.S. Over 50 hours of semi-structured interviews were conducted
to gather both factual information and the respondents' opinions about events, following
the general framework described in
In observed current practice, non-expert sailors rst rely on standard diagnostic owcharts and technical manuals to isolate one or more detected faults. When standardized methods and support materials are exhausted, the sailor contacts a remote shore-based expert for assistance. During their collaboration, sailor(s) and expert technician(s) often have only basic media support, in the form of email, chat, and paper-based documents. Thus, to resolve a problem successfully: 1) the technician must request a detailed account from the sailor of all actions taken prior to the request for technical assistance; 2) the sailor must convey any unique aspects of the operating environment, including recent upgrades and climatic conditions, 3) based on data received, the technician must diagnose the problem and prescribe corrective actions; and 4) the sailor must correctly perform the corrective actions and con rm successful resolution of fault(s). In current practice, no technological support is available for transmitting contextual information. Instead, the subject matter expert, or SME, must simply ask the sailor for clari cation and reminders of context. Maintaining a view of context is aggravated because the SME may asynchronously handle many problems concurrently, with substantial delays between e-mail messages from the sailor. 4
In all remote troubleshooting, some basic explicitly-relevant problem solving information is available to both participants. For example, in our fault diagnosis domain, in which a subject-matter expert (SME) guides a sailor in diagnosing a problem, this basic information consists of:
Reports on the actions (such as voltage tests, etc.) carried out by the sailor under direction of the expert.
However, our analysis shows that the troubleshooting process is also in uenced by an extensive set of contextual factors beyond this explicit speci cation, any of which can have important effects on the course of troubleshooting. The lack of support for providing the expert with such contextual information makes the development of context-based support a pressing need.
The following sections discuss 12 types of contextual factors identi ed in our transcripts as important to effective cooperation, with examples. Participants are identi ed as the shipboard non-expert sailor and the shore-based subject matter expert, SME, who assists the sailor remotely. 4.1
Much context comes from the characteristics of the non-expert and expert participants. 1. The non-expert's access to knowledge updates: A sailor whose knowledge has not been updated since training may need to be advised of recent developments: SME: there's been an advisory put out about that but these young third class guys would never know there was an advisory out ... 2. The non-expert's training/experience: More highly-skilled sailors can be given less detailed instructions and can convey their problems more effectively: SME: [I]f you don't have an experienced tech on the other end of the line, you can do distance support until you're blue in the face but if he can't describe the problem to you. 3. The expert's skill/experience: While all SMEs can be expected to have current knowledge and comparable training, in practice it is recognized that the SME's experience may play a key role in their suitability for a particular problem: SME: You know, someone will take the email and [think,] Hey this guy's really up to speed on high voltage, so they'll give it to me. Or this guy, you know, it's a no load problem and let's say someone else is better suited to take the assist. 4.2
Characteristics of the problem setting, both historical and current, play an important role in doing situation assessment on the current problem, suggesting possible diagnoses, and selecting diagnostic actions.
4. Problem history: Different ships may con gure the same equipment differently, resulting in different failure trends. Previous problem information can provide valuable guidance for focusing diagnostic effort:
SME: Quite often some ship's systems, they like to eat a particular circuit card. That's just the way it is. Maybe the SME can say, Oh, I know. This ship always has a problem with this [circuit card]. So, we're going to have to look in this particular area [to begin troubleshooting]. 5. Environmental effects: Environmental conditions may make particular failures more likely, making it appropriate to look outside the prescribed procedures:
Sailor: Say you lose RF (radio frequency) on your port forward quadrant...The worst thing that is going to happen is on [an aircraft carrier], the cord antenna is right next to CAT4 [a catapult for launching aircraft], so there is a lot of vibration. ... So a lot of the time the cable .. will shake loose... 6. Pre-contact actions:
Sailors commonly start a dialogue by providing this context, informing the SME of what was tried and did not work. In practice, it is dif cult for SME's to obtain an accurate picture of this context. Because they do not necessarily trust sailor reports to be accurate, they may ask the sailor to repeat some tasks which should have been done pre-contact, to ensure that they were done fully and correctly. 4.3
The troubleshooting process is shaped by institutional, social and cultural factors as well. 7. Institutional regulations and responsibilities: Navy policies require the sailor to follow prescribed procedures in a set order. This context limits the sailor's autonomy but enables the SME to make some initial inferences about steps the sailor has already performed. However, it may also decrease the accuracy of sailor-provided information, if the sailor is reluctant to reveal a deviation from policies. 8. Social and cultural standards: The decision of when to seek unof cial help before contacting an SME depends on social factors:
Sailor: [Y]ou might want to ask your Chief or the DivO4, Hey, we have been at this for a while, can we ask the USS Whatever [for some assistance], because this sounds like a problem they had a couple of months ago. And if he wants too, then [we'll make a request], [but] maybe my Chief doesn't want the USS Whatever to know that our system is screwed up. It is a pride issue. 4.4
The expert's guidance to the non-expert, and the overall course of troubleshooting, are also shaped by external limitations in capabilities: 9. Environmental constraints: Adverse environmental conditions can limit the diagnostic actions an SME may request:
Sailor, e-mail message to SME: The weather is pretty bad right now so I can't go out topside to get into the transmitter... 10. Resource constraints: The sailor and SME must adjust instructions according to the resources and facilities on hand:
Sailor, e-mail message to SME: I would like to verify the power supply and HVDU but I don't know if Guam or Yokosuka [Japan] have any facilities that have the bench test equipment for power supplies and HVDU's. 11. Schedule constraints: Both sailors and SMEs operate under strict schedules. Thus expectations for their interactions depend on the prioritization of their tasks and their availability to pursue the task at hand:
SME, reassuring a sailor: I will keep checking my mail this weekend so I won't leave you hanging. 12. Higher-level priorities and goals: Repairs are prioritized, and the goals of troubleshooting may depend on external factors. For example, under emergency circumstances a quick temporary x for a crucial component may be preferable to a long-term repair which would require waiting for a new part. 5
Our study of current practice illustrates the rich range of context that can be useful to remote troubleshooting; all of these context types would be valuable within support systems for remote cooperative troubleshooting. It also shows that baseline support is needed, simply to help convey the explicit problem, as illustrated in the following exchange: SME: I am very confused as to what the exact symptoms are. I want you to go to High Voltage special functions and do the following [tests]... After you do this I will have a better understanding your problem. Then I will walk you through more instructions.
Our strategy in applying these lessons is rst to provide this basic support and then
to augment it with additional capabilities. The overall aim was a combined system for
knowledge capture, transfer, and sharing to aid communication between sailors and
technicians. We have developed a testbed system that captures and conveys information about
previous diagnoses based on a case-based reasoning (CBR) system
CBR systems capture knowledge in the form of traces of problem-solving episodes, and generate new solutions by adapting solutions for similar prior problems. These problemsolution pairs can in turn be stored as cases in the case base, to be used for future problem solving. In our system, an interactive variant of CBR known as conversational case-based reasoning (CCBR) guides users in diagnosis and suggests solutions. This is done in part by building up a case for the current problem, in which the system de nes the values of a case's attributes according to the actions of the sailor and other available data.
A major contribution of this work, compared to normal CCBR, is that cases are treated
as a vehicle for cooperative knowledge sharing rather than simply as a resource for a single
user. We previously proposed the use of cases for information ow downstream in a work
process
A fault is detected, by either a functionality failure or during periodic preventative maintenance.
A sailor follows standard procedures as presented by the system, helping to stabilize the sailor's context by assuring that the sailor will have actually carried out the prescribed troubleshooting steps. Associated data collection and actions taken are stored in a new case.
If standard procedures are exhausted without the sailor resolving the fault, the sailor contacts the SME via the system, which transmits to the SME the case describing the situation so far.
Using the SME interface, the SME examines the case so far, transmitted from the sailor interface, and the system automatically retrieves cases of similar past troubleshooting scenarios to present to the SME.
The SME uses similar past cases and personal experience to suggest additional measurements and actions for the sailor to take. The system suggests potentially-relevant questions from prior cases, which the SME can select to transmit.
The sailor receives and executes these suggestions, with the results added to the current case by the system as an evolving record of the sailor's steps and the sailorSME interaction. If the fault is still unresolved, the sailor sends the SME the updated case and the cycle continues.
As we applied cases to collect and transmit information about the explicit problem
description, we observed that casesbecause they capture speci c information about
multiple types of information within an episode
The non-expert's access to knowledge updates: Because a sailor may have missed or forgotten knowledge updates, our system aims to assure that the non-expert's understanding context will include relevant documents, by enabling the SME to attach relevant bulletins, etc., to problem cases for just in time presentation to the sailor as the sailor follows instructions or procedures.
The non-expert's training/experience: When the current case is generated, the system logs the sailor's actions, to assure that the case contains information that the sailor may otherwise forget or need to be prompted to provide. This decreases the gap when an inexperienced sailor must attempt to provide a useful description of the situation to the expert.
Communication from the SME to sailor is facilitated by enabling the SME to transmit questions from prior cases in a form that is integrated into the sailor's normal troubleshooting interface and that can contain detailed guidance to re ect the sailor's level of training.
The sailor's pre-contact actions: These actions are all logged automatically by the system and transmitted to the SME in the form of a case. Thus, the SME can have con dence in the accuracy of the case without spending additional time in veri cation.
The system also stabilizes the sailor context, by requiring the sailor to complete the standard actions in order to be allowed to proceed.
Institutional regulations and responsibilities: The sailor's activities are guided by the system and logged in a case, so there is less chance for deviation from of cial procedure. This case can then be passed to and trusted by the SME.
The following contextual factors are not currently addressed in our system, but could be addressed to some extent without signi cant additional development:
The expert's skill/experience: When a case is resolved and stored for retrieval in future problem solving, a record of the SME who solved the problem can be stored as part of that case.
History of the problem setting: Cases can include information on the ship and parts involved, as potential retrieval cues to obtain a history of similar problems speci c to a ship and/or part or for data mining for problem trends.
Higher-level priorities and goals: Descriptions of the immediate problem can be augmented with information about priorities in uencing the choice of repair, e.g., an emergency situation, to use an initial lter in retrieval to obtain quick- x cases. The capture of environmental factors and constraints, social and cultural standards, and resource constraints is a subject for future research and is likely to require integrations with other methods.
The issues addressed in this paper relate to studies of communication of contextual
knowledge, computer support for cooperative work, and case-based reasoning; ubiquitous
computing provides interesting future directions.
Our view of context in problem-solving has important commonalities with Bre´zillon's
approach in
A future direction for this work is to examine methods for increasing system contextawareness and automatic responsiveness. The contextual information captured in our cases is based on capturing traces of sailors and SMEs with information systems. While considerable contextual information is available in this form, signi cant opportunities exist for drawing on additional information sources. For example, as sensors are installed on devices, or as device information can be automatically captured during preventative maintenance, it may be possible to broaden the range of environmental and device information captured and transmitted automatically. Ubiquitous computing methods promise to facilitate the sailor's troubleshooting task and facilitate information capture. 8
Human problem-solving depends on numerous contextual factors. Consequently, when relevant contextual information is dif cult for users to obtain directly, systems that furnish them with this information can provide important bene ts. In remote cooperative troubleshooting by ad-hoc teams, participants may begin their tasks with limited knowledge of team members' contexts. Based on an extensive study of the types of contextual factors shaping performance for this troubleshooting task, we have illustrated the rich context that it involves. Our analysis of types of contextual information used identi es a number of opportunities for context transmission to bridge gaps between non-experts and the experts who attempt to aid their troubleshooting. We have sketched directions for responding to some of these opportunities and their application in an implemented testbed system which uses a conversational CBR interface to guide sailor actions and cases as a vehicle to help bridge contextual gaps between experts and non-experts. Informal feedback on the system from potential users has been encouraging. One next step is to perform controlled tests to assess the bene ts and tradeoffs of the system, both to contribute to further design and as a form of validation of the set of contextual factors identi ed as important by our naturalistic study. Another future step is to further exploit the study's results, by exploring how the additional types of contextual information identi ed by the study can be extracted from the work ow for this task and presented when needed.