Troubleshooting search system aims at extracting relevant information to solve the problem at hand. It is often the case that documents in troubleshooting system includes an abundant amount of domain-specic categories. However, the useful information about the domain-specic categories, such as relationship between words and categories and relationship between categories, is not fully utilized in simple query search and faceted search. In this paper, we propose an information retrieval method boosted by the domainspecic categories. Given a problem query and categories, the troubleshooting search system is able to retrieve the relevant information of interest with respect to the selected categories. The experiment results show our proposal improves the recall.
Troubleshooting [
The target of information retrieval for troubleshooting [
SIGIR Workshop on Graph Search and Beyond ’15 Santiago, Chile
Published on CEUR-WS: http://ceur-ws.org/Vol-1393/ .
ine the similar situations of a problem within a certain
period of time, which facilitates an appropriate solution. The
troubleshooting search system requires the engineers to
input a short problem query to search the relevant information
from the documents. It may cause the lexical gap problem
[
Moreover, information about the domain-specic categories is not fully utilized in the problem query search. One way to use the category information is to make faceted search, in which the selected categories are used to lter the ranking results. For example, if a machine code is selected, all search results are restricted to the selected machine code.
However, the faceted search might have two problems in the troubleshooting search system. First, the information related to the selected categories can not be retrieved, since the retrieved information is limited to the selected categories. However, it is natural that system engineers tend to check relevant problems to facilitate their decision making. For example, given a selected machine code, the information about another machine code might be informative to solutions if two machine codes belong to the same machine series and have similar problems. Second, the ranking of search results is only dependent on the problem query but not on the selected categories. For example, a trouble code corresponds to a number of specic countermeasure codes. Given a selected trouble code, the information about its frequent countermeasures is expected to place higher in the ranked list of results.
To mitigate the lexical gap problem and the above
mentioned retrieval problems, we propose an information
retrieval method using a scoring technique. Our proposal
is extended from a word co-occurrence graph in the QSB
method [
The information retrieval for troubleshooting is related to
question answering [
Query Snowball (QSB) is a method for multi-document summarization that extracts the relevant sentences from multiple documents with respect to the given query. The basic idea of this method is to ll up the lexical gap between the query and relevant sentences by enriching the information need representation. In order to achieve it, a co-occurrence graph for the words in the queries and the documents is built. The words in the co-occurrence graph consist of three layers, which are Q words, R1 words, and R2 words. Q is the set of query terms. R1 is the set of words that co-occur with a query term in the same sentence. R2 is the set of words that co-occur with a word from R1, excluding those that are already in R1.
To extract relevant information with respect to the selected categories, we extend the co-occurrence graph in QSB by integrating two types of relations, including the relationship between words and categories and the relationship between categories. The reason to extend QSB is that the co-occurrence graph is exible to integrate the two relations.
The relationship between words and categories represents the distribution on words with respect to categories. If probabilities of words with respect to a given category are high, the information about these words is more likely to be retrieved given that category. The fundamental idea is that the distributions on co-occurrence probabilities of words with respect to dierent categories might be dierent. It is assumed that the words of higher probabilities with respect to a category are treated more important in that category. For example, a word appears more often in documents of a category than other categories. The word is therefore more important for that category than the others. Similarly, the relationship between categories represents the occurrences of categories. The information about categories, whose occurrence frequencies with respect to a specic category are high, is more likely to be retrieved. For example, a given category appears more often with specic categories than the others. The information about those specic categories with respect to the given category is treated more important than other categories. 3.1
In the rst step, we build two co-occurrence graphs for the words and category values in the problem queries and the documents, respectively. If a category value is associated with a document, the words in the document have edges with the category value. In other words, this category value is associated with all the words in the sentences of documents. Similarly, if a category value is associated with a problem query, the words in the problem query have edges with the category value. In other words, this category value is associated with all the words in the problem query.
In the second step, we build the co-occurrence graph for the category values in both problem queries and the documents. Suppose that a problem query corresponds to a document. In this graph, the category values associated with the problem query have edges with the category values associated with the document. As illustrated in Figure 1, the left side represents the co-occurrence graph for the R1 and R2 words and the category values associated with the documents; the middle part represents the co-occurrence graph for the Q words and the category values associated with the queries; the right side represents the co-occurrence graph for the category values in the queries and the category values in the documents. We dene CM as the set of the category values in the query set that are selected by the end-user, and CN as the set of the category values in the query set that are not selected by the end-user. Similarly, we dene CJ as the set of the category values in the document set that are selected by the end-user, and CK as the set of the category values in the document set that are not selected by the end-user. We also dene CMN = CM [ CN as the set of the category values for the queries and CJK = CJ [ CK as the set of the category values for the documents. where qsb(w) is the score for a word w in the QSB method. sctg(w) is the boosting term for the word w, which includes two new relationships about categories. More specically, the probabilities between words and categories and the probabilities between categories, which are calculated through the co-occurrence graph, are used to boost the score qsb(w). λ is a weight for the term sctg(w). It can be seen from Eq. (1) that when sctg(w) is larger than 0, exp(¢) will be larger than 1. When multiplied with qsb(w), it will give the word w a higher degree of importance. If the value of λ is set to be 0, sctg(w) does not take any eect. Note that sctg(w) is always larger than or equal to zero. The score of a sentence is a summation of the scores for any combinations of two words, which is simply calculated by multiplying the cqsb scores of the two words. 3.3
The relevant score of a word r1 (r1 2 R1) with respect to the category values can be formulated as follows: sctg(r1) = swc(r1, QrC1M ) + scc(r1, QrC1M ) (2) where swc(r1, QrC1M ) measures the relationship between the words and the categories. scc(r1, QrC1M ) measures the relationship between the categories. QrC1M is a set of top k query terms that co-occur most frequently with the word r1. In addition, the words in QrC1M follow a constraint that they should have edges with both the word r1 and the category values in CM .
The term swc(r1, QrC1M ) in Eq. (2) can be calculated as: swc(r1, QrC1M ) =
X q∈QrC1M
i=1 |CMqN | f req(ci, q) X θ f req(ci) q where CMN is the set of category values for the queries that also have edges with q. Let θ be β if ci 2 CM and ci 2 CMqN , and be 1 ¡ β if ci 2/ CM otherwise. f req(ci, q) is the frequency of sentences that include both ci and q, which can be also represented by the distribution on ci with respect to q. It can be seen that Eq. (3) measures the closeness degree between the word r1 and the category values in CM through the words in QrC1M , since the word r1 does not directly have edges with the category values in CM in the co-occurrence graph.
The term scc(r1, QrC1M ) in Eq. (2) can be calculated as: scc(r1, QrC1M ) = q∈QrC1M θ∈Γ ci,cj
X X X θ f req(ci, cj)
f req(ci) J q when θ = β. where Γ = fβ, 1 ¡ βg. ci 2 Cr1 and cj 2 CM Note that Cr1 is a set of categories in CJ that also have
J q edges with the word r1, and CM is a set of categories in CM that also have edges with the word q (q 2 QrC1M ). Let ci 2 CJ−Kr1 and cj 2 CM−qN when θ = 1 ¡ β. Note that CJ−Kr1 = CJr1K ¡ CJr1 and CM−qN = CMqN ¡ CMq . CJr1K is a set of categories in CJK that also have edges with the word r1, and CJqK is a set of categories in CMN that also have edges with the word q (q 2 QrC1M ). It can be seen that Eq. (4) measures the closeness degree of the category values in Cr1 J q . and CM 3.4
Similarly, the relevant score of a word r2 (r1 2 R2) with respect to the category values can be formulated as follows: sctg(r2) = swc(r2, QrC2M ) + scc(r2, QrC2M ) (5) where swc(r2, QrC2M ) measures the closeness degree between the word r1 and the category values which have edges with q words. scc(r2, QrC2M ) measures the closeness degree between the category values in the query set and the category values in the document set, which are respect to the word r2. QrC2M represents a set of query terms q (q 2 Q) that have close (3) (4) relationship with the word r2. Since the word r2 does not have edges with the Q words in the co-occurrence graph of the word-word relation, the measurement of the relation could be done through the R1 words by using the frequency, such as f req(r1, r2) and f req(r1, q). An intuitive example of the measurement is a multiplication of f req(r1, r2) and f req(r1, q) for the word r2 and the word q. The word q (q 2 QrC2M ) also holds two constraints that the word q is able to reach the word r2 in the co-occurrence graph through a specic word r1 and the word q should have edges with the category values in CM .
The term swc(r2, QrC2M ) in Eq. (5) is calculated as: swc(r2, QrC2M ) =
X r1∈Rr12 f req(r1, r2) swc(r1, QrC2M ) sumRr12 (6) where Rr12 represents a set of R1 words which have the top k highest frequencies with the word r2, and sumRr12 = Pr1∈Rr12 f req(r1, r2). The term swc(r1, QrC2M ) can be calculated by Eq. (3).
The term scc(r2, QrC2M ) in Eq. (5) can be calculated through Eq. (4) by substituting QrC1M with QrC2M . 4.
In this experiment, we use the maintenance reports from a leading construction machinery company in Japan. We collect a part of the maintenance reports of 4 dominated troubles from total 19 troubles. Note that equipment code is the category for the problem queries. Phenomenon code and the countermeasure code are the categories for the documents. In each data set, one query consists of a problem query, model code, phenomenon code, and countermeasure code. We also manually label the important sentences from the documents. For each query, we search the sentences in the documents, and evaluate the performance by comparing if the sentences in the top rank are matched with the labelled sentences. Note that precision, recall and F-score are used as criteria. Among the three metrics, recall is the most important criterion. The reason is that, in troubleshooting system, system engineers prefer to examine all similar cases until they feel condent to solve the problem at hand. In this experiment, the training data and the test data are the same, since system engineers nd out similar cases in the past from the troubleshooting system. Note that building and updating the co-occurrence graph can be done periodically in an unsupervised way.
For the comparisons, we select four baseline methods,
which are cqsb, qsb, lexsim, and lexsim+qsb. We name our
proposal by lexsim+cqsb. lexsim represents the lexical text
similarity between the problem query and the sentence in
the comments, which can be simply calculated by the cosine
similarity between two vectors with bag-of-words features.
One example of the bag-of-words features is the counts of
frequent words in documents. Note that stop words are
removed when counting the frequencies of words. lexsim+qsb
aggregates lexsim and qsb to obtain the nal ranking list,
which belongs to the rank aggregation problem [
0.9 0.8 0.7 0.6
As recall is the important criterion in troubleshooting search system, we calculate Macro Recall for each data set by setting the top l (l 2 f5, 10, . . . , 195, 200g) ranking sentences. Figure 2, Figure 3, Figure 4, and Figure 5 show the Macro Recalls of all the methods, when the number of the top l sentences changes from 5 to 200. It can be seen that the performances of lexsim + cqsb are better than other baseline methods. It is also noticed from Figure 5 that the performance dierences between qsb and cqsb are not obvious when the number of top sentences is increasing. The reason might be that the probabilities of words in informative sentences with respect to categories and the co-occurrence probabilities of categories are evenly distributed. Therefore, the second term on the right side of Eq. (1) does not dier over words to a large extent.
We also investigate MAP and F 3 score. For simplicity, we show their results in cases in which a best result and a worst result of Macro Recall for lexsim + cqsb are achieved. The results are illustrated in Table 1 and Table 2, which are from the trouble code 03 data set and the trouble code 05 data set, respectively. Note that β and the number of the top l sentences are set to be 1 and 100, respectively. It is shown that lexsim + cqsb outperforms lexsim + qsb, cqsb, and qsb in both cases. It is also noticed that, in the worst case, even if Macro Recall of lexsim + cqsb is better than the others, its F3 score is lower than that of lexsim. Note that F1 has the same trend as F3 in this experiment.
We also check the eect of the parameter β, as it inuences the score about the relationship between the categories. Table 3 shows the macro recalls of cqsb at dierent values of β and l (l 2 f100, 200g) in the data sets of trouble code 3 and trouble code 05. It is implied that 0.8 and 0.6 might be good values for cqsb to improve the recalls.
An information retrieval method using the scoring technique boosted by the domain-specic categories is proposed for troubleshooting search system. The knowledge about category information, which includes the relationship between words and categories the relationship between categories, is well integrated into a co-occurrence graph. The experiments on the maintenance logs proved the improvement of recalls, showing the eectiveness of using the category information.