In oil and gas sector, maintenance reports from a oating oil and gas production, storage and o oading (FPSO) unit play an important role for xing the failures on time and for learning from previous experiences. Documenting these reports in a structured way is an important task for such units. In this work, we experiment with di erent machine learning techniques to classify the maintenance reports. However the unstructured text from these reports combined with domain speci c terms, abbreviations, misspellings and multiple language use makes the automatic classi cation of these reports quite challenging. Our ndings show that how challenging this task is with the existing report structure and content.
O shore oil & gas installations are complex structures that are expensive to build and need continuous monitoring for maintenance and upgrading. It is vital to avoid any unnecessary production stops or accidents, as this would have serious implications for both the companies themselves and their sta . Financially just a short temporary halt of production comes at a substantial cost. More importantly, however, is the risk of serious life-threatening accidents if parts break down or malfunctions arise. O shore oil & gas production is by nature a challenging task, and the dependence on technology calls for proper procedures for reducing the risk of equipment failures and other accidents that a ect the security of people and climate.
Maintenance reports document maintenance activities that are either carried out on a regular basis or triggered by particular events. Whereas some of them address small repairs like the change of broken light bulbs or screws, others concern more time-critical problems that need to be taken very seriously by the management. The intention is to use these maintenance reports to structure the documentation of these activities for later use and facilitate learning and exchange of competence over time. Engineers acquire valuable experiences as they
Copyright held by the authors. NOBIDS 2018 take part in maintenance activities, but not all of them are equally experienced and no individual engineer can be expected to fully understand the structures and functions of a complete oil & gas installation.
Maintenance reports follow a particular format that forces the engineers to provide some basic information about the maintenance work. There are a number of structured elds that identify for example a piece of equipment, a location or a date, but the engineers are also encouraged to write a short textual description of the work they are carrying out. Even though these descriptions are fairly short and of variable quality, they are crucial when someone wants to check if their present problem has been faced and solved at some time in the past.
Failure codes are of particular interest in maintenance reports. Unfortunately, many reports do not have any failures codes entered, since they were not known at the time the report was created. For later maintenance activities this is problematic for two reasons: (i) A maintenance task is not treated correctly or timely because they fail to realize that similar tasks have been successfully dealt with in the past, and (ii) they are not able to generalize - or learn - from previous cases because a critical piece of data is missing in the documentation of old cases.
In principle, we can approach the missing failure codes as a classi cation problem. Experiments show, unfortunately, that the structured elds of maintenance reports are not su cient to produce a classi er with a very high accuracy. The question is whether the textual descriptions may help us ll in the failure codes instead. These texts were not written to support later classi cation and are probably quite typical for internal documentation in many companies. They were quickly written and with limited knowledge of the problem at hand, and they are best understood by their fellow engineers. The descriptions are neither precise nor complete, but they do reveal something that is potentially relevant for classifying the maintenance work. The contribution of this paper is two-fold: (i) we analyze the challenges of extracting the content of internal industrial texts and assess the usefulness of various linguistic operations, and (ii) we evaluate to what extent machine learning techniques can classify the maintenance reports on the basis of their textual descriptions.
The structure of the paper is as follows. After presenting the dataset in Section 2, we discuss the methods for analyzing the textual descriptions in Section 3. We then use some machine learning techniques to classify the maintenance reports and discuss the results of the experiments at the end. 2
The dataset used in this work stems from a oating production, storage and o oading (FPSO) unit in the Norwegian oil and gas industry in the North Sea. It contains a total of 4968 failure reports on various maintenance activities in the oil and gas eld. Failure reports are labelled with di erent code groups according to their association with di erent parts of the FPSO unit, such as electric generators, steam turbines etc. In the dataset there are 40 code groups. In each code group there are various fault types where each report can be associated with. However the fault type eld that may or may not be lled by the reporter. In the dataset the fault type eld is not consistently lled. In general, the reports consist of a number of structured elds and one text describing the failure. In Fig.1 the general structure of the failure reports can be seen.
Report texts describe asset failures from a FPSO unit in the North Sea. The average length of the reports is 75 words, where the minimum length is 7 and the maximum length is 1580 words. An interesting property of the reports in the dataset is that the usage of multiple languages. As it is shown in Fig.2, the languages used in the reports are mostly Norwegian (58.7%) and English (22.6%), however it is possible to see the usage of Danish (3.7%) and Swedish (0.4%) as well. In addition, in 13.2% of the reports multi-language use occurs within the same report. In total there are 32,011 unique words in the reports.
Dealing with the automatic analysis of the free, unstructured text has challenges. When we consider the technical failure reports at FPSO units, the data comes with even more challenges: { Misspellings: In the failure reports there are several misspelling of words and abbreviations in addition to the usage of incorrect grammar and un nished sentences. { Domain speci c terminology and jargon: The usage of domain speci c terms and acronyms is available in the failure reports. { Multi-language use: As mentioned earlier, 13.2% of the failure reports has the usage of multiple languages at once. { Abbreviations: For the ease of reporting there are several abbreviations available in the reports. Several of those abbreviations are not explained and it is only possible for the human domain experts to understand. { Non-standard structures: The structure of reports is not standardized.
Even though it seems that there is a form of structure in some of the reports, many of the reports do not follow the same structure. { Imbalance in report types: The distribution of 4968 failure reports among 40 di erent code groups is not balanced. As it is shown in Fig. 3 the di erence between the distribution of reports vary a lot.
3.1
Prior to conduct the classi cation experiments we conducted some normalization of the texts. This normalization includes some text simpli cation strategies as well as general NLP pre-processing techniques.
Sentence boundaries were standardized as much as possible, in order to enhance the performance of the classi ers. In particular, newlines and non-standard sentence boundaries were normalized by substituting the symbols with a basic dot. Regular expressions were used to do these transformations.
Besides, most reports include speci c dates and times describing when the report was made or edited and by whom. This information was also transformed into general named entities labels so as to reduce the vocabulary space the classi er has to deal with as much as possible. Figure 4 shows an example of an original maintenance report and Figure 5 illustrates the same text after simpli cation. This example illustrates how actual dates, times and proper names for equipment, location, people and measurement numbers, were mapped into general labels during normalization. Protocol questions included in most reports as templates for employees were also removed, with the purpose of excluding unnecessary information.
After these simpli cation, standard NLP pre-processing was performed, including tokenization, normalization of words with capital letters, stop-words removal and stemming, whereby words are reduced into their base form. NLTK was used for these purposes. Two di erent feature models were explored in this study. A baseline model, consisting on a bag-of-words model including TF-IDF frequencies, and two word embedding models, in particular distributed memory (DM) and distributed bag of words (DBOW) were explored. Gensim and scikit-learn were correspondingly used for these purposes.
The approach for obtaining such features varies slightly. TF-IDF models are tted individually with text data from each code group or class. In contrast, document embedding models are tted with the text data from all classes, and then each of the selected class infers embeddings from the same global model. Text classi cation is a fundamental eld that has been well studied by researchers and practitioners of a variety of elds, including arti cial intelligence, statistics, pattern recognition, cognitive psychology, computer vision and medicine. In this section, we rst formally de ne the classi cation task. Then di erent types of classi cation algorithms, speci cally, K nearest neighbor, Multinomial Naive Bayes, Random Forest and Support Vector Machine, are adopted and explained.
Problem de nition Let X = fx1; x2; ; xNg be the input set after pre-processing, and y be the output being assigned to a class c 2 C after generated by inference model f . xi is the input vector of the i-th document, and N represents the number of documents in training set. The classi cation problem can be denoted as: y = f (X ). Our goal is to assign y to a class c which is the same with ground truth class c.
K Nearest Neighbor The fundamental mechanism behind nearest neighbor method is to nd K training samples which are closest in distance to the newly arrived input point [4]. The distance can be any types of metric measurements according to speci c applications. In this paper, euclidean distance is adopted to nd the closest K points in training set.
vu D x)jj = tuX(xi2
i
d(x; x) = jj(x
xi2)
(
Naive Bayes Naive Bayes classi er classi es newly input documents using Bayes rule under the assumption of the independence between input features [7]:
D
y^ = argmacc2Cp(c) Y p(xijc) (
p(xjc) = (Pi xi)! Y pcxii
Qi xi!
i
(
Inf ormationGain(Xp; xi) = H(Xp) m X Ni H(Xj ) j=1 Np Support Vector Machine Support Vector machine (SVM) is another method that we use to categorize fault type labels of text descriptions [5]. SVM sets out to t a hyperplane with largest margin to nearest examples of any class. These examples are known as support vectors. The optimization process of SVM is to support vectors with largest margin between arbitrary classes while minimizing the lost function L. In our paper, the extension version of hinge loss [13] is adopted to achieve the best margin. Speci cally, for
L = X max(0; WjTx
WcTx + ) +
R( ) j6=c where Wj and Wc are model parameters. is the tolerance threshold to be tuned during training process. R( ) represents the regularization term used to prevent over- tting problems and model complexity, which can be either L1 or L2 regularization. represents the set of model parameters and is the regularization parameter. 4
Di erent experiments run in di erent phases: 1- NLP techniques in preprocessing 2- Machine learning algorithms for classi cation 4.1
Here we enumerate the evaluation metrics on classi cation tasks adopted in this paper. The formulae are shown in Table 1.
{ Accuracy (ACC): Despite its popular property in evaluation of classi cation tasks, some issues occur in an imbalanced domain. For instance, consider a problem where only 1% of the examples belong to the positive class, a accuracy of 99% is achievable by predicting negative class for all examples. Thus, ACC is at best misleading, when the ratio between positive and negative examples is signi cantly skewed. { Harmonic Mean of Precision & Recall (F ):The de nition is based on two metrics, recall and precision. Recall captures the completeness of how many positive examples are predicted positive, while precision captures the exactness of how many predicted positive examples are true positive. Because of an imbalanced multi-class data distribution in our experiments, we adopt macro averaging in [10] to summarize the classi ers performance over a set of classes C. Thus, the example size of a class does not a ect the measurement. For a given class c, F is de ned as the harmonic mean of precision and recall [12]. is introduced to adjust the relative importance of recall with respect to precision and is set to 1 in our case. { Cohens Kappa ( ): It compares an observed accuracy with and expected accuracy [3]. Speci cally, provides a score ranging from -1 to 1, from which the performance of a classi er can be evaluated with respect to random chance. The bigger the value is, the better the performance shows. The interpretation of score follows the work in [1]. { Geometric Mean Accuracy (GMA): The metric takes the geometric average (GMA) of the recall score in each class and it bene t from the use of geometric mean when there is a multiplicative or exponential relationship between class scores averaged especially for imbalanced data distribution [11].
Besides, two guessing strategies are chosen as baselines: { Strati ed (STRAT): It generates predictions by respecting training sets class distribution. { Most Frequent (MFREQ): It always predicts the most frequent label in the training set. Hyper parameters are selected through grid-search for di erent tasks and classiers. Then values with best performance of classi cation tasks are selected. Each sampled experiment for a particular feature extraction method will be scored in terms of its exhibited F macro averaged score with K = 5 in K Nearest Neighbor predictions.
where tp, tn, f p and f n denote true positive, true negative, false positive and false negative respectively P recM = Pc2C pjrCejcision(c) where precision(c) = . RecM = Pc2CjCrjecall(c) where recall(c) = Pi PI(yiiI=(cy)iI=(cy^)i=c) where po denotes the observed accuracy and pe denotes the expected accuracy of classi cation performance
As described before, there are several properties in oil and gas maintenance reports which make them particularly challenging to be automatically understood: specialized terminology and expressions, multiple languages, misspelled and nonstandard words, inconsistent syntax and lack of proper sentence boundaries.
In order to assess the impact of the di erent simpli cation and pre-processing techniques, as well as feature extraction models, a careful evaluation was performed. This evaluation was implemented in three steps: (i) assessment of the feature extraction methods; (ii) evaluation of singular simpli cation and preprocessing techniques; (iii) evaluation of simpli cation and pre-processing methods in combination. 5.1
As results from experiments with each of the feature extraction methods show, each method performs di erently on di erent models. Figures 6, 7 and 8 show the distributional graphs containing all the models sampled during hyperparameter search. In these gures, the vertical axis represents the total number of models that achieved any of the F scores along the horizontal axis. Note that axis scales are not consistent among the graphs.
The graphs show that the performance of document embeddings (DM and DBOW) varies signi cantly. Although TF-IDF show a more consistent performance across various hyperparameters, and exhibited the best performance of all the three methods, it also includes the worst performing model. the performance of classi ers with person-name ltering. Also it is possible to see that di erent classi ers perform better on di erent experiments.
Table 10 shows the results of the classi ers when performing standard NLP pre-processing. Interestingly, the SVM experiment including stemming yielded the best results, while the worst results were obtained for the MNB classi er under the same conditions. The reverse can be seen with non-alphanumeric ltering. In general, these pre-processing methods had a much a bigger impact on the performance of SVM, yielding results above the baseline. In contrast, MNB got results over the baseline only when non-alphanumeric ltering was performed.
Lastly, we evaluated the performance of the di erent pre-processing methods in combination. The top ve performing combinations of pre-processing methods for SVM and MNB can be seen in Tables 11 and 12.
qs: question structure ltering; sw: stop-words ltering; l: lowercase (no capitalization letters); st (stemming); dnt: datetime-number-tag ltering; nlta: non-alphanumeric ltering; dt: datetime ltering
Although there is no consistent ranking between the two classi ers, however, lowercase, stop words ltering and non-alphanumeric ltering appear amongst the techniques with most impact on their performance. This is not surprising, as these techniques are generally used as pre-processing steps in most NLP pipelines. Slightly more unusual is the fact that stemming does not seem to be very relevant for the classi cation algorithm's performance. We suggest that this might be due to the fact that there is a high number of spelling errors and specialized vocabulary which can make the vocabulary space simpli cation more challenging. 6
In this work we looked into the maintenance reports from a oating production, storage and o oading (FPSO) unit in the Norwegian oil and gas industry. These reports are used to structure the documentation of these activities for later use and facilitate learning and exchange of competence over time.
Failure codes which some of the reports in the dataset include, help the
engineers to classify the type of failures in order to treat the failures on time
and to learn from the similar failures happened in the past. However, not all the
reports include failure codes. In this work, we use machine learning in order to
predict the missing failure codes. By using the textual description of the failure,
after pre-processing of the text and feature extraction, we experimented with
three di erent machine learning classi ers: K Nearest Neighbor, Naive Bayes
and Support Vector Machine. As explained in the previous sections, the results
show that di erent classi ers outperforms other methods on di erent tasks. But
unfortunately none of the methods are good enough to be useful in this task
and tested methods do not solve challenges. The challenges of the dataset, as
explained in detail in Section 2.1, makes data preparation and analysis very hard.
Maintenance reports have special characteristics. Usage of multiple languages at
once, unstructured text with misspellings, technical terms, abbreviations are
quite challenging for the existing natural language processing methods to deal
with.
6. David D Lewis and William A Gale. A sequential algorithm for training text
classi ers. In Proceedings of the 17th annual international ACM SIGIR conference
on Research and development in information retrieval, pages 3{12. Springer-Verlag
New York, Inc., 1994.
7. Andrew McCallum, Kamal Nigam, et al. A comparison of event models for naive
bayes text classi cation. In AAAI-98 workshop on learning for text categorization,
volume 752, pages 41{48. Citeseer, 1998.
8. Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel,
Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss,
Vincent Dubourg, et al. Scikit-learn: Machine learning in python. Journal of
machine learning research, 12(Oct):2825{2830, 2011.
9. Jason D Rennie, Lawrence Shih, Jaime Teevan, and David R Karger. Tackling
the poor assumptions of naive bayes text classi ers. In Proceedings of the 20th
international conference on machine learning (icml-03), pages 616{623, 2003.
10. Marina Sokolova and Guy Lapalme. A systematic analysis of performance measures
for classi cation tasks. Information Processing & Management, 45(