Troubleshooting in the telecommunication industry is a time-consuming task, often involving text understanding, which is challenging to automate due to its domain/company-specific features. This work aims to build a model to retrieve solutions for newly reported problems in automated and quick ways. To this end, we present BERTicsson, a BERT-based model that uses two main stages for (i) retrieving a shortlist of candidate answers for new problems and (ii) raking them accordingly. We study the performance of BERTicsson using Ericsson's troubleshooting dataset and show that it significantly improves the accuracy of the recommended answers compared to non-BERT models, such as BM25.
Troubleshooting in modern telecommunication systems is a slow process [1]. Many failures lead to service downtime or other forms of harm to the customer experience; thus, they must be quickly detected, categorized, and resolved. Usually, when engineers observe a fault in a running system that they cannot solve on site, they create a Trouble Report (TR) to track the information regarding the detection, characteristics, and an eventual resolution of the problem [2].
Figure 1 shows a typical troubleshooting process that Figure 1: The steps of a troubleshooting process. has six steps. The process starts by detecting a problem in step 1, followed by reporting it as a TR during steps 2 and way using NLP models. To this end, we present BERTics3. Then, this TR is analyzed and corrected in steps 4 and son, a multi-stage model to automate steps 4 and 5 of the 5, and finally, the proposed solution is verified by engi- process in Figure 1. BERTicsson is a recommender and neers in step 6. Steps 4 and 5 often involve understanding text-ranking system based on BERT [3] that receives a textual data in TRs, which can be challenging to auto- history of TRs (i.e., old problems and their solutions) and mate due to its domain-specific and company-specific newly reported problems as input and returns a ranked features. Moreover, the text may contain many abbre- list of recommended solutions to each problem. Both viations, typos, tables, and numerical data, making the TRs and new problems description are given in natural process more dificult. However, with today’s impressive language. development in machine learning, specifically in Natural Upon receiving input data, BERTicsson initially cleans Language Processing (NLP), we can shorten the process them at the pre-processing stage, and then, through the by analyzing historical TR data and infer a solution to Initial Retrieval (IR) stage, it retrieves a candidate list new problems faster and more accurately. of answers relevant to the problem. Finally, at the Re
In this work, we tackle the problem of retrieving so- Ranker (RR) stage, it ranks the candidate list provided lutions in the troubleshooting process in an automated by the IR stage concerning the problem. In the IR stage, BERTicsson uses Sentence-BERT [4], a variant of BERT, to make a representation of the problems and answers, and in the RR stage, it takes advantage of monoBERT [5], a two-input classification BERT model.
We use Ericsson’s 4th Generation (4G) and 5th Generation (5G) TR dataset through the experiments and show that BERTicsson provides a high accuracy while keeping the latency low. We compare BERTicsson with BM25 [6], a popular ranking model, and show that BERTicsson improves the Recall of the recommended answers by around 65%.
dered set of texts retrieved from a corpus of documents in response to a query for a particular task. This section reviews some of the pre-BERT and BERT techniques for text-ranking. task and uses dynamic masking by changing the masked tokens during the training epochs, ALBERT [14] reduces 2.1. Pre-BERT Text-Ranking Methods the number of parameters of the original BERT model by Before the appearance of BERT [3], the Exact Match- implementing the cross-layer parameter sharing, Distiling (EM) [7] and neural information retrieval tech- BERT [15] uses distillation to pre-train BERT, and ELECniques [8, 9, 10, 11] have been the primary text-ranking TRA [16] pre-trains a discriminator that has to distinmethods. EM solutions mainly rely on the term frequency guish if the sentences forwarded to the model have a and document frequency, where the former shows the replaced token or not. number of times that a term occurs and the latter shows The first attempt of using BERT for text-ranking adopts in how many documents it appears. A query and a docu- the BERT architecture to make a relevant score between ment have a high score if they both use the same terms. the query and a document [17]. The input to this model Indeed, this limits the algorithm’s applicability and its is a query and a document, and the output is a contextual performance in case of having vocabulary mismatch prob- embedding of them, which is used for deriving the similem [12], i.e., when the query and the documents use larity score between them. BERT-based models, however, diferent words to refer to equivalent things. present some limitations, as they only accept sequences
The pre-BERT neural information retrieval methods of less than 512 tokens, which limits the length of input are divided into two main architectures: representation- text. They also present high complexity and are slow in based approaches and interaction-based approaches. A text-ranking tasks. representation-based model learns a dense vector repre- Traditional text-ranking approaches have one-stage sentation of the query and the documents independently. architecture. However, the state-of-the-art models follow Then, it computes the similarity between the representa- a multi-stage architecture (Figure 2), usually formed by tions using cosine similarity or inner product and ranks two main stages: (i) an Initial Retrieval (IR) stage that the documents accordingly for the given query [8, 9]. On extracts a candidate set of text, and (ii) a Re-Ranker (RR) the other hand, an interaction-based model focuses on the stage that creates a ranked list of the candidate set. The interaction between each query’s and document’s terms, IR stage is fast, allows the model to discard easy candiand a similarity matrix is created. This matrix undergoes dates, and passes a smaller set of documents to the RR further processing to extract a similarity value [10, 11]. stage. The RR stage is slow, but as the input of documents is significantly smaller, it maintains low latency.
The number of documents that pass from IR to RR can 2.2. BERT-based Text-Ranking Methods be defined in diferent ways, such as keeping a certain BERT [3] is a language embedding model that learns con- number of documents with the highest scores or keeping textual representations of words in a sentence. BERT is a fixed percentage of all documents. pre-trained on a large corpus of text in an unsupervised setting, with two diferent learning objectives: Masked 3. BERTicsson Language Modeling (MLM) and Next Sentence Prediction (NSP). In MLM, some words in the input sentence are We aim to make a model for retrieving accurate solutions hidden from the model, and the model should predict to newly reported problems in telecommunication with the original word of the masked token based on the con- low latency. To this end, we present BERTicsson (Figtext of the other non-masked words in the sentence. In ure 3), a recommender and text-ranking model that is NSP, BERT receives two sentences, A and B, as input, composed of three stages: (i) the pre-processing stage and it should predict whether sentence A is followed by that cleans the input TRs (i.e., the new problems (queries) sentence B. and the corpus of documents), (ii) the Initial Retrieval
Recently, several variations of BERT have appeared. (IR) stage that retrieves a top- candidate list of docuFor example, RoBERTa [13] removes the NSP pre-training ments with answers relevant to the query, and (iii) the Re-Ranker (RR) stage that ranks the top- candidate list provided by the IR and outputs the final list of ranked documents. In the rest of this section, we present these three stages in depth.
RR stages. The input TR is a newly reported problem and corpus of documents containing telecom/companyspecific language, and the output is the query and the documents ready to be given to the IR and RR stages.
The pre-processing stage has five steps:
The IR stage gets the pre-processed query and docu- that consists of two neural networks with shared weights. ments, and after analyzing them, creates a top- candi- Each of these two branches has two layers: a BERT layer date list of the most relevant documents (answers) to the and a mean pooling layer (Figure 5). The Sentence-BERT query. The candidate list should have all relevant docu- receives two sentences as input: query and document ments to the query (if possible); however, the order of the . First, it tokenizes and and forwards the tokens to documents in the candidate list is not important at this the BERT layer to create a contextual embedding of each stage, meaning that the documents with the most rele- token. Next, the BERT layer output is given to the mean vant answers do not need to be at the top positions of the pooling layer to create the fixed-size representation of list. The IR stage should have low latency to quickly man- each sentence. Once we have the representations of and age a large amount of data. We use Sentence-BERT [4] to , denoted by and , respectively, we use Equation 1 build this stage. Sentence-BERT is a representation-based to measure their similarity value. The similarity value model that creates dense vectors and as embeddings will correspond to the ranking score and be used to create for the query and the document , respectively (Figure the top- candidate list.
We can use any of the BERT models in Sentence-BERT; however, we use DistilRoBERTa [15] in our implementation. Sentence-BERT behaves diferently in the training and inference phases. During the training phase, it receives a batch of pairs of relevant queries and documents, and it then adjusts its weight to represent queries and documents similar if they are relevant to each other.
However, this is a time-consuming task; thus, after the training phase, we store the representation of all the documents (only the documents, not the queries). Therefore, at the inference phase, we only need to compute each query’s representation and compare it with the stored representation of the documents, which is fast.
date documents relevant to and returns the final list of top- ranked documents, such that ≤ (Fig- In this section, first, we describe the structure of the ure 3). In this stage, we use monoBERT [5] (Figure 6). dataset (TRs) and the input and output of the model and The input to monoBERT is composed of a query and then present the training and the inference process. a document , and the special tokens [SEP] and [CLS] in form of “[CLS][SEP][SEP]”. Once monoBERT tokenizes 4.1. Data the input sequence, it forwards them to the internal BERT to create the contextual embeddings for all the tokens. To train and test BERTicsson we use the Ericsson trouNext, it forwards the embedding of the [CLS] token to a bleshooting dataset consisting of finished 4G and 5G single linear layer that outputs a scalar value indicating radio networks TRs over the past year. The language the probability of the document being relevant to the of these TRs is telecom/company-specific, which is very query . diferent from a general-domain text. Generally, each TR
We choose monoBERT for the RR stage due to its high in the dataset has the following fields: accuracy. However, the main drawback of monoBERT is its latency. To overcome this issue, we limit the length of the candidate list passed through the IR stage (it just sends the top- candidates of documents to the RR rather than the whole list of documents). Moreover, as Figure 6 shows, we use ELECTRA [16] inside monoBERT in our implementation. The training phase of RR is timeconsuming, as monoBERT receives pairs of queries and documents from the whole corpus of the documents to update its weights to classify correctly relevant pairs and non-relevant pairs. However, monoBERT receives a query in the inference phase and only the top- relevant candidates. Thus, it classifies the pairs formed by the query and each document in the candidate list. • Heading/Subject: A short sentence that gives a
summary overview description of the problem. • Observation: A longer text describing the observed behavior of a problem, including any useful information for its solution (e.g., logs and conifguration). • Answer: A longer text that contains the resolution given to the fault as well as the reason for the fault. • Faulty Product: It is a specific code of the product on which the fault is reported. In our implementation, we create an extra field, called Faulty Area, from the product name that is mapped to the Faulty Products. By studying historical TRs, source transformers library [20]. Then, we fine-tune it to work with telecom/company-specific data. The training set is composed of the query and document pairs (, ), which are input into the model in batches of samples, {(, )}=1. After making the embedding of each query and document , denoted by and , respectively, the model minimizes the negative log-likelihood for softmax normalized scores: ∑︁
⎤ (, )⎦ =1,̸= The IR and RR stages are supervised-learning models. We train these stages separately as each requires diferent training types. Both stages contain a BERT model in their structure, i.e., the IR stage has Sentence-BERT, and the RR stage uses monoBERT. In the IR stage, we take a model pre-trained using the MSMARCO dataset [19] that includes search queries and passages from a search engine. The model is available in the Hugging Face open
ceives a query and outputs a ranked list of documents.
When a fault is detected, a TR is submitted, and we extract the Observation, the Heading, and the Faulty Product from it. The Faulty Product is then mapped to the corresponding Faulty Area. Then, we concatenate these fields to form the query (Figure 10). Note that we store the representation of all the documents in the corpus after the training phase. So, during the inference time, the IR stage only needs to compute the representation of the new query, which is a quick process.
Once the representation of the new query is computed, Table 1 the IR stage computes its similarity with the diferent Diferent BERT models in Sentence-BERT. documents and generates a candidate list of top- doc- BERT model DistilBERT RoBERTa DistilRoBERTa uments for the RR stage. When the RR stage receives Recall@1 27.2% 26.5% 28.3% the top- candidate list and the query, it processes them Recall@3 39.3% 37.8% 39.7% and creates the final ranked list; the recommended Recall@5 45.8% 44.0% 46.2% documents to a new TR. At the inference time, we must Recall@10 54.4% 53.6% 55.2% limit the computations of the RR as much as possible, as Recall@15 59.4% 58.8% 60.5% it is a time-consuming stage. The IR stage that outputs a candidate list allows this to happen, as the RR stage processes only answers instead of the whole corpus. is. These last two metrics will help us evaluate how well If is the total size of the corpus of answers, we need correct answers are placed in the ranking. a candidate list with a length of ≪ . That way, by having the two stages and pre-saved representations of 5.2. Initial Retrieval Stage Results all the documents, we only need to do one forward pass through a BERT model at IR for the query, and To evaluate the performance of the IR stage, we conduct times at RR for each candidate document, a total of + 1. four experiments: (i) comparing three BERT models in However, if we do not use the IR stage, the number of Sentence-BERT, (ii) evaluating the impact of including forwarding passes through BERT would be for each the Faulty Area to the query, (iii) comparing the query, which is much greater than + 1. performance of Sentence-BERT with BM25 [6], and (iv) studying the relationship between the IR cosine similarity score and the ranked lists. In all these 5. Evaluation experiments, we use Recall@K as the evaluation metric.
First, we introduce the metrics we use in the experiments and then study the performance of the IR and RR stages separately, and finally, we evaluate the performance of the whole model. We use the dataset described in the previous section to train the model and test the model using 15% of all data points. Since we test the model using the TR dataset, we know the correct document (answer) for each problem (query) in the dataset, and we can also check if this document is placed in a high position in the resulting recommended list. We assume a binary relevance between a query and a document, i.e., whether the document is relevant or not.
Mean Reciprocal Rank (MRR), and Normalised Discounted Cumulative Gain (nDCG). The Recall is the fraction of relevant documents for a query in the entire corpus retrieved in the ranked list. Recall@K is the Recall at a cutof . However, the Recall does not consider graded relevance or the positions in the ranking. The Reciprocal Rank (RR) captures the appearance of the first relevant document, which is the multiplicative inverse of the rank of the first correct document. For example, if the first relevant document appears at position 3, then RR is 1/3. MRR is the average of RR of results for a set of queries. nDCG shows the usefulness of a document based on its position in the ranked list, meaning that the earlier a document appears in the ranked list, the more useful it Diferent BERT models in Sentence-BERT. Sentence-BERT is composed of a BERT model and a pooling layer. Here, we compare the performance of Sentence-BERT using diferent BERT models. In particular we consider DistilBERT [15], RoBERTa [13], and DistilRoBERTa [15]. Table 1 shows the Recall@K of the IR stage using diferent BERT models in SentenceBERT. As we see, the diference in performance is small; however, DistilRoBERTa performs slightly better than the others. Therefore, in the rest of the experiments, we use DistilRoBERTa in Sentence-BERT.
Having Faulty Area in the query. Here, we evaluate the performance of the IR stage in two cases: (i) making the query by concatenating the Heading and the Observation, and (ii) making the query by concatenating the Faulty Area, the Heading, and the Observation. These cases are applied both in the training and inference phase. Table 2 shows the results of this experiment, and as we see, including the Faulty Area in the query significantly improves the Recall@K of the IR stage. It confirms that adding more information to the query (in form of the Faulty Area) helps the model to recognize which type of answers will work best for a query.
compare the performance of the IR stage in making the candidate list in two cases of using Sentence-BERT [4] and using BM25 [6], which is a popular EM method. Here, we use the Okapi BM25 implementation. As we see in Table 3, Sentence-BERT outperforms BM25 by improving
Heading + Observation 28.3% 39.7% 46.2% 55.2% 60.5%
Faulty Area + Heading + Observation 30.2% 43.1% 49.0% 58.2% 64.0%
BERT models ELECTRA DistilRoBERTa Ensemble Model Recall@1 36.6% 34.1% 37.1% Recall@3 48.5% 47.3% 48.6% Recall@5 53.5% 52.3% 54.0% Recall@10 59.8% 59.6% 60.3% Recall@15 64.0% 64.0% 64.0%
MRR 0.44 0.42 0.45 studying how the RR similarity scores change along with the ranked lists. To evaluate these, we use the Recall@K, the MRR, and the nDCG.
Diferent BERT models in monoBERT. In the RR stage, we use monoBERT [5], and here, we study the performance of monoBERT, in case of using three diferent BERT models: ELECTRA [ 16], DistilRoBERTa [15], and an ensemble of ELECTRA and DistilRoBERTa, where we combine the scores of these two models by computing their average. Table 4 shows the results, and as we see, ELECTRA has a better performance than DistilRoBERTa. Although DistilRoBERTa is a faster model compared to ELECTRA, we need a more accurate model in this stage. Therefore, in the rest of the experiments, we use ELECTRA in monoBERT. We also see that the ensemble model outperforms the other models; however, the improvement is not significant enough while requiring more computational resources.
The RR similarity scores and the ranked list. Here, we study how the RR similarity scores change along with the ranked list. In Figure 12, we plot the mean and the standard deviation of all the queries in our test set. There are only 15 ranking orders in the Figure 12 as the candidate list forwarded to the RR stage is the top-15 list.
We need to decide how many documents we want to show as the top- recommendations in output. Since the scores decrease uniformly and the variance is very high, we decided that a reasonable number of top- rec
The impact of the RR stage. The RR stage receives a top- candidate list from the IR stage, and then it the Recall@1 by around 65% (from 18.2% by BM25 to re-ranks them, such that the correct document climbs 30.2% by BERTicsson). to the top of the list. Here, we study how the RR stage improves this ranking. To do so, we compare the The IR similarity scores and the ranked list. This position of the correct document in the candidate list experiment evaluates how the cosine similarity scores in two cases: (i) after the IR stage and (ii) after the RR change along with the length of the top- ranked list. stage. In this experiment, we set = 15. As we see in We plot the mean and the standard deviation of all the Table 5, the RR stage improves the MRR by 12% and queries in our test set in Figure 11. Here, the X-axis nDCG by 9%, and increases the Recal@K (for small ); shows the length of the candidate list (i.e., in top-), meaning that although we get the correct document and the Y-axis shows the similarity value. As we see, the from the IR stage, the RR stage improves the ranksimilarity score decreases rapidly in the first positions ing by pushing the correct document to the top positions. in the ranked list, and it decreases more slowly from position 20. Given this result, we consider = 15 is a reasonable length for the candidate list, so the IR stage sends the top-15 documents to the RR stage.
ommended answers is = 5, as it includes the top-5 highest similarity scores, and it is a reasonable number of recommendations to show to an engineer.
Latency vs. accuracy. In this experiment, we study the latency of BERTicsson for diferent candidate list sizes (i.e., diferent in top-). We compare MRR, as a measure of accuracy, with the latency of the whole model (the IR stage latency + the RR stage latency). As we see in Table 7, for larger , the increase in accuracy is minor compared to the increase in latency, which goes from 0.578 with = 15 to 3.82 with = 100. Therefore, the top-15 is a reasonable size that provides a good accuracy while keeping the latency low.
Latency of the two stages. In this experiment, we study the latency of the IR and the RR stages in the inference time. During the inference, we have the pre-computed representations of the corpus of documents, so the IR stage only needs to compute the representation of the query. We compute the latency of each stage by averaging the time each takes to find the top- ( = 15) relevant documents to a query.
As we see in Table 6, it takes 28 on average for the IR stage to make a candidate list and 550 on average for the RR stage to get the final list of ranked documents.
If we only use the RR stage without the IR stage, then the RR stage needs to process all documents instead of just the top ; thus, the latency of the model would increase to minutes just for one query. The latency of the RR increases proportionally to the length of the candidate list. By keeping the candidate list small, we can maintain this low latency. Moreover, if we compare the two IR approaches (i.e., Sentence-BERT and BM25), we see that BM25 does not reduce latency as good as Sentence-BERT.
In this part, we evaluate the latency of BERTicsson by In the last experiment, we evaluate if BERTicsson can conducting two tests: (i) studying the latency of the IR produce similar ranking lists for similar TRs expressed and the RR stages and (ii) studying the impact of the diferently. There is a high probability that diferent cuscandidate list length on the accuracy and the latency of tomers raise diferent TRs for the same underlying probthe model. lem individually. The problems are written in diferent words and lengths but point to a similar or equal problem. We call these TRs, duplicate TRs. We want to check if BERTicsson can recommend the same documents to duplicate TRs. To this end, we compute the ranked lists for all the duplicate TRs, and consider two cases: (i) the model performs well by having the correct document is in the top-15 list, and (ii) the model fails. By analyzing the ranked list of the model for the cases that it performs well, we see that the correct document is ranked at a similar position in 70% of them.
We also try to identify duplicates by checking which document they rank at the top of the list regardless of whether it is correct. If we take all the duplicate TRs, we can check if the document they rank on top is the same as their duplicates. Of all the duplicate TRs in the test set, we can identify at least one duplicate in 40% of the samples. We have to keep in mind that we have not given the model any indication of similar TRs. So, if the model can find duplicate TRs without explicit training, it means that it has learned the domain-specific company language and can make inferences about it. [2] L. Jonsson, Machine Learning-Based Bug Handling in Large-Scale Software Development, volume 1936, Linköping University Electronic Press, 2018. 6. Related Work [3] J. Devlin et al., Bert: Pre-training of deep bidirectional transformers for language understanding, Many BERT-based text-ranking models have been devel- arXiv preprint arXiv:1810.04805 (2018). oped over the past years, such as EPIC [22], ColBERT [4] N. Reimers et al., Sentence-bert: Sentence embed[23], and ANCE [24]. They all use BERT in diverse ways dings using siamese bert-networks, arXiv preprint to rank documents concerning a query and present their arXiv:1908.10084 (2019). results with general-domain data. One of the popular [5] R. Nogueira et al., Multi-stage document ranking applications of text-raking is bug analysis and resolution, with bert, arXiv preprint arXiv:1910.14424 (2019). which is studied well in literature [25]; however, there is [6] S. Robertson et al., The probabilistic relevance not much work on using BERT for it. framework: BM25 and beyond, Now Publishers Inc,
Companies like Ericsson have developed some solu- 2009. tions for automating bug resolution using an ensemble [7] D. Harman et al., Information retrieval: the early of diferent pre-BERT techniques (such as LSTM and years, Foundations and Trends® in Information LDA) [26]. However, these solutions do not consider Retrieval 13 (2019) 425–577. the latency of the process as part of their performance [8] P. Huang et al., Learning deep structured semantic metric. Another example of an ensemble of pre-BERT models for web search using clickthrough data, in: techniques is [27], where the authors formulate the prob- Proceedings of the 22nd ACM international conferlem as a non-convex optimization problem and solve it ence on Information & Knowledge Management, using a heuristic solution with a focus on accuracy. Con- 2013, pp. 2333–2338. sequently, their approach leads to a sub-optimal solution. [9] B. Mitra et al., A dual embedding space model for
There are also many multi-stage BERT-based ap- document ranking, arXiv preprint arXiv:1602.01137 proaches for text-ranking. Some examples are duoBERT (2016). [5] or DeeBERT [28]. They present the results using [10] J. Guo et al., A deep relevance matching model general-domain data. However, to the best of our knowl- for ad-hoc retrieval, in: Proceedings of the 25th edge, none of them consider domain-specific tasks like ACM international on conference on information automating the resolution of telecom TRs. and knowledge management, 2016, pp. 55–64. [11] C. Xiong et al., End-to-end neural ad-hoc ranking 7. Conclusions with kernel pooling, in: Proceedings of the 40th International ACM SIGIR conference on research In this work, we present BERTicsson, a BERT-based and development in information retrieval, 2017, pp. model for analyzing Trouble Reports (TR) to retrieve 55–64. solutions for newly reported problems. BERTicsson re- [12] G. Furnas et al., The vocabulary problem in humanceives a dataset of existing TRs and, after pre-processing system communication, Communications of the data, trains the model in two stages: Initial Retrieval ACM 30 (1987) 964–971. (IR) stage and Re-Ranker (RR) stage. The IR stage uses [13] Y. Liu et al., Roberta: A robustly optimized bert preSentence-BERT to create a candidate list of answers for training approach, arXiv preprint arXiv:1907.11692 a problem (query), and the RR stage uses monoBERT to (2019). rank the candidate list according to the problem. We [14] Z. Lan et al., Albert: A lite bert for selfcompare BERTicsson with BM25, a popular Exact Match- supervised learning of language representations, ing model, and show that BERTicsson can recommend arXiv preprint arXiv:1909.11942 (2019). the best possible solutions to a new error report from [15] V. Sanh et al., Distilbert, a distilled version of bert: Ericsson with higher accuracy and lower latency, given smaller, faster, cheaper and lighter, arXiv preprint the dificulty of the domain-specific data. Moreover, the arXiv:1910.01108 (2019). model is consistent as it recommends the same answers [16] K. Clark et al., Electra: Pre-training text encoders to similar TRs expressed diferently. as discriminators rather than generators, arXiv preprint arXiv:2003.10555 (2020). [17] R. Nogueira et al., Passage re-ranking with bert, References arXiv preprint arXiv:1901.04085 (2019). [18] D. Chicco, Siamese neural networks: An overview, [1] W. Wong et al., A survey on software fault localiza- Artificial Neural Networks (2021) 73–94. tion, IEEE Transactions on Software Engineering [19] T. Nguyen et al., Ms marco: A human generated ma42 (2016) 707–740. chine reading comprehension dataset, in: CoCo@ NIPS, 2016. [20] T. Wolf et al., Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45. [21] M. Henderson et al., Eficient natural language response suggestion for smart reply, arXiv preprint arXiv:1705.00652 (2017). [22] S. MacAvaney et al., Expansion via prediction of importance with contextualization, in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020, pp. 1573–1576. [23] O. Khattab et al., Colbert: Eficient and efective passage search via contextualized late interaction over bert, in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020, pp. 39–48. [24] L. Xiong et al., Approximate nearest neighbor negative contrastive learning for dense text retrieval, arXiv preprint arXiv:2007.00808 (2020). [25] J. Jiang et al., How to mitigate the incident? an efective troubleshooting guide recommendation technique for online service systems, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1410–1420. [26] N. Ferland et al., Automatically resolve trouble tickets with hybrid nlp, in: 2020 IEEE Symposium Series on Computational Intelligence (SSCI), IEEE, 2020, pp. 1334–1340. [27] Y. Wang et al., Generalized ensemble model for document ranking in information retrieval, arXiv preprint arXiv:1507.08586 (2015). [28] J. Xin et al., Deebert: Dynamic early exiting for accelerating bert inference, arXiv preprint arXiv:2004.12993 (2020).