This paper is aimed to explore the application of Artificial Intelligence/Machine Learning (AI/ML) to software engineering research problems. Which activities of software engineering use AI/ML the most for solving research problems? The scope of the paper is to preliminary review research papers published in Asia-Pacific Software Engineering Conference 2018 (APSEC 2018) proceedings and researches conducted at the Department of Computer Engineering, Chulalongkorn University (CPCU). The author manually reviews papers with some keywords such as machine learning, neural network, and natural language processing. The result shows that machine learning is used in coding and software quality improvement activities more than other activities.
Several research problems in software engineering particularly in requirements engineering, defect prediction, and coding are solved using AI/ML. Tahira Iqbal et al. present literature review of AI/ML for requirements engineering research problems [Iqb18]. Robert Feldt et al. present the review of AI in SE [Fel18]. Previous researches [San19], [Poo18], [Mek12], [Man11], [Sre16], [Kae19], [Phe19] at CPCU have applied AL/ML methods in RE, coding, software quality improvement and maintenance. In this paper, the author intends to investigate which SE activities that are often used AI/ML methods to solve research problems. The most AI/ML used activity will be summarized. Research problems of this paper include
RQ1: What is the current state of the art in Software Engineering activities problems which are resolved with AI/ML methods in APSEC 2018?
RQ2: What is the current state of the art in Software Engineering activities problems which are resolved with AI/ML methods at CPCU?
The scope of this research is to extract information from APSEC 2018 proceedings and Software Engineering group at Chulalongkorn University, Thailand. The results of the study may be considered to possibly update AI/ML contents of some courses of the Software Engineering curriculum.
Section 2 briefly describes related research. The methodology is explained in section 3. Results and conclusions are described in section 4 and section 5 respectively. 2
Robert Feldt et al. present “the AI in SE Application Levels (AI-SEAL) taxonomy” [Fel18]. Applications are categorized according to their point of AI application (process or product), the type of AI technology used and the automation level (1 to 10) allowed.” Types of AI include Symbolist, e.g., inverse deduction, Connectionist, e.g., backpropagation, Evolutionary, e.g., genetic programming, Bayesians, e.g., probabilistic inference and Analogizers, e.g., kernel machines. Seventeen papers of previous RAISE workshops (out of 44 papers) are papers with the application of AI to software engineering. The papers are classified based on the three aspects. The results show that there are 12 process, 3 product and 2 runtime-related papers. Eight of them are Analogizer, five Symbolist, and one for Evolutionary and for Connectionist. Most of them have low level of automation (level 2-3).
Tahira Iqbal et al. conduct literature review to obtain an overview of how ML are used in requirements engineering (RE) which includes requirements elicitation, requirements analysis, requirements documentation and requirements verification [Iqb18]. The paper summarizes as follows. 1. In requirements elicitation and discovery phase, several kinds of research use mining, ML, and recommendation system to classify requirements into improvement request or not, into bugs, features and junk and to discover evolutionary requirements and related requirements. 2. In requirements specification and analysis phase, ML can use to identify if a set of requirements is Non-functional Requirements (NFR) or not and is functional Requirements (FR) or not, to distinguish FR from NFR, to find Prioritization of Requirements, and to identify if a NFR is security requirement. 3. In requirements validation phase, some researches validate on consistency and traceability. 4. In requirement management activity, some researches focus on visualization of a large group of requirements in order to make better decision and on using ML to classify and cluster information in requirements specification into requirements or information and to grouping similar and related requirements and place them contiguously. 3 3.1
Two aspects consist of software engineering aspect and AL/ML aspect. Since Tahira Iqbal et al. review papers which apply ML to RE [Iqb18] and Robert Feldt et al. present papers that use AI on either process or product aspect [Fel18], this paper further identifies specific process activities of software engineering by which AL/ML methods are used in solving problem. Processes as software engineering aspect include Requirements Gathering, Analysis, Design, Coding, Testing, Maintenance, Software Quality Improvement (Product). AL/ML methods as the second aspect include 1. natural language processing, 2. supervised learning (support vector machine (SVM)), 3. unsupervised learning (genetic algorithms, clustering/classification, similarity, K-Nearest Neighbour (KNN), and 4. reinforcement learning (neural network, Bayesian network, Naïve Bayes). 3.2
To answer RQ1, 82 regular papers of Asia-Pacific Software Engineering Conference 2018 (APSEC 2018) are reviewed. Seven-teen papers containing keywords on AL/ML methods are found and studied. To answer RQ2, seven interviews and some paper reviews are performed at the Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand. Five papers [San19], [Poo18], [Mek12], [Man11], [Sre16], [Kae19] use AI/ML for solving software engineering problems. 4 4.1
API usage patterns are automatically generated from the natural language queries [Tin18]. Rules-based regularization method is used to get concise usage patterns. The encoder of the proposed method uses the recurrent neural network with long short-term memory (LSTM) units. Comparisons with other methods are presented. Doc2Vec is an NLP tool that uses neural networks [Ama18]. Comments of original java code and the comment-erased version are assessed with Doc2Vec. Similarity score of each version is computed and checked if the erased-comments version has a high value or not. Shinyama et al. analyze code comments to boost program comprehension using a decision-tree based classifier [Shi18]. Three different classifiers are built for each element: Extent, Target and Category. SOQDE is a supervised learning-based (random forest) question difficulty estimation model [Has18]. STAR is a specialized tagging approach for docker repositories [Yin18]. Logistic regression-based classifier is used to determine whether a tag should be assigned to a repository. Four methods of tagging are compared.
An automatic approach using KNN and Random forest [Kim18] is proposed to validate log levels in a class or a method: Trace, Debug, Info, Warn, Error, or Fatal. A tool for Tuning the Level of Parallelism of Spark Applications Optimizations with KMeansClustering and StreamingWordCount is proposed [Ros18]. An approach with neural network, naive Bayes, logistic regression and SVM, DTPre based on decision tree [Moh18] is proposed to predict which pull requests will get reopened in GitHub. SLAMPA tool recommends code snippets with statistical language Model [Zho18] using a deep neural network called Recurrent Neural Network (RNN). 4.1.3
A hybrid analysis method is designed to detect malicious JavaScript code [He18]. Several classifiers are used in constructing classification models, such as Random Forests (RF), Logistic Regression (LR), Naive Bayes (NB) and Support Vector Machine (SVM). Detecting Duplicate Bug Reports with Convolutional Neural Networks (CNN) is presented in [Xie18].
A ML-based approach is proposed to categorize and predict Invalid vulnerabilities on common vulnerabilities and exposures [Che18]. A machine learning model adopts several classic classification algorithms including naïve Bayes, multinomial naive Bayes, SVM and Random Forest for learning from the whole dataset of invalid CVEs. A Comparison of Nano-patterns and software metrics in Vulnerability Prediction is presented in [Sul18]. A vulnerability prediction model using the nano-patterns extracted from vulnerable and neutral (we use the term “neutral” to refer to methods where no known vulnerability exists) code of different software systems. Three machine-learning techniques are used to classify vulnerable code including Naive Bayes (NB), Support Vector Machine (SVM) and Logistic regression (LR).
A Top-k Learning to Rank (LTR) Approach using Random forest is designed to predict cross- project software defect [Wan18]. A bug localization model is constructed with two main parts including character-level convolutional neural network (CNN) and recurrent neural network (RNN) language model [Xia18]. *SE activities: RE: Requirements Engineering, Design, Coding, Testing, SWQ: Software Quality Improvement (SWQ-defect, SWQ-security), **Natural: Natural language processing, ML: Machine Learning ***NB:Naïve Bayes, Max:MaxEnt, DT: Decision Trees, RF: Random Forest, SVM: Support vector machine, NN: Neural Networks, BN: Bayes Network, LR: Logistic Regression, ARL: Association Rule Learning, EBL:Explanationbased learning, 4.2
This section explores research conducted at the Department of Computer Engineering, Chulalongkorn University, Thailand. There are 33 faculty members. Twelve faculty members have main researches in AI/ML and five in SE. Only two faculty members solve software engineering research problems using AI/ML.
Table II shows research papers which use AI/ML methods solving software engineering research problems. Naive Bayes method is used to classify short text of requirements [San19]. Association rule learning (ARL) is used to find impact factors for rejection of pull requests on GitHub [Poo18]. Explanation-based learning in a metaprogramming approach is used to detects of Object-Oriented design defects [Mek12]. This paper uses machine learning methods (Naive Bayes, Logistic, IB1, Ibk, VFI, J48 and Random forest) to predict bad-smells design from software design model [Man11]. Defect-related keywords are discovered using natural language process (NLP) by analyzing user feedback to extract defect related keyword [Sre16]. Prioritizing software maintenance plan uses Analytical Hierarchy Process (AHP). Software problem report types are classified using machine learning [Kae19]. Mobile application user reviews are classified for generating tickets on issue tracking system.
Software quality improvement (SWQ-defect) has more papers using AI/ML methods to solve research problems than other software engineering activities which answers research question#2 (RQ2). However, the result from only 7 papers is not sufficient to make any general conclusion.
This paper preliminary investigates on using AI/ML of software engineering activities from papers published in APSEC 2018 and at the Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Thailand. In APSEC 2018, there are more papers in coding and in software quality improvement using AI/ML methods than other software engineering activities. At CPCU, most AI/ML researches are focused mainly in theoretical aspects and in applying in several application domains. Only two faculty members uses AI/ML in software engineering research problems. Future works include 1. review more papers with automatic tool 2. extend scope to cover research conducted in industry 3. find a possibility to include AI/ML into SE courses/curriculum.