Software development processes are complex to monitor as they involve the coordination of many resources working with diferent tools. This makes it hard to apply mining techniques for monitoring the process. A key challenge for using traces of tools such as version control systems (VCS) is to find meaningful abstractions in order to identify the work that was actually done. In this paper, we use data from VCS to analyze the actual progress of software-development processes. We develop a technique that is able to mine the activity types of which the development processes consists. We implement our technique as a prototype in Java and evaluate its outputs in terms of efectiveness. In this way, we are able to graphically uncover new behavioural patterns in real-world data from existing open-source GitHub repositories.
In this paper, we provide a technique for capturing the progress of a project in such a way that it becomes clear what work activity is being done over time. We define fundamental concepts for representing these processes, upon which we develop a novel analysis technique. Our prototypical implementation of this technique is able to represent the status of the development process as well as the activity that is being done.
The rest of the paper is organized as followSesc.tion 2details the problem, positions our contribution against existing literature, and defines the requirements for the design of the artifact that solves the stated problemSe.ction 3defines preliminary concepts and presents the approach to mine the activitiesS.ection 4describes the implementation of the artifact and shows its application to real-world projeScetsc.tion 5concludes the paper.
Problem description. The problem discussed in this paper is the monitoring of software engineering projects. These projects present the following characteristics. First, they are hardly repetitive. That is, while best practices learned from one project endeavour can be reused, it is never the case that the project is rerun exactly in the same way. Second, they are worked on collaboratively by many participants who regularly document their progress in a semi-structured way. Third, the work is performed under clear constraints in terms of time, budget and quality. Fourth, the workflow does not follow an imperative process model and is not managed by a process engine. Fifth, despite the aforementioned limitations, project managers require transparency of the process in terms of being able to distinguish what kind of work activities where done when and by which resource. In this paper, we consider the terms “type-of-work” and “activity” as synonyms.
Project managers need tools that help them understand where the project is currently standing and how they have performed in retrospect. Being able to see what work was done and when opens up possibilities to understand ineficiencies about how the process was conducted. Moreover, managers need to access information in a timely manner. Therefore, a representation of progress over time is important. Theoretical models of the development process, such as the Rational Unified Process (RUP,) are useful for planning software projects but they fall short when it comes to monitoring.
Fortunately, software development projects store rich trace data. Participants (e.g., resources, users) use many tools for diferent purposes. A common tools is used in any professional software engineering project areVCSs. These systems allow users to collaboratively work on the same project. They manage the diferent versions of files created by users at any point in time. As well, they keep track of all the changes done by resources at all times. These kind of systems represent a starting point for analyzing projects. As software projects may contain hundreds of thousands files, it becomes prohibitive to manually check what work was done in the project. This calls for automatic analyses and reporting.
Typicalactivities (i.e., type of work) traced byVCSs fall under the categoriCesode, Documentation, Test, etc. MajorVCSs like Git and Subversion, do not provide direct support for understanding the activities executed by developers. However, these tools provide rich event logs which record all the changes made to any filLei.sting 1 presents an excerpt of trace data from a publicly available GitHub repository. Specifically, the trace data shows information about two commits, which are activities performed by developers to save their work progress. These commits have a unique identifier and provide information aboiu)tthe author (i.e., the human resource who issued the commit)i;i) the date (i.e., a timestamp recording the instant when the commit was made); iii) a natural-language textual description filled in by the author; andiv) a list of files that were either modified (M), added (A) or deleted (D).
Listing 1: Excerpt from VCS log data from git commit b0346a47df142394da820e1e5d0f7e31b41a70d3 Author: s41m1r Date: Wed Feb 4 13:05:14 2015 +0100
Deleted TODOs D MiningSVN/TODOs M MiningSVN/src/reader/GITLogReader.java commit 30c5e536e88501295aa3f226645953c69e8f3947 Author: s41m1r Date: Wed Feb 4 15:31:05 2015 +0100
Works with GIT (hopefully ) M MiningSVN/src/reader/GITLogReader.java
A MiningSVN/src/test/TestReadGIT.java
As real-life event logs may contain a large amount of commits, it is imperative to use automatic tools to discover the activities. While it is possible to take into account the commit messages and applyNatural Language Processing (NLtPe)chniques to classify the various change3s],[ Listing 1 suggests that there are no guarantees that the textual descriptions are informative about the activity. For this reason, this work focuses on the type of file that was modified rather than relying on the commit comments.
Therefore, the problem is how to exploit low-level trace data for extracting project knowledge that is informative to the manager. We translate this problem into the following requirements. RQ1. (Processing of VCS event logs). The prototype must extract valuable information from VCS data.
RQ2. (Identification of the activities). The prototype shall classify what activity is done and when.
RQ3. (Computation of KPIs). The prototype shall providKeey Performance Indicators (KPIs) that are understandable by project managers.
RQ4. (Visualization of project status). The prototype shall provide a high level overview of the project.
Related work. Literature related to the aforementioned requirements can be classified into two main groups:(i) software engineering; and(ii) business process management.
Contributions in(i) focus on event data generated by systems likVeCS, issue tracking, bug
tracking, mail archives, etc. They mainly aim at either finding correlations between activities
performed by resources and the artifacts in the repositor4i]eosr[ at analyzing the evolution
of changes over time 5[]. These works typically provide powerful techniques that help with
processing events 6[] from software data and further abstracting them into coarse-grained
activities [
Contributions in(ii) fall under theprocess mining umbrella. Typical approaches focus on transforming software development data in process-mining compatible event-logs [1121],, by making assumptions on what to consider as a case identifier. Other works focus on enabling process analytics on top of fine-grained events from evolving artifac1t3s].[ Finally, there are process-aware works that deal with software repositories. In particular, the work 1fr4o]m [ analyses bug resolution processes and the work fro2]mu[ses VCS data to analyse teams. Most of the techniques in the process mining area have specific requirements about their input (i.e., an event log with defined case, activity, and timestamp attributes). These works cannot be readily applied to data from software developmen1t5][. In this paper, we focus on automatic identification of the activities based on file types as described in9[]. Moreover, this work is process aware and presents the data from a perspective which is more targetted towards project managers.
We developed a technique that takes as input an event log frVoCmS and extracts visual insights as well asKPIs about the development process. In the following, we describe the steps of our technique.
Preprocess VCS log file. The input of this phase is a log in thuenified dif format , which is ofered by major VCSs such as Git and Subversion. The information retrieved by tVhCeS is configurable by the user. More specifically, it is possible to obtain the basic information shown in Listing 1 along with details on the diferences among versions of the same file (i.e., which and how many lines changed from version 1 to version 2 of file X). In order to extract such information, the raw event log is parsed. We used the parser fr1o6m]. [This parser generates events which are then stored intoDaatabase Management System (DBMS)for further processing.
From a log event, we extract the following entitiie)sP:roject;ii) User;iii) Commit; iv) File; andv) Edit. Projectrepresents the software project at hand. By including this entity in the data model, we can gather data over multiple projects and store them in the same dataUbaseser. is the user who performs change operations on the fileCs.ommit represents the status of the repository at a given point in time. Commits must contain a revision number, a timestamp and the user who issued them.Fileis a single file of the repository identified by its full pathE.dit captures the change as numbers of lines added to or removed from a file. This step fulRfillQs1. Classify activities. Having the data stored in DaBMS, enables us to run several analyzes already at this level by simply issuing SQL queries. For example, we can obtain all changes that happened to single files during their lifetime. For the scope of this work, we collect all the file paths, all the changes that happened to files, the amount of change in terms olfines of code (LOC), the type of change (e.g., addition, modification, deletion), the commit identifier, and the user who did the change. Next, we automatically categorize the type of change. For this, we apply regular expressions on path attribute in accordance to the classes provi9d]e.d in [ Examples of classes arTeesting, Coding, User interfac,eetc. There are in total fourteen classes. When a type of change does not belong to any class, it is put under the categUonrkynown. .* .*\/doc(-?)book(s?)\/.* .*\/info .*\.txt((\.bak)?) .*\.man .*\.tex .*\.jpeg .*\.bmp .*\.chm .*\.vdx .*\.gif .*\/locale(s?)\/.* .*\.po(∼?) .*\.charset(∼?) .*\.ui .*\.gladep(\\d?)((\.bak)?)(∼?) .*\.theme .*\.mp3 .*\.mp4 .*\/media(s?)\/.* .*\.ogg .*\.jar(∼?) .*\/src\/.* .*\.r((\.swp)?)(∼?)) .*\.py((\.swp)?)(∼?) .*\.php((\.swp)?)(\\d?)(∼?) .*\.svn(.*) .*\.git(.*) .*\.cvs(.*) .*\.conf .*\.cfg .*\.project .*\.ini .*\.prefs .*\.cmake .*\/install-sh .*\/build\/.* .*makefile.* .*readme.* .*\/changelog.* .*\/devel(-?)doc(s?)\/.* .*\.sql .*\.sqlite .*\.mdb .*\.db .*\.test(s?)\/.* .*\/.*test\..* .*/test.*\..* .*\/library\/.* .*\/libraries\/.*
We have adapted the regular expressions to our case and enriched the list of rules from literatureT.able 1shows the activity types and the main regular expressions we use to classify ifles onto specific types of work. For the sake of space, the majority of the regular expressions is left out. The reader can access the full list of regular expressions on our GitHub repository whose link is provided in the following section.
For the categorization we consider both the extension of the file and its path. For example, a ifle with the path /test/file.java is labelled aTsesting rather thanCoding. To achieve this, we sort the matching rules in order of specificity. At a higher level, a commit involves multiple files. In order to fit the commit into a specific class, we rely on majority voting as follows. We iterate over the list of changes afected by the commit and sort the number of changes by their activity and the amount of change. We select the activity that is associated to the highest number of changes. With this step we fulfillRQ2.
Compute KPIs. Next, we compute KPIs with the help of theDBMS. This allows for a customized set ofKPIs to be implemented. In the scope of this paper, we reproduced some of the main KPIs form literature9[]. We divide them into basic (absolute and relative) and specialization metrics. Basic metrics focus on descriptive statistics such as frequency counts of how many times each user works on a file. Specialization metrics focus on the measuring imbalance of work towards a specific file or author. Imbalance is captured by the Gini inequality ind1e7x].[ (absolute) project workload ∑∈,∈ (, ) workload of a specific activity ∑∈ (, ) number of authors in the project ∑ ∈ number of activities in the project ∑ ∈ tsipveitciieasliozfattihoenporfoujescetr involvement the ac- ∈ (∑∈ (, specialization of relative user involvement over the activities of the project specialization of relative workload across all activities in the project ∈ ( ∑∈ (, ) ) ∈ ( ∑∈ (, ) ) )) We implemented the following basic project metricpsr:oject workload (PWt)y,pe of change workload (TW)n,umber of authors in project (NAP,n)umber of types of work in project (NTP.) Furthermore, we also implemented the following specialization metrspicesc:ialization of author in each activity type (PIS,)specialization of relative author in each activity type (RPIS), and specialization of relative project workload (RPWS).
Let be the set of all unique users in the projectt,he set of all activities in the project. Files that were edited in the context of a commit can be associated to a u s∈er and an activity ∈ computed as described above. We adapted the definition oKfPIs from literature to support the extraction of information about the activity types as follows. As a first step, we redefine the two basicKPIs that are involved in the calculation of all oKthPeIsr. User–Activity–Workload (UTW) is the number of files relative to activit y, a user edits over the entire history of the activity.User–Activity–Involvement (UTI)is 1 if a user has been involved in (i.e., has edited at least once) a file with activit y . It is 0 otherwise. Finally, using these definitions, we can present inTable 2how the rest of theKPIs is computed. With this step we fulfillRQ3. Visualize results. The last step of our technique deals with the presentation of the results. As our prototype needs to be informative to project managers, we chose to graphically display the results of the previous two steps on a friendly user interface. The user interface takes as input the results of the classification of the activities as well as the set of the compuKtePdIs. The main goal of the user interface is to show two fundamental aspects of the project at hand. First, it visualizes the evolution of each activity aggregated by customized periods of time (e.g., weeks, months). By doing so, our prototype helps at vizualing the general behaviouRrUaPs phases. This enables the project manager to compare the ideal project evolution to the actual one. Second, we display the variouKsPIs in a dashboard (e.g., barchart). Further information, such as the commit identifiers, file names, amount of changes, and users who worked on the ifles are also made available. This allows the project manager to zoom into specific parts of the project for more detailed analyses. With this step we fuRlfilQl 4.
Our prototype is named ActiVCS and is available as open source on GitHhtutpbs(://github.com/ PaulKner/ActiVCS).
Visualization of projects. Figure 1shows one output of our prototype. This visualization plots the evolution of the automatically extracted activciotideesandtest. This visualization is part of a UI that shows several other properties. The domain expert can execute a number of further actions. These actions are the followiin)gl:oad an event logii;) save an event log for reuse and avoid parsing it ane wiii;) get help informationi;v) change granularity of the timeline by choosing to display data daily, weekly or monthvl)yc;hange data series on X-axis between commit-level and file-level; andvi) four buttons to change size and type of the plots. The plots shown in the frame illustrate the evolution of the identified types of work over time. The user can choose between having this information on commit- or file-level.
It is possible to interact with each point on the plot shown in the main frame. Depending whether the X-axis is showing the information at commit-level or at file-level, a pop-up menu that summarizes the information related to that point of the plot is showKnP.Is about the overall project are also available. Especially, our tool ofers a separate tab in which it is possible to visualize bar-charts based on the values of the KPIs (e.g., what are the actual activities being done and by how many authors, the size of the project, the GINI index, etc). Further information ofered by the tool consists of tabular information about the measures presentedSeicntion 3.
With this information at hand and their knowledge of the domain, the project manager can investigate, for example whether the team dise factofollowing a specific software methodology. Analyses of real-life open-source projects. We use ActiVCS to analyze real-life open source projects from GitHub. We chose thirteen projects of diferent size, age, user counts, and programming languages. Selection criteria included haiv)iangr,epresentative variety in terms of programming languageii,) variety in the sizei,ii) at least having two similar projects, iv) highly active versus inactive, anvd) fast versus slow growing projects.
This resulted in the following projecMtsP.AndroidChart (MPis a visualization library for Android platformsT.orque2D (Torque)is a software engine for the development of video games. Openage (openage)is an open source clone of the Age of Empires II enginIen.cubator-dubbo (inc) is a RPC (remote procedure call) framework for Jajevkay.llis a blog-aware written Ruby that generates websites from user contenstc.rapyis a Python based framework to extract data from 24 websitesbrew is a software that automatically installs missing packages for MacOS and Linux operating systems.Algorithms – Java (Javai)s a collection of diferent Java algorithms. Flask (flask) is a lightweight Web Server Gateway Interface (WSGI) web application framework created in PythonT.ablesawis a framework for the transformation and visualization of data, implemented in Java.Okhttp (ok) is an HTTP client for Java and AndroiRde.trotfit (retro) is another HTTP client for Java and Androeid.itor.js (editori)s a JavaScript based editor software for the creation of documents and the transformation into JSON format.
Table 3shows the KPIs resulting from these projects. Columns contain the following information: number of commits (COM),project workload (PWt)y,pe of change workload (TWn),umber of authors in project (NA P,)number of types of work in project (NTP,)specialization of project workload (PWS)s,pecialization of author in each activity type (PIS), code (C) , documentation (D) , testing (T) and unknown (U). Entries have been sorted by thsepecialization of project workload (PWS.)This means that, for instance, resources of projeMcPt are highly occupied. Therefore, load balancing should be considered if the managers want to improve the capacity of the team to handle new tasks in the near future.
Finally, the types of work code (C), documentation (D), testing (T) and unknown (U) that are displayed in the table make up the most frequently identified types that were detected with ActiVCS. As expected from software development processes, the main workload in most analyzed projects were coding activities. One exception to this observation is the jekyll project. ActiVCS has detected 7001 file changes with an unknown type within the Ruby software application. Projects like these would provide a solid basis for the identification of additional ifle types. The total amount of workload that was captured across all projects is 172761. The number of file changes which could not be classified by ActiVCS and were marked as unknown is 16402. This represents 9,49% of the total workload. Excluding the jekyll project from the analysis would reduce this margin to 5,94%. Many of the identified projects neglect the creation of documentation. All of the analyzed projects with a workload value for coding activities of 8000 and more also have a noticeable amount of registered testing activities. This could refer to the fact that large software development processes have a need for automated test activities which must be frequently updated.
Discussion. ActiVCS can be used as a tool for checking conformance to existing development methodologies.Figure 1shows a screenshot of the types of work Code and Test, taken from the ok project, described previously. A project manager can now check that work of the type Code and Test was consistently done throughout the lifetime of the project. Moreover, it is possible to observe that Code and Test were active together most of the time, with Code starting earlier. A typical development methodology that presents such patternagiisle.
This might confirm that the de factosoftware development method corresponds with what the enterprise has decided. Alternatively, the enterprise might be following a Waterfall development model. In that case, this pattern may point at a lack of control on the project. The managers can use their domain knowledge along with the providfeadctual information for better decision making.
This paper provides an artifact for analyzing event logs fVrComS. Our tool is able to visualize the type of work that was executed in a software development process. Starting from finegrained changes in the diferent versions of files, it allows to understand what activity was done and when. It also provides important KPIs, such as the efort distribution. These features are important for managers to understand whether the project is deviating from target goals, and in case take corrective actions.
In the future, we will integrate the ActiVCS tool with the Gantt chart miner fr1o]m.T[his would ofer to the project manager complementary views on the current status of the project. We plan to conduct user studies with managers in order to receive more feedback from domain experts. Finally, we have already conducted a study on selecKtPinIsgfor software development