-Numerous empirical studies analyse evolving open source software (OSS) projects, and try to estimate the activity and effort in these projects. Most of these studies, however, only focus on a limited set of artefacts, being source code and defect data. In our research, we extend the analysis by also taking into account mailing list information. The main goal of this article is to find evidence for the Pareto principle in this context, by studying how the activity of developers and users involved in OSS projects is distributed: it appears that most of the activity is carried out by a small group of people. Following the GQM paradigm, we provide evidence for this principle. We selected a range of metrics used in economy to measure inequality in distribution of wealth, and adapted these metrics to assess how OSS project activity is distributed. Regardless of whether we analyse version repositories, bug trackers, or mailing lists, and for all three projects we studied, it turns out that the distribution of activity is highly imbalanced. Index Terms-software evolution, activity, software project, data mining, empirical study, open source software, GQM, Pareto
Numerous empirical studies aim to understand and model
how open source software (OSS) evolves over time [
In this article, we wish to understand how activity is spread
over the different members of an OSS project, and how this
activity distribution evolves over time. Our hypothesis is that
the distribution of activity follows the Pareto principle, in the
sense that there is a small group of key persons that carry
out most of the activity, regardless of the type of considered
activity. To verify this hypothesis, we carry out an empirical
study based on the GQM paradigm [
The remainder of this paper is structured as follows. Section II explains the methodology we followed and defines the metrics that we rely upon. Section III presents the experimental setup of our empirical study that we have carried out. Section IV presents the results of our analysis of activity distribution in three OSS projects. Section V discusses the evidence we found for the Pareto principle. Section VI presents related work, and Section VII concludes.
To gain a deeper understanding of how OSS projects evolve, we follow the well-known Goal-Question-Metric (GQM) paradigm. Our main research Goal is to understand how activity is distributed over the different stakeholders (developers and users) involved in OSS projects. Once we have gained deeper insight in this issue, we will be able to exploit it to provide dedicated tool support to the OSS community, e.g., by helping newcomers to understand how the community is structured, by improving the way in which the community members communicate and collaborate, by trying to reduce the potential risk of the so-called bus factor1, and so on.
To reach the aforementioned research goal, we raise the following research Questions: 1) Is there a core group of OSS project members (developers and/or users) that are significantly more active than the other members? 2) How does the distribution of activity within an OSS community evolve over time? 3) Is there an overlap between the different types of activity (e.g., committing, mailing, submitting and changing bug reports) the community members contribute to? 4) How does the distribution of activity vary across different OSS projects?
As a third step, we need to select appropriate Metrics that will enable us to provide a satisfactory answer to each of the above research questions. For our empirical study, we will make use of basic metrics to compute the activity of OSS project members, and aggregate metrics that allow us to compare these basic metric values across members (to understand how activity is distributed), over time (to understand how they 1The bus factor refers to the total number of key persons (involved in the project) that would, if they were to be hit by a bus, lead the project into serious problems evolve), and across projects (to compare the situation between different OSS projects).
To obtain the basic metrics of OSS activity, we will extract information from three different types of data sources we have at our disposal: version repositories, mailing lists, and bug trackers. For each of these data sources, we can define metrics that extract and reflect a particular type of activity: Development activity: the activity of developers committing source code to a version repository, measured as number of commits.
Mailing activity: the activity of project members posting messages to a mailing list, measured as number of mails. Bug tracker activity: the activity of persons interacting with a bug tracker, measured in three different ways: number of new bug report submissions, number of comments added to existing bug reports, number of changes to existing bug reports.
Since we are not only interested in a static view of a particular snapshot of an OSS project at a particular moment in time, we will extract each of the above activity metrics during the entire life of the considered OSS projects.
Since several of the research questions require a comparison of the basic metrics (across persons, across projects, and over time), we need aggregate metrics that combine the basic metrics. This is valuable, in particular, if we want to reason about the distribution of activity across OSS project members.
To study such distribution, we borrow ideas from econometrics. This discipline uses statistics and metrics to analyse economic data. As an example, various aggregation measures of statistical dispersion have been proposed (e.g., the Hoover, Gini, and Theil indices) and applied to assess the inequality of the wealth distribution among people, regions, countries, and so on.
Recently, some of these aggregation measures have been
used for analysing evolving software systems. Vasa et al. [
Let X = fx1; : : : ; xng be a set of values indexed in ascending order (8i 2 1 : : : n 1; xi xi+1). The sum of all these values will be called xtotal (Equation 1). The mean of all values will be called x (Equation 2).
xtotal = x = xtotal n n X xi i=1
The Hoover index, defined in Equation 3, is one of the simplest ways to assess inequality of wealth or income. Its value is the ratio of incomes to take up from the richest part of the population to redistribute to the poorest one so that the incomes become perfectly equal. A Hoover index of 0 represents perfect equality, while a value of 1 represents perfect inequality.
n H (X ) = 1 X 2 i=1
xi xtotal 1 n
The Gini index, defined in Equation 4, provides a more complex (but also more representative) way to assess inequality of income. The cumulative function of income distribution is represented by a perfect diagonal if all entities in the population would have the same income, and by a curve under the perfect line otherwise. The Gini index is the value of the surface area between the perfect line and the curve, divided by the surface area below the perfect line.
G(X ) = 1 n 2 0
1 n X ixi i=1 CC Xn xi ACC i=1
Yet another index to assess inequality of income or wealth
is the Theil index. It is based on the Shannon entropy [
In order to obtain replicable and verifiable results, we do not only need a good methodology. At least as important is automated tool support that enables us to extract data from the different data sources, compute the basic and aggregate metrics based on this data, statistically analyse the obtained metrics, and visually confirm the results.
To this extent, we use and extend our generic framework for
analysing open source software projects. This framework,
presented in [
For the specific purposes of this article, we added a new module to compute activity distribution (i.e. the relative activity of each involved person), and different variants of activity are supported. In particular, we implemented the three types of activity defined in Section II-B. We also added a module for computing aggregation indices. Among others, the Hoover, Gini, and Theil index are currently supported. The existing statistic analysis and visualisation modules can directly exploit the information computed by the activity distribution module and the aggregation index module to produce statistical output representing the inequality indices.
Many types of distributions in which people are involved
correspond to the so-called Pareto principle: roughly 80% of
the effects stem from approximately 20% of the causes [
Answering the first research question of Section II-A boils
down to finding empirical evidence for the Pareto principle
in OSS project activity distribution. One should note the
difference between the Pareto principle and the related notion
of Pareto distribution [
The second research question of Section II-A corresponds to determining whether the Pareto principle is present throughout the entire life of the project, and whether it emerges, stabilises or disappears over time.
We will analyse the distribution and evolution of activity on the following OSS projects: Brasero (projects.gnome. org/brasero), Wine (www.winehq.org), and Evince (projects. gnome.org/evince). They have been selected based on a variety of factors: popularity, age size, availability of the necessary data sources for analysis, and so on. Some of the characteristics of the three selected projects are presented in Table I.
Figure 1 shows the cumulative activity distribution in the Brasero community, for three types of activity: commits made to the version repository, mails sent to the mailing list, and changes made to bug reports in the bug tracker. These distributions are shown for the last version of Brasero we analysed, namely the one available on November 2010. For the previous versions, with the exception of the earliest versions, we get similar results.
OSS project main programming language versioning system age (in years) size (in KLOC) # of commits # of mails # bug reports # commiters # mailers # bug reporters
TABLE I MAIN CHARACTERISTICS OF SELECTED OSS PROJECTS. THE REPORTED VALUES HAVE BEEN COMPUTED FOR THE LAST VERSION, NOVEMBER 2010. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
These distributions illustrate that there always is a small core team of persons that account for most of the activity. For the commit activity, 3 out of 193 persons carry out about 70% of the total number of commits. Even more striking is the fact that a single developer accounts for 60% of the total number of commits. For the mail activity, 7 out of 92 persons sent about 60% of all the mails. For the bug report change activity, 5 out of 253 persons carry out about 40% of all bug report changes.
While a detailed statistical analysis of the exact type of distribution is left for future work, we do find clear support for the Pareto principle: for the commit activity, 20% of the most active committers contribute to about 85% of all commits. For the mail (resp. bug report change) activity, 20% of the most active committers contribute with about 75% of all mails (resp. bug report changes).
Figure 2 compares the activity distribution between different types of bug report activities available in the bug tracker: submitting new bug reports, changing existing bug reports, and commenting on existing bug reports. It provides similar 0.8 0.7 distributions as those found in Figure 1.
To answer research question 3 of Section II-A, we determined the overlap between different categories of activities (committing, mailing, changing bug reports), by analysing those persons that were involved in different activities. Figure 3 presents the results of this analysis. The triplet notation (c%; m%; b%) used in each of the intersections corresponds to the percentage of activity of a particular individual that contributed to more than one activity category. For example, there is a person with (61%; 11%; 20%), indicating that he contributed to 61% of the total activity of the top 20 most active committers, to 11% of the mail activity of the top 20 most active mailers, and to 20% of the bug report change activity of the top 20 most active bug report changers.
As expected, we find a clear overlap of activity. The two most active committers (out of the top 20) are also very active mailers and bug report changers. In fact, the same two persons account for 67% of the top 20 commit activity, 34% of the top 20 mail activity and 27% of the top 20 bug report change activity in Brasero. We also observe that three of the 20 most active mailers are also active as top 20 bug report changers.
Note that the analysis process for obtaining the results in Figure 3 was manual, which explains the restriction to the top 20 most active individuals only. The reason is that it is quite challenging to automate a reliable identification of identities (logins, e-mail addresses, names) that correspond to the same person. As can be seen in Figure 3, the total sum of all contributors per activity category is 19 instead of 20 for committers and mailers. The reason for this is that, during the manual analysis, we observed that two different identities actually corresponded to the same individual (because he has used two different e-mail addresses or logins over time). Therefore, we merged the corresponding data into a single entity.
To find out how the distribution of activity evolves over time, we used the econometric aggregation measures introduced in Section II-C. Figure 4 displays the evolution of three indices (Hoover, Gini and Theil) for the commits in Brasero. Each data point in this figure corresponds to a different distribution such as the ones shown in Figure 1. We observe that, regardless of the index used, the values do not fluctuate a lot, and tend to stabilise over time. For Gini, for example, we see that the index remains most of the time between 0.8 and 0.9, indicating a very unequal distribution of commit activity for all observed versions. This corroborates what we already observed before: a low number of individuals contribute most of the commits.
Figure 5 shows how the Gini index differs across the different activity categories we analysed over time for Brasero: commits, mails and bug report changes. Again, the results correspond to what we observed in the distributions of Figure 1. In all cases, the activity is unequally distributed across individuals. This is especially the case for the commits (with a single committer accounting for 60% of the total number of commits), explaining the high value of the Gini index. For mail activity and bug report change activity, there is also an unequal distribution, but less flagrant than for the commits. This explains why their Gini index curve is below the one for the commit activity.
A study of the evolution of the Evince community provides similar results. Perhaps an important difference is that the development activity is a bit more equally distributed than for Brasero (where we found a single person responsible for 60% of the total commit activity).
Figure 6 shows the cumulative activity distribution for three types of activity for Evince: commits to the version repository, mails sent to the mailing list, and changes made to bug reports in the bug tracker. Evince has two top committers, each accounting for 15% of the total commit activity. The Pareto principle is also clearly present in Figure 6: 20% of all committers contribute to 80% of the total commit activity, 20% of all mailers contribute to 70% of the total mail activity, and 20% of all bug report changers contribute to 88% of the total bug report change activity.
Figure 7 provides a different view on the same data, obtained by manually analysing the top 20 of most active persons for each activity category. As for Brasero, the 4 most active persons of the Evince community contribute to each of the three activity categories. Together, they account for 35% of all top 20 commits, 26% of all top 20 mails, and 36% of all top 20 bug report changes.
Figure 8 shows the evolution of the three econometric aggregation indices on the evolution of Evince’s commit activity. After a steep startup phase, we see that the indices start to stabilise rapidly to a more or less stable value. This value is lower than for Brasero, since the distribution of commit activity is a bit more equally distributed for Evince.
Since Figure 8 reveals a similar pattern for all three indices, in Figure 9 we restricted ourselves to the Gini index only. We used it to compare the evolution of the commit, mail and bug report activity of Evince. For each of these activities, we observe that the Gini index stabilises rapidly to a more or less 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Fig. 6. Cumulative distribution of activity for Evince (November 2010). The distribution with blue circles corresponds to the commit activity. The distribution with red squares corresponds to the mail activity. The distribution with green triangles corresponds to the bug report change activity. constant value, indicating that the way in which the community is structured is fairly stable. For commits and mails, the high Gini index indicates that the activity is not equally distributed over the community members. For mails, the Gini index is lower so this activity is more spread over different persons.
A study of the Wine community reveals that the number of persons involved is much bigger than for the other two systems studied. This was already apparent from Table I.
Figure 10 also provides evidence for the Pareto principle. 11% of all committers account for a total of 90% of commits. Similarly, a total of 13% of bug report changers account for 80% of all changes.
As a side note, we observed that the use of logins and accounts in Wine was poorly structured. For example, many committers have 4 or 5 email addresses that are rarely used. This may be explained by the fact that the Wine community is very open: it is very easy for new persons to become active in this community.
Figure 11 shows the overlap of activity of the top 20 most active persons in each activity category for Wine. Again, we needed to merge two different identities corresponding to the same individual into a single identity (explaining a total sum of 19 instead of 20 for committers). In contrast to the previously analysed projects, none of the 20 most active persons contributed to three different activity categories. Another major difference was that the core group of active persons for Wine was significantly bigger, explaining the smaller percentages we obtained for the most active persons involved in a particular activity.
Figure 10 shows the cumulative activity distribution for Wine committers, mailers and bug report changers. The most active committer accounts for 13% of the total commit activity, the two most active bug report changers each account for 13% and 11% of the total change activity, respectively.
Concerning the mail activity of Wine, we observed that there is one huge mailer wineforum-user@winehq.org accounting for 48% of the total project mail activity. It turned out that this mailer was in fact an automated transcription of the discussion forum in the mail system. After excluding this outlier from the mail activity data set, we found that the most active mailer only accounts for 4% of the total mail activity.
Figure 12 compares the evolution of Wine’s commit activity using three different aggregation indices. We observe that the Gini and Theil indices are very high and continue to increase over time, indicating a very unequal distribution of activity, with a large group of inactive persons and a small group of active persons.
Figure 13 displays the evolution of the three types of activity for Wine over time, using the Gini index as aggregation measure. For mail activity, the Gini index is initially much lower than for the commit activity, but after a while its value starts to increase to comparable values. This high value is largely explained by the presence of a single artificial mailer (corresponding to the wineforum). For the bug report change activity, we also observe an increasing growth of the Gini index, revealing an increasing inequality of activity distribution over time. 0.9 0.8
The results shown in the previous section provide strong
evidence for the Pareto principle. The activity of contributors
to open source projects is not equally distributed: in all three
studied projects, a core group of persons appears to carry
out the majority of the work. This kind of behaviour may
be related to the way in which open source developer
communities are structured. According to [
Our empirical analysis showed the usefulness of applying
results from econometry to the analysis of the activity in
software ecosystems. We used three different aggregation
indices of statistical dispersion, the Hoover index, Theil index
and Gini index. They all gave similar results, i.e., they tend to
evolve in the same way for a given OSS project’s history.
This probably implies that one can freely choose any of
these indices to assess the evolution of activity distribution.
This corresponds to the findings of Vasilescu et al. [
For all three OSS projects we studied, the distribution of activity was initially more equally distributed, but over time the activity tends to become concentrated in a core group of persons that is significantly more active than the others. This knowledge is quite important, as the sudden disappearance of some members of the core group may have an important impact on the future of the software project. In other ways, we found empirical evidence of the so-called bus factor, the total number of key persons that would, if they were to be hit by a bus, lead the project into serious problems. Note that this was less the case for Wine, by far the biggest of the three projects, where the activity was more equally distributed over the most active committers than for the other two projects.
For all types of activities, we found a long tail of persons whose activity rate can be largely neglected: during the entire lifetime of the project, they contributed once or twice to one of the considered project activity categories (commits, mails and bug reporting). We also observed that, except for Wine, the most active project members take part in all these activities.
As a potential threat to validity, during our experiments,
we encountered some problems to determine which persons
contribute to different activity categories (Figures 3, 7, 11).
This was mainly a manual and error-prone process that took a
lot of effort. In the future we intend to automate this process.
To achieve this, an efficient identity merging algorithm is
needed that allows one to identify matches between entities
participating in different data sources. But for poorly
structured projects such as Wine this may still be quite problematic
and difficult to automate. We have studied this topic in more
detail in [
Some researchers focus on the study of core teams [
Because the data related to software development activity
is dispersed and incomplete, and due to the heterogeneity of
data formats used in software development, an important effort
must be made in order to collect and represent all available
data in a uniform way [
Numerous studies have found evidence for the Pareto
principle (or, more generally, a power law distribution) in
human-related networks [
In this article, we studied and compared the evolution of OSS project activity. Following the GQM paradigm, our main research goal was to understand how activity is distributed in OSS projects over time. We considered three categories of activity: committing data and code, sending mail and changing bug reports. We extracted and analysed such activity based on three different types of data sources: version repositories, mailing lists, and bug tracking data. We carried out an empirical analysis over three different long-lived OSS projects for which this data was available: Brasero, Evince and Wine.
For all three studied projects and for all considered activity categories, we found evidence for the Pareto principle. The activity distributions showed a strong inequality in the activity of different persons involved in an OSS project: there is a small group of very active members, and a much bigger group of largely inactive members. For two of the three studied projects, the core group of most active members takes part in more than one activity category. In Wine this was much less the case.
In order to gain understanding in how OSS projects evolve, we studied this inequality of distribution over time. To do so we relied on statistical techniques borrowed from econometrics. We applied three economic aggregation measures (the Hoover, Gini and Theil index). The evolution of activity distribution appeared to follow two kinds of behaviour. The first one is typical of a totally new project: at the beginning, the activity is more or less equally distributed, but quickly we observe a tendency towards a more unequal distribution where the activities become more concentrated in a small core team. In the second type of observed behaviour, the activity distribution is already imbalanced since the beginning of the project, and this imbalance continues to become more pronounced over time.
Studying who are the most active persons involved in each type of activity, we discovered that these persons are often very active in different activity categories. For Brasero and Evince, the two projects in which we observed this behaviour, the project is led by a small group of very active members wearing several hats at the same time. We have not yet been able to
identify the cause of overlaps between activities, because our definitions for measuring activity need to be refined further.
While Brasero and Evince show a similar evolution of activity distribution and similar overlaps between most active persons’ activity categories, the Wine software project appears to have a different behaviour. It has a significantly bigger community, there is significantly less overlap between activity categories, and we observed a higher inequality in the activity distribution. We can only speculate as to the causes of this.
While the work presented in this article is very promising, many challenges lay still ahead of us.
We intend to carry out a detailed statistical study of the activity distributions we presented in this article. In particular, we would like to find out which kind of statistical distribution they represent. Can we find evidence for power laws, Pareto distributions or other types of statistical distributions?
We also intend to analyse the activity distribution for subgroups and subprojects, and study how the structure and size of the project community and the software itself affects the type of activity distribution. Another aspect worthy of further study is the evolution of the activity of individuals involved in OSS projects. Can we find evidence for a learning curve for newcomers in a project? Does the activity of core members increase or decrease over time? Can we identify certain typical evolution patterns of activity?
Within an OSS project we would like to study the statistical correlation between different characteristics such as software quality, stakeholder activity, and software size. We also want to study how each of these characteristics correlate across different projects.
The notions of activity studied in this article, and the metrics used for computing them, are perhaps too high-level. We could define and use more fine-grained and more specific definitions, in order to get a more detailed picture of how project members interact, and in order to come up with new effort estimation and effort prediction models based on the activity of project members.
It remains an open question whether the type of activity distribution we observed in our empirical study is specific to open source projects. Our hypothesis is that this is indeed the case. To verify this hypothesis, we would like to repeat our experiment on proprietary, closed source software projects. The main challenge here it to get access to such data, in particular, the evolutionary data related to the activities of the developers involved in these projects.
Finally, the long term goal is to provide assistance or guidance, through interactive or automated tool support, for all stakeholders involved in an (open source) software project. Users may rely on information such as the bus factor to decide on using a particular open source software product. Developers may use it to identify the core developers and influential persons. Project managers may wish to control the activity distribution, to estimate or predict the effort, or to reduce the bus factor risk through a better distribution of activity.
The research is partially supported by (i) F.R.S.-FNRS FRFC project 2.4515.09 “Research Center on Software Adaptability”; (ii) research project AUWB- 08/12-UMH “ModelDriven Software Evolution”, an Action de Recherche Concerte´e financed by the Ministe`re de la Communaute´ franc¸aise - Direction ge´ne´rale de l’Enseignement non obligatoire et de la Recherche scientifique, Belgium.