Development bots are being used by maintainers of GitHub repositories to perform repetitive or error-prone tasks. While multiple approaches have been proposed in the past to identify such bots in GitHub repositories, they are either inaccurate, not general enough (e.g., based exclusively on specific types of activities) or require a large amount of data that makes it dificult to use them in practice. In this paper, we show that information about the activities performed by contributors in the diferent software repositories to which they contribute can be used as a basis to distinguish bots from human contributors. These activities can belong to one of 24 distinct activity types such as deleting a branch, creating a tag, publishing a release, opening an issue, commenting on a pull request, and so on. Relying on a dataset of 830K+ activities performed by 713 contributors to GitHub repositories, we identify a number of features allowing to distinguish bots from human contributors. These features are the number of activity types, distribution of inequality in the time taken to perform these activities, distribution of inequality in the number of activity types performed in each repository, ratio between the number of owners and repositories, and the median time taken to switch between repositories to perform another activity. In follow-up work we aim to leverage these features to provide a bot identification model that considers all the activities performed by contributors.
software development. A prerequisite for considering bots is the ability to identify their presence in software The use of collaborative software development practices development activities. While there exists several bot in social coding platforms (such as GitHub, GitLab, Gitea detection models such as BIMAN [7], BoDeGHa [8] or and the likes) has become omnipresent [1, 2]. These social BotHunter [9], they are usually based on limited number coding platforms allow project contributors to perform of activity types and features (e.g., repetitive patterns various activity types in their software repositories, such in the comments they create in issues and pull requests, as opening an issue, opening a pull request, commenting commmit messages, or the presence of bot in their name) on a pull request, pushing a commit and reviewing code. or they require a significant amount of data to reliably
Social coding platforms also enabled the use of develop- distinguish bots from humans in practice.
ment bots that, just like human contributors, can perform In this article, we identify a series of features to
quantivarious activities in software repositories [3, 4, 5]. The tatively distinguish bots from human contributors based
main distinction is that their activities are automated, typ- on relevant information such as their activity types, the
ically based on some external trigger or schedule. Bots number of repositories they are involved in, the time it
enable the automation of error-prone and repetitive tasks takes to carry out or shift between activity types, and so
to allow human contributors to focus on other more cog- on. Such a set of distinguishing features will be used to
nitive activities [5]. Some bots are even among the top create a bot identification model to eficiently and
relicontributors in the software repositories to which they ably identify whether a repository contributor is a bot
contribute [
While the use of bots in open source software reposi- GitHub Events API. The distinguishing features could tories can alleviate maintainer workload, their presence also be used to classify bots into diferent categories (i.e., poses challenges for empirical software engineering re- diferent types of bots), to understand the role that bots searchers that aim to study socio-technical aspects of are playing in collaborative software development, and the positive or negative impact they may have on the SATToSE’23: Seminar on Advanced Techniques & Tools for Software collaborative development process. Evolution, June 12–14, 2023, Salerno, Italy Leveraging on a GitHub activity dataset that we have a"lexnaantadrraej.danec.cahni@daummboanras.mac@.bue m(Ao.nDs.eacca.bne);(Nto.mC.mhiednasm@buamraomn)s;.ac.be created in earlier work [3], the identification of distin(T. Mens) guishing features in the current article will be based on a 0000-0002-5824-5823 (A. Decan); 0000-0003-3636-5020 (T. Mens) dataset of 713 GitHub repository contributors, of which © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License 305 bots and 408 humans, accounting for a total of 830K+ CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) activities belonging to 24 diferent activity types that are performed in 46,190 GitHub repositories from 25th November 2022 to 9th March 2023. 2. Related Work the author name, (ii) repetitive commit messages, and (iii) features related to files changed in commits. Golzadeh et al. [8] developed BoDeGHa that uses a classification model to identify bots that are involved in commenting issues and pull requests. They further extended this approach to git commit messages [14]. Chidambaram et al. [15] extended BoDeGHa by leveraging the predictions from multiple repositories to determine the type of account. Abdellatif et al. [9] developed BotHunter, that extends BoDeGHa by integrating additional information from the account’s profile and from the commits, issues and pull requests the account is involved in. We are not aware of any bot identification tool that consider all the activity types performed by a software project contributors, justifying the relevance of our current study that covers a wider range of activities.
All the above-mentioned tools and studies consider a restricted set of activity types (e.g., committing code, commenting on pull requests and issues) performed by bots in software repositories. Since bots perform many more activity types, considering all their activities would help to increase bot identification accuracy, while at the same time enabling the detection of bots that are performing tasks not considered by existing bot identification tools.
On bot-based studies. Many empirical studies on
bots have been conducted in the past. Some of these
studies involved analysing the working style of bots [
Erlenhov et al. [
80 bots and 208 humans from the dataset, and their 3,308 corresponding activities. This left us with 305 bots and 408 humans, accounting for a total of 830,503 activities. Table 1 summarises the final dataset that will be used for our analysis.
than humans.
3.2. Statistical method of activities. For example, a bot that keeps dependencies Whenever appropriate, for the features that will be stud- up-to-date will create pull requests only, and is unlikely ied in Section 4 we carry out statistical tests to compare to push commits directly to the repository or to comment the distribution values for bots and human contributors, some issue. On the other hand, we expect human conusing the non-parametric Mann-Whitney U test (a.k.a. tributors to perform a wider range activities across the Wilcoxon rank-sum test) since most of the considered repositories to which they contribute. For example, human contributors close or merge incoming pull requests in the repositories they maintain while they create pull activity type across the repositories (i.e., creating pull rerequests and issues in the external repositories they con- quests). On the other hand, depending on the repository, tribute to. To verify this hypothesis, we computed for a human contributor may be involved in a limited numeach contributor the number of activity types performed. ber of activity types (e.g., opening an issue) or a larger Fig. 2 shows the distribution of this number of activity number of activity types (e.g., pushing commits, closing types, distinguishing between bot and human contribu- or merging pull requests, closing and commenting issues, tors. creating releases).
24 21 s pe17 y t y it itv13 c a f o re 9 b m u n 5 1 bots
We observe a clear visual diference in line with our intuition, with a median number of 3 activity types for bots compared to 13 for humans. To confirm that there is a significant diference between the number of activity types performed by bots and human contributors, we performed a Mann-Whitney U test between the two distributions. The null hypothesis is rejected ( < ) indicating that there is a statistically significant diference between the number of activity types performed by bots and human contributors in software repositories. The efect size turned out to be large ( =0.88).
ity types than human contributors.
Intuition: Bots are more consistent than humans in performing their intended activity types across the repositories they contribute to.
1.0
tributor the Gini coeficient of its number of activity types across repositories. Fig. 3 shows the distribution of these Gini coeficients. One can visually observe that the number of activity types performed by bots are more equally distributed (i.e., Gini is closer to 0) across repositories than for human contributors. To statistically confirm our intuition, we performed a Mann-Whitney U test between both distributions. The null hypothesis was rejected ( < ) indicating a statistical diference between the two populations, with a large efect size ( = 0.76).
formed by bots tend to be more equally distributed across repositories than the number of activity types performed by humans.
humans, and tend to be used within repositories belonging to the same organizations/owners.
Not only bots are involved in less activity types than human contributors, but we expect them to be more con- We expect bots to be active in more repositories than sistent in performing these activity types across the mul- humans as they do not have any workload restrictions. tiple repositories to which they contribute compared to We also expect bots to be active in more repositories that of humans. Consider again the case of a bot keeping belonging to the same owner or organization since it dependencies up-to-date. Such a bot, regardless of the would make sense to deploy a bot in all the repositories repository it is deployed in, will consistently have one of the same organization or owner as long as the bot is satisfactory. For humans, on the other hand, we ex- owners or organizations. On the other hand, a ratio close pect to observe a more diverse behaviour, in the sense to 0 indicates that most of the repositories the contributor that human contributors might choose to go outside the is active in belong to the same owner or organization. boundaries of the repository owner or of its organization in order to contribute to external repositories owned by 1.0 another user or organization.
0.8
The above intuition is based on the idea that bots are not sufering from context switching to the same extent as human contributors. As automated tools, bots can easily be active in multiple repositories at the same time, while we expect human contributors to focus their workload on one repository at a time. As a consequence, we expect the time required to go from one repository to another one to be greater for human contributors than for bots.
To capture this intuition, we computed for each contributor the time to switch between two repositories. bots
humans
regularly than humans.
Bots are not subject to the same limitations as human contributors in performing their tasks. For example, bots do not have to sleep or eat, and can aford to work at any time of the day. Their tasks are usually triggered by external events (such as a newer version of a dependency) or by activities made by other developers (e.g., a pull request is submitted to the repository and a bot evaluates
more regular basis than human contributors.
Bots are used in software repositories to perform automation of error-prone and repetitive tasks, and they are frequently among the top contributors in some software repositories. While the use of bots in open source software repositories can alleviate maintainer workload, their presence poses challenges for empirical software engineering researchers that aim to study socio-technical aspects of software development. A prerequisite for considering bots is the ability to identify their presence in software development activities. This paper is a first step towards building a bot identification model taking into account the various activities that contributors do on GitHub.
In this paper, based on a dataset of 830K+ activities made by 305 bots and 408 human contributors, we proposed five features that can be used to distinguish bot and human contributors, and we showed through statistical tests that these features efectively difer between these two types of contributors. We found that bots tend to be specialized in the sense they are involved in a smaller number of activity types than humans. We showed that bots tend to equally distribute their intended activity types across multiples repositories, and that bots are usually involved in repositories belonging to a smaller set of distinct owners or organizations. We also found that bots take less time to switch between the repositories they are involved in and that they tend to perform their activities more regularly than humans do.
In future work, we plan to rely on these features to create and evaluate a classification model distinguishing between bots and humans based on their activities. Since contributor activities are obtained from the events returned by the GitHub Events API, and since this API returns at most 300 events, we will also evaluate the impact of the number of activities that need to be provided to the model on its accuracy. If it turns out that the model is still accurate enough based on at most 300 activities, we will implement the model as part of a tool that can be used in practice on a large scale without requiring access to other data sources than a simple call to the GitHub API.