Rugby is a process model for continuous software engineering which allows developers to continuously deliver prototypes and obtain feedback supporting software evolution. There is a reference implementation of Rugby with commercial enterprise tools used in university capstone courses. However, since these tools are expensive, there is a need to study less expensive alternatives which are available on the market to evaluate whether they can be used in Rugby. In this paper, we compare a second reference implementation with the existing one, focusing on the core use cases and non-functional requirements of Rugby.
Increasingly dynamic environments lead to shorter development cycles [FS14]. Continuous software engineering (CSE) refers to the organizational capability to develop, release, and learn from software in rapid cycles [Bo14] providing the capability for software evolution [HF10]. Rugby is a process model that proposes a development lifecycle for CSE [KAB+14]. It combines elements of Scrum [Sc04] and the Unified Process [JBR+99] with additional workflows to support CSE and software evolution: review management, release management and feedback management [KAB+14]. Developers work in a project-based organization with multiple projects to deliver executable prototypes and to obtain feedback. Bruegge et al. developed a reference implementation of Rugby that is applied in university capstone courses using commercial enterprise tools: JIRA, Bitbucket Server, Bamboo and HockeyApp [BKA15].
However, Rugby is not limited to these commercial tools. Expensive tools cannot
easily be used by individuals and organizations, e.g. open source projects or young
startups. The existing reference implementation also poses challenges, e.g. in terms of
usability, because the tools have a high learning curve. Therefore, we investigate
Rugby’s use cases for its three additional workflows and one Scrum core workflows: issue
management. We conducted a research [TKB15] which shows for each of these
workflows, there are many popular options available: Bugzilla, Redmine, Trac, GitHub
Issues and Mantis as the Issue Tracker, GitHub, SourceForge, GitLab, Klin, CodePlex
and Codeplan as the version control system (VCS), Jenkins, TeamCity, Travis,
Hudson, Wercker, Team Foundation Server and GitLab CI as the continuous integration
(CI) server, and Appaloosa, Crashlytics, GoCD and HockeyKit as the continuous
delivery (CD). Considering the tight collaboration between these categorie
In this paper, we created example projects with these tools to be able to compare them in each category [TKB15] with respect to the most important Rugby use cases for developers, managers, users and also with regards to configurability and flexibility. At the end, using GitHub tools, Jenkins and Crashlytics, which all target on the open source projects, we create a second reference implementation and use it in different example projects to compare and evaluate the differences of how the main use cases are implemented. We also give recommendations about which reference implementation can be used.
Rugby is a process model that includes workflows for CD. It allows part timers to work in a project-based organization with multiple projects for the rapid delivery of prototypes and products. Using CD improves the development process in two ways: First, Rugby improves the interaction between developers and customers with a continuous feedback mechanism. Second, Rugby improves coordination and communication with stakeholders and across multiple teams in project-based organizations with event based releases [KAB+14]. Figure 1 shows the integrated continuous workflow with abstract tools and transitions. The workflow starts each time a developer commits source code to the version control server, leading to a new build on the continuous integration server. If the build was built successfully and if it passed all test stages, the team can decide to upload it to the delivery server, which then notifies users about a new release. Each release includes release notes, which are collected automatically by the continuous integration server and can be edited in the manual release step if necessary. The user can download the release and recognize easily, which features and bugs were resolved in that. He can use an embedded mechanism to give feedback in a structured way. This feedback is collected on the delivery server and forwarded to the issue tracker [KA14].
Rugby is a customizable process model that can be adapted and refined to the team’s needs [KAB+14]. It supports CSE and software evolution with one Scrum core workflow together with three additional workflows: (1) issue management needs an issue tracker, (2) review management needs a VCS, (3) release management needs a CI and a CD server, (4) feedback management needs a CD server and an issue tracker [KAB+14]. These workflows are also customizable and can be included in other agile processes, in particular Scrum and Kanban [KAB+14].
For each of these workflows, we select alternatives and compare them with the corresponding commercial tools in the existing reference implementation with respect to the most important core use cases and non-functional requirements in each workflow. At the end of each section, there is a comparison table which shows scores for each feature of the tool (combination). Scores range from 1 (very poor quality or support) to 5 (excellent quality or support). In case a tool does not support a feature, a dash sign is shown (-).
Issue tracking is a software tool that enables the developers to record and track the status of all issues associated with each configuration object in the project [Pr05]. In this section, we compare JIRA by Atlassian and GitHub Issues for issue management.
JIRA has a customizable workflow for issues which makes it easy to tailor based on different needs in each project. The issue workflow consists of statuses and transitions that an issue goes through during its lifecycle modeled as a finite state automaton. The workflow can be edited or a new one can be created from scratch. JIRA supports sprint planning and sprint reviewing as defined in Scrum.
On the other hand, GitHub Issues, does not have customizable workflows for issues. However, a list of milestones can be configured with a corresponding due date. A milestone shows the last edit date, percentage of completed tasks, the number of open and closed issues and merge requests, which can give a general overview of the project progress. With some configurations, GitHub Issues can support agile project management. Using milestones and labels, it is possible to simulate sprint backlogs and versions which are not available by default [Ke15].
Therefore, the main advantages of JIRA in comparison with GitHub Issues are its customizable workflows for issues and support for variety of issue types as well as different reports and charts. GitHub Issues main advantage is its usability which is very simple and easy to work in comparison with JIRA. Relevant features for Rugby Customizable issue states and transitions Configuration possibilities Taskboard support Versions support Backlogs support Sprint planning/review support Usability JIRA 5 5 5 5 5 5 3
GitHub 1 1 3 4 (using Labels) 3 (using Milestones) 3 5
VCS is a system that keeps track of files and their history and have a model for concurrent access [Ot09]. In this section, we compare Bitbucket Server, formerly known as Stash, by Atlassian and GitHub for review management.
Both Bitbucket and GitHub use Git as a distributed VCS to control the source code versions. Features like merge requests (a.k.a. pull request), branch management and merge management are the most important functions [CS14]. Therefore, Bitbucket and GitHub function similarly in their core features. Bitbucket only supports sharing code files as snippets (known as Gists) with additional plugins. Its web interface lacks inline editing unless additional plugins are used.
In contrast to GitHub, Bitbucket’s web interface does not show tags. However, this does not affect its functionalities, since it supports all the GitHub’s features without needing to use labels. In terms of usability, GitHub is slightly better than Bitbucket. It is simple and everything is easy to find in the menus. One important difference between Bitbucket and GitHub is that they use different mechanism to show compare view (diff) [Pe15]. Although Bitbucket’s approach is more complex and has an overhead, it provides more accurate and useful merge request diff.
Relevant features for Rugby Merge requests Inline code commenting Web inline editing Sharing code files and snippets support Commit comments Accurate compare views Branch/merge management Collaborative code review Usability
Bitbucket Server 5 5 4 (using plugins) 4 (using plugins) 5 5 5 5 4
CI is a software development practice where developers integrate their work frequently, which leads to multiple integrations per day. Each integration is verified by an automated build, including test, to detect integration errors as quickly as possible [FF06]. In other words, CI is the process of automatically building and running tests whenever a change is committed. On the other hand, CD is a software engineering approach in which teams keep producing software in short cycles and ensure that the software can be reliably released at any time [Ch15]. Rugby uses an enterprise app store as the CD server, a web portal through which end users can access, download, install approved software applications, send feedback about the application and report crashes. In this section, we compare three combinations of CI and CD, Bamboo and HockeyApp, Travis and Crashlytics, and Jenkins and Crashlytics.
Travis is a maintenance free CI server, since it is a hosted service and all the installations and updates are performed server side. Although Jenkins can be a hosted service as well, it is usually use on premise, needing administration effort for installations and updates. Bamboo can be used as a cloud service or as an on premise solution. Jenkins does not use a database for storage which makes it flexible and portable. Only by copying the configuration, Jenkins jobs can be easily migrated across multiple instances which is not possible in Bamboo. Therefore, maintainability in Jenkins is easier.
Additional build agents are needed to scale CI servers horizontally. Bamboo licenses are priced per agents that need to be installed and configured. In Jenkins, unlimited amount of build agents is available and configuration of build agents is very easy. The build agents are configured via Jenkins UI and all of the tools can be installed automatically. Therefore, Jenkins is easier to scale than Bamboo.
A deployment project in Bamboo is a container for holding the software project which is deployed. Besides, plan branches represent a build for a branch in the version control system. When the plan branch build succeeds, it can be automatically or manually merged back into master. Bamboo deployments allow a plan branch to be deployed to a test environment and the feature source code can be tested and evaluated in a real server environment before the code is merged back to master. In Jenkins, there are available plugins to support deployment and branches as well, while in Travis they are supported by Travis’ configuration file. Concluding, all tools support deployment and branches. However, in Bamboo, when build jobs call deployments, it is not possible to go back to the build job to perform some post-deployment tasks. In addition, inability to provide input parameters, especially for deployment jobs, is a problem in Bamboo as well.
Each release includes release notes, which describe features and resolved bugs. The release notes are collected by the CI server and can be edited in the manual release step if necessary. Non of the CI solutions can create release notes automatically.
Code testing is the process of automatically building and running tests whenever a change is committed. Code integration is the process of automatically integrating the code after a committed change. All three CI tools support testing and integration.
It is not possible to get crash logs in Travis, without setting up scripts to upload them to a third-party service after completion of a build. In Bamboo and Jenkins, there are a lot of useful information regarding crash logs in test output display. Bamboo has an easy to use UI, Travis’ UI is even more simple. However, Jenkins’ UI looks out of date. Both HockeyApp and Crashlytics, notify the users about new releases and support feedback notification as well. They enable the CI server to automatically upload a new build and support the download of releases, new versions, push notifications and publication configurations. Relevant features for Rugby Build plan configuration New commit/branch detection Release notes creation Testing Integration Build status notification Deploy build to CD server Feedback/new release notification App version automatic upload Release download CI scalability CI maintainability CI usability App version download usability for the user Support of multiple mobile platforms
In Section 3.3, we mentioned that Rugby uses an enterprise app store as the CD server. In this section we compare the combinations of HockeyApp and JIRA, and Crashlytics and GitHub Issues. Both Crashlytics and HockeyApp store crash reports and errors as issues. However, since GitHub Issues does not support multiple issue types, it is not possible to convert issues to appropriate work items like bugs or improvements.
In Crashlytics, different user groups can be defined and different users can be added to them. HockeyApp supports five user roles: owner, manager, developer, member and tester. HockeyApp supports more feedback management features in comparison with Crashlytics; while the user can reply to the developer’s questions and his feedback records usage context, developer can pull them and reply to them.
Therefore, the main differences between HockeyApp and Crashlytics are the multiple mobile platforms support, user device registration, developer device management and voting mechanism to prioritize feedback requests. While HockeyApp supports different platforms, Crashlytics is only limited to iOS and Android. Although both of these tools have a good usability, HockeyApp is more simple and easier to use. While users can register their iOS and Android devices in HockeyApp, they can do so only for the iOS devices in Crashlytics. HockeyApp also supports developer device management which is not supported in Crashlytics.
Crashlytics takes into account that often a crash occurs and assigns it an impact level. It notifies when a specific crash is more critical than another one. As a particular crash is reported more and more, Crashlytics tracks that information and calls out the crashes that should be dealt with next. On the other hand, HockeyApp provides more depth and accuracy of crash logs. However, it does require more setup compared to Crashlytics. Relevant features for Rugby User device registration User feedback upload User reply to developer question User attach media Developer device management Developer feedback pull Voting mechanism Usage context record Feedbacks and analytics for developer Store crash reports and errors as issues Issue conversion to appropriate work item (issue) HockeyApp + JIRA 4 (iOS and Android) 5 5 5 5 5 5 5 5 5 5
Crashlytics + GitHub Issues 3 (iOS) 5 5 5 3 5 5 5 5 1
In this paper, we create a second reference implementation with tools that can be used for free in open source projects, using GitHub tools, Jenkins and Crashlytics, and compare it with the commercial reference implementation. The second reference uses GitHub Issues as the Issue Tracking, GitHub as the VCS, and Crashlytics as the CD server. For CI, there are two options to choose: Travis and Jenkins. Both of these CI servers are candidates to become the CI solution. Cloud vs On Premise of the code repository is the most important factor in choosing, followed by project type. If the project is open source, Travis would be the better option, because there is no setup time and fee. If it is a company project with privacy concerns, Jenkins is a better option, because the setup time is minimal and maintaining the server is rather simple.
Since JIRA, Bitbucket and Bamboo are all Atlassian tools, they can integrate seamlessly using minimal effort and have a high overall usability as well, while the learning curve might be high for new users. On the other hand, GitHub Issues, GitHub and Travis are from GitHub and they also integrate with each other. GitHub tools are simpler and have fewer features than Atlassian tools, so they are easier for new users and can be used for free in open source projects. In addition, maintenance is not an issue while working with GitHub tools, because they are all hosted in the cloud.
Considering the price plan of different tools, the proposed reference implementation using open source tools can be used for free, especially for small startups that work on public repositories. However, if private repositories and concurrent jobs in CI are necessary, the price varies and there is not too much difference between the two reference implementations in the terms of price. Using the commercial reference implementation for middle sized companies, with around 100 persons, can be expensive. For such middle sized companies, the open source reference implementation cost is about half of the commercial one.
Therefore, both the existing commercial reference implementation, using JIRA, Bitbucket, Bamboo and HockeyApp, and the proposed open source implementation, using GitHub Issues, GitHub, Travis/Jenkins and Crashlytics, cover most of Rugby’s important use cases. The commercial solution is a better choice when security and privacy are important and repositories should be private. On the other hand, when the project can be public, the open source solution should be preferred. Rugby can be implemented with different tools, either for commercial projects or open source projects. However, there is always a tradeoff between the tools quality and their price. It remains to compare other tools for their use in Rugby projects in the future works. [FS14] [Sc04] [JBR+99] [HF10] [KAB+14] [KKP+15] [BKA15] [Pr05] [KA14] [Ke15] [Ot09] [CS14] [Pe15] [FF06] [Ch15] [TKB15]