Researches come up with scientific criteria to compare diefrent tools and spend huge eofrt to run the comparison. The presentation of the comparison results is an open issue. The “Ultimate Comparison Framework” is one solution enabling a) collection of evaluation data using markdown and b) presentation of the data set in a web application. 1 https://ultimate-comparisons.github.io/ultimate-deployment-toolcomparison/ 2 https://ultimate-comparisons.github.io/ultimate-IoT-platform-comparison/
Oliver Kopp
The accumulation of knowledge is an essential condition for a field to be scientific and
to develop [
The development of the Ultimate Comparison Framework was driven by a) sustaining
search results in the context of finding the most fitting tool for a task and b) oefring a
framework for comparative studies crafted by students in the context of their software
engineering trainings (cf. “Fachstudie” [
The framework is called “Ultimate Comparison Framework”, because it oefrs creation of multiple “Ultimate Comparisons” of diefrent tools in diefrent settings. The name “Ultimate Comparison” stems from the claim that the framework is easy to use (covering creation and maintenance of data as well as presenting data) and that it will be used by many researchers to present their research results. This paper is a first step into this direction by making it known to a broader community.
Users of the framework are a) researchers wanting to sustain their survey results, b) researches investigating other surveys, and c) industry users interested in introducing a new tool or framework and aiming for a scientifically-grounded comparison of existing work.
Already published “ultimate comparisons” include: Comparison of Cloud Deployment and Management Tools1, Comparison of IoT Platforms2, Comparison of TimeSeries Databases3, Comparison of Message Brokers4, Comparison of Graph Libraries for JavaScript5, Comparison of Web-based IDEs6, Comparison of Literature Management Software7, and Comparison of LATEX Building Helpers8. The list of available comparisons is constantly updated at https://ultimate-comparisons.github.io/. In other words: the Ultimate Comparison Framework is not limited to a fixed set of tools, but can be used to compare tools, where comparison critera can be defined and the result can be presented as table.
The idea stems from the PaaSfinder web application [
This paper demos the framework by showing the Ultimate Comparison of Open
Source Time-Series-Databases [
A person interested in comparison results, opens the web page presenting the Ultimate Comparison. He sees a table showing the compared tools as row headings and criteria as column headings. In each cell, the matched criteria are listed. For instance, if a tool is oefred both under MIT and a proprietary license, the cell for “License” shows “MIT” and “proprietary”. The person can sort the table by clicking on a criteria name. It is also possible to show details on each tool. An example table is presented in Fig. 1. 3
Data in a comparison may get outdated or data is missing. Then, a user can update the data using the typical GitHub flow 9. The user searches for the Markdown file containing the data of the tool (for instance the time-series database Timely), modifies it accordingly, and sends a pull request.
In case a new comparison should be set up, its name, description and comparison criteria need configured. This is enabled by a YAML configuration file. Each criterion bundles a set of possible values: Possible data types for a criterion are text, enums, and numbers. For instance, the code license can be enumerated and results of performance 3 https://tsdbbench.github.io/Ultimate-TSDB-Comparison/ 4 https://ultimate-comparisons.github.io/ultimate-message-broker-comparison/ 5 https://ultimate-comparisons.github.io/ultimate-graphframework-comparison/ 6 https://ultimate-comparisons.github.io/ultimate-webIDE-comparison/ 7 https://ultimate-comparisons.github.io/ultimate-reference-managementsoftware-comparison/ 8 https://ultimate-comparisons.github.io/ultimate-latex-makers-comparison/ 9 https://guides.github.com/introduction/flow/ measurements can be provided. Each criterion is input as Markdown heading. In case of text, the text has to be put below the heading. In case of an enum, all matching enums are listed as markdown list (e.g., “- Apache-2.0” below “## License” in the case of a single license). In case of numbers, the number is given below the heading.
When designing a criterion, it is important to ensure that it can be used for comparing diefrent tools. For instance, if performance of tools depend on the environment, the chosen environment should be described. Alternatively, an Ultimate Comparison for each diefrent environment could be setup.
To set up an ultimate comparison, basic knowledge of GitHub pages and the usage of CI/CD tools is necessary. Furthermore, writing YAML is an essential software engineering skill. To input data into an ultimate comparison, the knowledge of using GitHub as well as entering Markdown are necessary.
The web interface is implemented using Angular. The data is converted from Markdown to JSON using Java on a CI server (currently TravisCI). The JSON is read by the web interface running on the client side in the browser. The web interface is hosted in the branch “gh-pages” of the respective comparison and oefred to users by the GitHub Pages oefring10. The source code of the implementation is open source and available at https: //github.com/ultimate-comparisons/ultimate-comparison-framework. 5
This paper presented the Ultimate Comparison Framework oefring a collaborative comparison framework. It stores the data in Markdown files enabling the usage of arbitrary text editors to add or modify data. The data is rendered as static HTML page showing all criteria, the evaluated tools, and the fulfillment of each criterion in a table. It is an open discussion point whether tables are the best option to present comparison results. Future work has to evaluate other rendering possibilities such as charts. 10 https://pages.github.com/
Facts such as write performance can also be captured in an Ultimate Comparision. The framework currently oefrs to fetch the last commit and decide on that fact if development is stalled. There is currently no way to trigger complete setup of each system, measure the performance, and update the result in an Ultimate Comparison. This is left as future work, for instance for TSDBBench11.
Currently, there is no advanced analysis functionality oefred. One idea is to oefr clustering of the results. Currently, this can only be done by defining an enum criterion, where each enum value presents one cluster. Then, the user can sort by that criterion.
The next development ideas are a) oefring a browser-based user interface to input data (instead of relying on GitHub) and b) using Wikipedia tables as data input; either as data source for displaying or as data source for synchronizing the local Markdown ifles. Future work includes measurement of the time required to publish and maintain results in comparison to other approaches such as scientific publications or tables in WikiPedia articles.
Acknowledgments We want to thank Stefan Kolb for the idea and inspiration of this tool Further, we want to thank Andreas Bader, Armin Hüneburg, and Christoph Kleine for discussions on the UI and driving the implementation. This work is partially funded by the Federal Ministry for Economic Aafirs and Energy (BMWi) projects Industrial Communication for Factories (01MA17008G), NEMAR (03ET4018), and SmartOrchestra (01MD16001F). 11 https://tsdbbench.github.io/