As part of quality management, it is important to ensure that Git repositories meet certain quality criteria depending on the type of the respective project. In academia, for instance, repositories of finished research projects should always comprise a readme file providing usage instructions. The range of heterogeneous resources within modern Git repositories leads to a large number of conceivable quality criteria, thus increasing the efort for validating Git repositories against the project type specific quality criteria. At the same time, there is a lack of approaches and tools supporting this validation. Hence, this paper proposes and discusses an approach based on RDF and SHACL for validating a given Git repository against a set of quality criteria defined for its intended project tyQpuea.Re, a working prototype implementing the approach, is provided and discussed.
Today, the version control software Git1 and related platforms like GitH2uabre standard tools for developing and sharing software. Version control systems store data within so called repositories in the form of a filesystem tree1[]. Developers can connect to the repository and edit the files in the tree as needed. Every committed change is registered and stored such that older revisions of the files can be viewed retrospectively. On top of this core functionality, platforms like GitHub provide a continuously growing number of powerful features like issue tracking, release management, wikis, CI/CD2][, and many more. As a result, modern Git repositories comprise many heterogeneous resources beyond pure software. Further increasing the complexity, diferent relationships exist between the resources and the users of a repository: There are one-to-one relationships, e.g., one repository can have exactly one description, one-tomany relationships, e.g., one repository can comprise one or more branches, and many-to-many relationships, e.g., one or more issues can be assigned to one or more contributors.
In software engineering, there are numerous tools and practices for producing high-quality
software [
CEUR Workshop Proceedings Git repositories, there is a range of conceivable quality criteria. In fact, all resources of a Git repository can theoretically be the subject of some quality criteria, thus making it dificult to evaluate the quality of one or even multiple Git repositories that have to follow certain criteria across an organization consistently. At the same time, there is a lack of tools supporting the validation of Git repositories against such quality criteria, thus motivating the present paper.
In this paper, an approach based on the Resource Description FrameworRkD(F) [
We provide a prototypical tool callQeudaRe3 implementing the approach. The tool allows users to easily check wether Git repositories of interest comply with the quality criteria they should fulfill according to their intended project type. Even though this paper focuses on repositories of academic projects, the approach and thereby the tool can readily be applied to other kinds of Git repositories.
The remainder of this paper is organized as follows: Sectio2ninvestigates important foundations and related work. In Sectio3n,the approach is explained thoroughly. Then, Sectio4n focuses on the implementation of the approach by introducing our working prototype. Subsequently, the approach and the implementation are discussed in Sec5t,iboenfore drawing a conclusion in Section6.
In the beginning, this section introduces a set of useful quality criteria with respect to literature and our academic background, which is the first contribution of this paper. Afterwards, the candidate technologies for implementing the proposed approach are investigated.
Although Git and related platforms like GitHub are standard tools in software development today,
there is few scientific literature explicitly discussing quality criteria for Git repositories. Instead,
the available literature mainly discusses topics around so called data repositories for storing
datasets. Examples of such data repositories provide interactivity and support collaboratively
working on the data6[
3https://github.com/uniba-mi/quar(evisited 2022/09/07); pronounced likequare, the Latin word forwhy, which is a reference to the tool’s ability to provide explanations why a repository’s validation did not succeed. Actually, one could understand a modern Git repository as a highly collaborative and featurerich form of a data repository mainly for the software development domain. This is supported by the fact that papers lik9e] [mention code as a part of the data stored in research data repositories. Regarding quality indicators for code, the paper mainly elaborates on code reproducibility. To achieve this, lists of required dependencies, documentation, user guides, and in the best case virtual runtime environments provided via tools like Doc4koerrplatforms like Code Ocea5n are recommended.
Apart from the scientific literature, there are posts on blogs and Q&A sites addressing quality indicators of Git repositories. For instance, answers to questions on StackOver6floawnd Quora7 state that comprehensive documentation, ease of installation, up-to-dateness, the activity on the repository, the friendliness of the community, the extensibility of the provided code, and the usage of software testing are important indicators.
In the end, the repository maintainers decide which quality criteria they want to enforce,
thus calling for a flexible approach. Based on our academic expertise and the literature, we
compiled a first set of generic quality criteria categories that can be useful for academic Git
repositories (and others). Tabl1e gives an overview of the considered quality criteria categories
on the -axis and the afected Git repository resources on th e-axis. As shown, we distinguish
ifve categories 8: Existence criteria check if a resource is available at qaulaln,tity criteria check
if the correct number of a resource is availabnlea,ming criteria check if a resource is named
appropriately,content criteria check if a resource comprises the correct contents, asntadtus
criteria check a resource’s status.
4https://docker.com(visited 2022/09/07)
5https://codeocean.com(visited 2022/09/07)
6See https://stackoverflow.com/questions/2252743(v8isited 2022/09/07)
7See https://www.quora.com/How-do-you-measure-a-good-github-reposito(rvyisited 2022/09/07)
8There are numerous other conceivable quality criteria and criteria categories which, however, cannot be
covered here comprehensively. For instance, one could impose an exotic criterion addressing the quality of writing
of some file’s content (cf. [
We define that a repository is of high quality if ∀ ∈ ((, )) , where is the set of all quality criteria of the intended project tyapned is a predicate that evaluates to true if complies with .
One strength ofRDF [
An alternative for specifying constraints in the form of restrictionsRDonF graphs is to
leverage the OWL 2 Web Ontology LanguagOe W(L) [
The following section describes how these RDF-related technologies can be employed to tackle the problem motivating this paper.
Modern platforms like GitHub typically provide comprehensive APIs and wrappers for accessing and manipulating repositories in code. Given such an API, the straightforward option for testing whether a Git repository complies with sets of quality criteria is to directly access the API or employ one of the available wrappers to check the individual quality criteria directly on the raw repository data. However, this approach has the drawback that an appropriate representation of the quality criteria and an engine for validating the repository against them has to be implemented single-handedly. For flexibility, it is thus more expedient to exploit already available technologies that sufice for the use case. In our specific caseR,DF is well-suited for creating a generic and abstract intermediary representation of a Git repository because 1) platforms like GitHub already provide explicit Uniform Resource LocatUoRrsL(s), a subset 9https://github.com/RDFLib/pySHACL(visited 2022/09/07) of IRIs, for most of the repositories’ resources and 2) the graph structuresRDofF are able to naturally model the complex relationships between the heterogeneous repository resources and its users.
Regarding the actual validation, boOtWh L in combination with a semantic reasoner and SHACL in combination with aSHACL validator are capable of checking wether RaDnF representation of a Git repository complies with certain quality criteria. In the former case the quality criteria would be modelled as class expressions, in the latter as a separate shapes graph. However, there is an decisive advantage oSfHACL: Semantic reasoners forOWL likePellet terminate as soon as one violation of a class expression is encountered. As a result, only the ifrst violation can be reported to the user. Applied to our use case, users would only receive a report mentioning one violated quality criterion even if the Git repository of interest potentially comprises many more. In contrastS,HACL validators likpeySHACL continue even if one or more constraints are violated and ultimately provide a full report of all violations. Therefore, SHACL was chosen for the approach that is mainly proposed and discussed in this paper. For completeness, our tool also features OaWnL-based approach, which will be mentioned in Sections4 and 5, too.
The remainder of this section uses the project tyfinpieshed research project ( ) as a running example to explain thSeHACL approach. With respect to the currently supported criteria from Table1, we define that repositories of finished research projects, have to comply with the quality criteria depicted in the upper section of Ta2b.lNeote that the selected criteria reflect our understanding of a finished research project, i.e., a project whose development has finished but whose results, code etc. should still be usable by others. For comparison, the table shows another project type, nameliynternal documentation ( ). Since repositories with this intended project type are for internal use, we impose less constraints for it.
The following sections will explain how the three-step process from Sec1tiiosnimplemented using theSHACL approach. Using this approach, the goal of the process is to evaluate the formula∀ ∈ ((, )) from Section2.1 by checking the compliance of the repository’s RDF representation with a shapes graph.
As described previously,SHACL uses a separate shapes graph to specify the constraints for anotherRDF graph. In the context of this paper, the created shapes graph provides specifications for the supported project types. Every project type specification includes constraints on the properties that entities of the project type have to comply with as define d b∈y . To give an example, Figure1 (a) shows the fragment of the shapes graph definin g 5 ∈ . The depicted properties with the leadinsgh prefix originate from the SHACL vocabulary 5[]. More specifically, the sh:property property links a project type entity to so called property shapes which are structures for defining the constraints. Thseh:path property defines the path in the RDF graph to which the constraint applies. In the example, the propersthi:emsinCount and sh:maxCount are leveraged to define the constraint that an entity w itahs its intended project type has to possess exactly one instance of a properhtays_branch that links the entity to a branch entit1y0. Similarly, shapes for a l l ∈ are created. The exact design of each shape has 10Section4.2 provides more examples regarding the implementation of quality criteria.
that repositories of projects with the type finished research project ( )/internal documentation ( ) have to comply with.
∈
Quality Criterion ∈
Quality Criterion 1 2 3 4 5 6 7 8 9
There shall be no open issues. There shall be a license.
There shall be a readme file.
The readme file from 10 shall include the sections Installation and Usage. The repository shall be private.
There shall be a repository description.
There shall be a readme file.
Category State Quantity
Existence Quantity
Quantity Naming Existence Existence Content Existence Content Category State Existence Existence Content to suficiently capture the constraint but can otherwise be chosen according to taste because the structure of thReDF representation is not determined yet. Nevertheless, we recommend designing the shapes as concise as possible.
types:FinishedResearchProject y t r e p o r p : h s sh:path props:has_branch sh:maxCount sh:minCount (a) 1 https://github.com /uniba-mi/quare h c n a r b _ s a h : s p o r p https://github.com /uniba-mi/quare
/tree/dev https://github.com /uniba-mi/quare /tree/main (b) props:has_name props:has_name "dev" "main" tion of QuaRe’s GitHub repository.
In SHACL, the shapes graph is independent from thReDF graph that is validated against the shapes graph. To ensure compatibility between the shapes graph and the repository representation, the repository representation has to be created based on the same properties that are used for the constraints in the shapes graph and their structure. The naive option would be to just create the representation such th atoaflalll supporte d in the shapes graph could be validated based on them. This way, the time to create the representation becomes unnecessarily long and the representation itself unnecessarily big due to the diversity of possible quality criteria, though, without any benefit. To mitigate this problemR,DtFhe representations of repositories could be cached by exploitiRnDgF’s serialization capabilities. However, caching comes with the risk that older versions of repositories are validated. This is problematic because the most recent versions might have new problems regarding the quality criteria while some problems might already have been fixed. For this reason, it is more expedient to choose another option which is to createmainimal representation just-in-time that only comprises the information that is necessary to check the constraints of the intended project type. Due to the diversity of possible quality criteria, one cannot rely on the API to directly provide exactly the information that is required to represent ea∈c h with respect to the shape structures determined in the first step, though. Instead, the derivation of each triple from the available repository data has to be implemented manually as the algorithm in Al1gorithm shows: After initialization, the wrapper is used to fetch the repository data that is relevant for each . Based on the retrieved data, RDF triples are derived that suficiently reflect the characteristics of the repository with respect to the SHACL constraints. The derived triples constitute thmeinimal RDF representation of the repository.
Algorithm 1 Generating a minimal RDF representation of a Git repository.
Require: The
of the repository, the set of quality criteria of the intended project ty,pe and an empty set of tripl es, i.e., the RDF representation of the repository.
To give an example, Figure1 (b) shows a fragment of theRDF representation ofQuaRe’s GitHub repository3. As depicted, two instances of thperops:has_branch property are created that link the repository entity to entities representing the branches available in the repository, i.e., one branch entity with the namdev and one branch entity with the nammeain. The result of the second step is therefore aRnDF representation of the Git repository that is adequate in the sense that it reflects the repository appropriately with respect to the properties used in the shapes graph such that it can be validated against the shapes graph in the upcoming third step. for ∈ do
← () ← . ℎ ( ← () ← ∪ {} end for return ) ▷
The derivation depends on the respectiv e .
Finally, aSHACL validator implementation, in our capsyeSHACL, is used to validate the Git repository’sRDF representation from the second step against the shapes graph from the first step. If theRDF representation complies with the project type specific constraints defined in the shapes graph, the repository is of high quality as specified by the formula in Secti2o.1n. As depicted in Figure1, QuaRe’s repository features two branches while only exactly one branch is permitted by 5 ∈ , thus causingpySHACL to report that this constraint is violated when the repository is validated again st .
Based on theSHACL (and theOWL) approach, the single-page applicatioQnuaRe3 was implemented, which allows testing one or multiple GitHub repositories against the quality criteria of their intended project type via a web interface. The web interface is implemented using S11velte and is connected to a backend written in Python which is executed on the same machine. For working withRDF and ontologies, the backend employs thrdeflib 12 and Owlready213 libraries as well aspySHACL and Pellet. For interaction with the GitHub API, tPhyeGithub14 wrapper is leveraged. In its current stage, the application comprises two pages, which are described in the following sections.
The Validation page (cf. Figure 2) provides a form for entering an arbitrary number of GitHub repositories and their respective intended project types. By pressing t+heand − buttons users can add or remove input fields as needed. Users can also provide their personal GitHub access token, which is required if they want to validate private repositories or make multiple validation requests in short succession. Via a switch, users can choose if the backend should use the SHACL or theOWL approach. When the form is submitted using the designated button, one individual request for each repository name and project type combination is sent to the backend triggering the execution of the chosen approach. If the validation in the backend succeeds, a green check mark is displayed. Otherwise, a red error symbol is shown accompanied by a button revealing thpeySHACL or Pellet explanation when pressed. Since the raw explanations are not useful for users unfamiliar wiStHhACL and OWL, a basic template-based verbalization feature was implemented that extracts the parts of the explanation addressing the reasons why the validation failed and transforms them in natural language. The verbalized explanations are presented alongside the raw explanations. Furthermore, there is a button for saving the form inputs locally such that they do not have to be entered every time.
11https://svelte.de(vvisited 2022/09/07) 12https://github.com/RDFLib/rdfli(bvisited 2022/09/07) 13https://github.com/pwin/owlready(2visited 2022/09/07) 14https://github.com/PyGithub/PyGithu(vbisited 2022/09/07) using the SHACL approach was issued. As shown, the validation has failed and the button revealing the raw and verbalized explanations has been pressed. Note that the screenshot was taken when the repository was still almost empty which is why basically all constraints of were violated.
The other page provided in the web interface is thSpeecification page (s. Figure 3) where the
available project type specifications can be viewed. Users are able to choose if they want to view
theSHACL specifications, as defined by the employed shapes graph, or the OWL specifications,
as defined by the used class expressions. The screenshot shows that there are currently four
supported projects types, namelfinyished research project , internal documentation, i.e., and
from above, ongoing research project, and teaching tool. To give an example of a project
type specification, the tab of theteaching tool project type is expanded revealing its quality
criteria as defined by theSHACL shapes graph in the form of a list. Each list entry corresponds
to another quality criterion. Using thhaes_branch example from Section3, the forth entry
in the list defines that thReDF representation of a Git repository containingteaaching tool
has to comprise at least two instances of htahse_branch property. The final entry shows a
more sophisticated constraint stating that there has to be a node that is 1) reachable via the
has_readme and has_section properties and 2) has the literal valUuseage. The combination
of thesh:qualifiedValueShape, sh:pattern, and sh:qualifiedMinCount properties define
a so calledsh:QualifiedMinCountConstraintComponent [
In Section2, we already mentioned the comprehensive validation reportSsHofACL as one of its main advantages compared to the alternaOtiWveL approach. Apart from thisS,HACL’s underlying principle of separating constraints, i.e., the shapes graph, and actual data is another strong point because it facilitates reviewing constraints for the supported project types. In contrast, the ontology of tOheWL approach includes both class expressions and the repository representation impeding the readability. Thus, StHheACL approach surpasses theOWL approach in terms of developer friendliness and maintainability, too.
Regarding our implementationQ,uaRe lacks certain features that are planned to be added in
the future. First, we want to implement that users can edit the specifications directly in the app
on theSpecification page, instead of editing them manually in the backend. Although this feature
is important forQuaRe, its implementation was postponed because it addresses non-trivial
topics beyond the focus of this paper, i.e., the mapping oSfHACL and OWL expressions to
and from user interfaces (cf.1[
An assessment of the tool’s performance was also conducted. For this, the time it takes for the backend to perform theSHACL and theOWL approach was measured on a standard desktop PC. As repositories, the top twenty trending GitHub reposito1r5iewsere used, which are well maintained and rather large. As intended project types, we u seadnd . Figure 4 shows that theSHACL approach is faster in both scenarios even thougPhellet stops the validation as soon as a violation is encountered. Hence, tShHe ACL approach provides a complete picture of the violationasnd is also faster. Furthermore, the figure shows that the project type does not significantly afect the runtime. This is due to the fact that fetching the repository data from GitHub is the most time-consuming part of the process. In fact, the steps 1/2/3 from Section1 account for 0.32%/91.75%/7.89% of the total duration using StHheACL approach and for 0.10%/76.49%/23.36% using theOWL approach on average, based on our measurements. Accordingly, comparatively large (in terms of branches, issues, etc.) repositories account for the depicted outlier1s6. Finally,QuaRe only supports GitHub repositories at the moment. Support for other platforms can be implemented as long as APIs and/or wrappers are available. 30 25 20 s d n co15 e S 10 5
SHACL, TF
OWL, TF
SHACL, TI
OWL, TI
This paper proposed a SHACL based approach for validating a Git repository against a set of
quality criteria specified by the type of project within the repository and compared it to an
alternativOe WL based approach. For this, we also provided and discussed an implementation
of both approaches calleQduaRe. It should be emphasized that the general idea of specifying
and validating requirements for certain artifacts baseRdDoFn representations of them could be
applied in other contexts as well, even for software and software systems. In fact, conversions
and interfaces between the Unified Modeling LanguageUM(L) and SHACL [
16To mitigate the impact of their size, tRhDeF representations of repositories could be cached by exploiting RDF’s serialization capabilities. As discussed in Secti3o.n2, caching is not a good option for our use case, though.