Multiple studies have consistently identified a prevalent issue in software projects, wherein they often fail to meet the initial expectations of end customers. The main reasons for software project failures often include setting unrealistic goals and not having clear requirements, among other things. At the same time, low-code platforms have gained prominence as a popular method for information systems development in the field of technology. They present a visual and model-driven paradigm that simplifies the process of creating software solutions. Our research team has been actively engaged in developing a DEMObased Low-Code Platform, DISME (Dynamic Information System Modeller and Executor), designed for the swift deployment of information systems. In this paper, after implementing an Information System for Cognitive Rehabilitation, NexusBRaNT, within DISME, the platform's usability underwent a comprehensive assessment encompassing qualitative and quantitative techniques. Employing the Think Aloud method for qualitative evaluation yielded insightful feedback across diferent facets of the platform. Participants lauded the platform for its engaging, intuitive, visually appealing, and user-friendly attributes. The subsequent quantitative assessment, employing the System Usability Scale, corroborated these observations, yielding an impressive overall usability score of 89.25%.
CEUR ceur-ws.org
Many studies have highlighted that numerous software projects do not meet the initial
expectations of end customers. In a comprehensive study cited from [
On the flip side, information systems (IS) and software, in general, often exhibit rigidity
and present challenges when it comes to making modifications, demanding substantial efort
to align with ever-evolving requirements and regulations. Traditional software development
infrastructure necessitates the use of a multitude of tools, such as Integrated Development
Environments (IDEs), modelling tools, Database Management Systems (DBMS), Object-Relational
(O-R) mapping frameworks, Graphical User Interface (GUI) editors, deployment and compilation
utilities, and more. Moreover, there is a growing global trend towards the creation of dynamic
and adaptive software that can either directly execute models or autonomously generate code
based on them. Low-Code Platforms (LCPs) represent an innovative approach that
consolidates these tools into a single system. This innovative method automates business operations,
reducing the need to switch between diferent tools and, more importantly, streamlining the
integration and maintenance of consistency among the implementation artefacts produced by
various technologies, as referenced in [
An LCP serves as a versatile hub for swift application development, deployment, execution,
and management. It leverages declarative, high-level programming abstractions, which
encompass model-driven and metadata-based programming languages, alongside simplified one-step
deployment processes. These platforms ofer comprehensive support for user interfaces,
business processes, and data services. They exhibit the potential to significantly reduce both the
time and costs associated with establishing, implementing, and sustaining various operational
processes. Furthermore, by streamlining these processes, LCPs create the opportunity to allocate
more of a company’s human resources to research and development tasks. This reallocation
ensures that the outcomes of their work more closely align with the specific needs and objectives
of the business [
With DISME (Dynamic Information System Modeller and Executor) reaching its alpha stage,
we recently conducted an experiment of developing a projected information system to
support cognitive rehabilitation, NexusBRaNT, in parallel, with a traditional software engineering
approach and with a low-code approach, in order to formally compare the needed efort and
system complexity. NexusBRaNT was designed to provide access to the BRaNT1 (Belief Revision
applied to Neurorehabilitation Therapy) project’s back-ofice and to assist healthcare
professionals, specializing in cognitive rehabilitation, including psychologists, neuropsychologists,
and therapists. The main features of the platform are patient management, neuropsychological
testing, and cognitive training. The results, published in [
The research contributions of this paper are providing an assessment of the usability of the developed DISME system on the health domain, through the analysis of NexusBRaNT.
User testing stands out as a widely acknowledged approach for pinpointing authentic issues
that could influence user performance and preference [
Section 2 presents our research context, which covers the DEMO-based Low-Code Platform and the implemented Information System to support Cognitive Rehabilitation. Section 3 presents the study’s context with participants’ characterization and the method and procedures followed. Section 4 presents the main contribution of this paper, namely the evaluation of the usability of DISME using qualitative and quantitative methods. Conclusions and future work are found in Section 5.
This section aims to ofer readers a thorough understanding of DISME. We will start with a comprehensive overview of DISME to ensure that readers are well-informed. Subsequently, we will delve into an examination of the information system utilized in this study, specifically within the health domain.
Despite the eforts made by LCPs to decrease entry barriers and facilitate application
development, there are concerns regarding their restricted usability, which acts as an inhibitor to
their adoption. Due to these usability constraints, novice citizen developers continue to rely
on IT department developers to create their applications. Furthermore, it is widely recognized
that utilizing LCPs necessitates some training, although the required amount of training is
comparatively less than that of traditional development approaches [
DISME is an open-source low-code software platform built upon the foundation of DEMO
methodology [
Regarding the drivers for adopting LCPs discussed in [
BRaNT is dedicated to creating technological tools that support at-home cognitive rehabilitation, harnessing the power of artificial intelligence, and is also committed to ofering solutions to enhance the resilience of healthcare systems. Its primary objective is to enhance the quality of life for patients by delivering tailored and more frequent interventions.
As part of this research project, one of the tools under development involves creating a cognitive profile that utilizes artificial intelligence to optimize the customization of prescriptions. This technology ofers therapists a range of tailored recommendations for highly personalized and adaptable cognitive rehabilitation, through a virtual reality-based simulation of daily life activities.
For access to the BRaNT project’s back ofice, an online platform called NexusBRaNT was projected. This web-based system is specifically designed to serve healthcare professionals specializing in cognitive rehabilitation, including psychologists, neuropsychologists, and therapists, among others. In order to ensure that this system efectively met the fundamental requirements of these health professionals, it was necessary to clearly specify their various requirements, making it adaptable and comprehensive.
A comprehensive analysis was conducted to identify the relevant transactions within the
NexusBRaNT domain. This analysis was achieved using the Transaction Description Table
(TDT), as introduced in [
The main features and tasks of this system include patient management, neuropsychological assessment management, and cognitive training management. Patient management functionalities enable health professionals to add new patients, input and browse clinical information, track patient progress, calculate absolute performance, and provide clinical interpretation. The system also facilitates the management of neuropsychological assessments, allowing users to create assessment sessions, input evaluation data from assessment instruments, view assessment results, and provide clinical interpretation based on the findings. Furthermore, the system includes cognitive training management capabilities, enabling users to create customized cognitive training programs, monitor their progress, and access detailed information about each training session.
We then conducted an extensive analysis to specifically identify the relevant fact types within the BRaNT domain, which encompass concept types and attribute types. These fact types pertain to instances regularly utilized in daily operations. This efort yielded a comprehensive and high-level perspective on all the concepts within the NexusBRaNT domain, abstracting away from their attributes. These fact types are represented in a diagram following the Concept and Relationships Diagram (CRD) proposed in [18, 19]. The CRD resolves any ambiguity associated with DEMO’s Fact Model keywords like entity and property by employing the more generic terms concept and attribute and treating both as facts. A portion of the resulting diagram is shown in Figure 2.
In summary, we pinpointed a total of 224 fact types within the NexusBRaNT case, which included 193 attribute types aggregated into 31 concept types. Of these 31 concept types, 20 are specifically dedicated to Neuropsychological Assessment Instruments available for registration, collectively comprising a total of 144 attributes. However, for the sake of diagram clarity, we have opted to showcase only a curated selection of 6 of these concept types. These six have been chosen to illustrate some of the supported neuropsychological assessment instruments within NexusBRaNT and their respective concept types.
Once we had established the core concepts and their relationships, the next step was to identify the relevant attributes linked to each concept. This process is facilitated through the Concept Attribute Diagram (CAD), an approach akin to, or an “expansion”, of the CRD introduced earlier, as outlined in [18, 19]. The CAD employs collapsible boxes to represent concepts. When you expand a box, it reveals the associated attributes, each displayed on a separate line. The left side of each line specifies the attribute’s value type, while the right side presents the attribute’s name. For attributes with categorical value types, a list of possible values is also provided, as described in [18, 19]. To maintain clarity, Figure 3 ofers a partial view of the CAD, featuring selected concepts with their expanded attributes.
In order to supplement the diagrams shown above with comprehensive textual information, we have the Fact Description Table (FDT), also proposed in [18, 19]. This table mirrors each concept from the diagram and provides a detailed description of all the associated attributes. The FDT functions as an exhaustive and trackable reference, establishing connections between attributes, their origins, and their transactional history within the system. A segment of the FDT for the NexusBRaNT case is available in Figure 4.
Within this representation of the FDT, we present an in-depth examination of each concept alongside its corresponding attributes. The table ofers a comprehensive overview of all attributes linked to each concept, providing details on their scope, source, concept name, value type, referenced concept or category values (depending on the value type), descriptions, and the tasks responsible for creating and modifying the concept. By structuring this information systematically, we gain valuable insights into the characteristics and interconnections of each concept within the system. This table functions as an invaluable resource for comprehending the diverse attributes associated with each concept, facilitating efective conceptual analysis.
This partial view of the complete information system model highlights the NexusBRaNT system’s intricate information requirements. This complexity led us to choose it as the ideal subject for our initial usability study of DISME.
We next present a characterization of the study’s participants, followed by the method and procedures for the qualitative and quantitative validations performed on the usability of DISME.
To assess the perceived functionality of the usability of NexusBRaNT implemented in DISME, we recruited a sample of health professionals in the field of psychology, with domain knowledge of the modelled processes and implemented system (N = 10, nine females and one male, Mdn age = 34, age range = 28-58 years). All participants have a Human and Social Sciences background. Namely, among the participants, two are currently pursuing a Bachelor’s Degree in Psychology, one is currently pursuing a Master’s Degree in Clinical, Health, and Well-Being Psychology, and the remaining seven are health professionals in the field of psychology (Scholar levels: Bachelors degree N = 2; Masters degree N = 7; and Doctoral degree N = 1). Notably, two of these health professionals are directly associated with the BRaNT project, adding valuable expertise and insights to the study.
A comprehensive study was conducted to evaluate the usability of DISME implementing NexusBRaNT. The study employed the Think Aloud method and the System Usability Scale (SUS) to gather qualitative and quantitative data on the platform’s usability.
The Think Aloud Method is a technique used in usability testing and cognitive psychology to gain insights into a person’s thought processes while performing a task. During the thinkaloud method, participants are asked to verbalize their thoughts, observations, and decisionmaking as they navigate through a task or interact with a system. They are encouraged to express their feelings, confusion, and any dificulties they encounter. This method allows researchers to understand the cognitive and perceptual processes of users in real time. By listening to participants’ verbalizations, researchers can gain valuable insights into the strategies, assumptions, and mental models users employ while interacting with a system. This helps identify usability issues, comprehension problems, and areas for improvement [20].
The System Usability Scale is a cost-efective and dependable measure used for evaluating the usability of systems on a global level. It serves as a usability scale that provides reliable insights into the user experience. The SUS is designed to assess the usability of a wide range of systems, including software, websites, and various technological interfaces. It ofers a standardized questionnaire that users can complete to evaluate the usability of a system. By employing the SUS, researchers, and practitioners can obtain valuable information regarding the overall usability of a system in a straightforward and eficient manner [ 21].
In these individual testing sessions, participants were introduced to the DISME platform’s fundamentals, including a brief explanation of low-code platforms. Then, the NexusBRaNT project and its main functionalities were presented. Finally, participants were guided through DISME’s Dashboard’s sections, providing an explanation of each section and its relevance to the project.
After familiarizing the participants with DISME and the NexusBRaNT case and their respective functionalities, participants were initially given the opportunity for free exploration of the DISME platform, allowing them to familiarize themselves with its interface, features, and functionalities. Throughout the session, participants had the opportunity to ask questions and provide feedback to ensure their understanding and engagement with the platform. Following this exploration phase, participants were asked to perform a series of specific tasks that represented everyday activities in the NexusBRaNT domain. The tasks were Patient Registration, Clinical Information Registration, Neuropsychological Assessment Session Creation, Neuropsychological Assessment Instrument Registration, which included the automatic calculation of the patient’s cognitive profile, and Cognitive Training Program Creation.
During the tasks, participants were encouraged to vocalize their thoughts and provide feedback using the Think Aloud method. This allowed researchers to gain insights into the participants’ cognitive processes, challenges encountered, and perceptions of DISME’s usability. Valuable feedback and suggestions were provided, highlighting both areas for improvement and the strengths of DISME.
Additionally, the System Usability Scale was administered to participants upon completion of the tasks. The SUS questionnaire, consisting of ten statements related to usability, was rated on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The responses were collected and analysed to quantitatively assess participants’ perceptions of DISME’s usability.
The combination of the Think Aloud method and the SUS questionnaire yielded valuable insights into the user experience of health professionals using a low-code platform for managing information in the context of cognitive rehabilitation.
In this section, we will be presenting the results of the qualitative and quantitative methods employed in this study, namely the Think Aloud and the System Usability Scale methods.
The Think Aloud sessions conducted with the participants yielded valuable feedback that emphasized several areas for improvement and positive aspects of DISME. We next present highlights of the evaluation.
Participants provided feedback on the task organization in the platform. Some suggested combining all tasks into a single section instead of separating them based on process initiation. Others preferred the separation but recommended grouping tasks initiated after a process by process type, particularly for instrument registration in assessment sessions, or by neuropsychological executive function assessed. As for the tasks’ panels, there was a request to order them in chronological order according to the system’s operation, as some participants found the current task order based on popularity confusing. It was also mentioned by one participant that notifications accumulated quickly, indicating the need for a more manageable notification system. Almost all participants suggested automating the filling of normative data in neuropsychological assessment instruments based on patient information, rather than manual insertion using provided images. However, DISME’s implementation of NexusBRaNT strictly adhered to the traditional system’s requirements, which required manual insertion. Implementing the change in DISME’s NexusBRaNT implementation would be straightforward.
Concerning the positive aspects of the platform, participants found the platform highly engaging. One participant even showed interest in using it for personal projects. They praised the intuitive and visually appealing design, describing the forms as simple and easy to navigate. The choice of colours received widespread appreciation, with one participant stating they wouldn’t change them and another comparing them favourably to the traditionally implemented NexusBRaNT with a dark colour scheme. Participants liked DISME’s colour range and the ability to easily customize the Dashboard’s task colours, which is done in a straightforward manner in the customization component and is directly applied to the Dashboard. The platform’s intuitiveness and responsiveness were commended for making tasks manageable and enjoyable. One participant expressed satisfaction with the well-organized task sections, facilitating navigation and task completion. Regarding the graphical representation of pending and executed tasks, one participant mentioned its usefulness for colour-blind individuals, which was their case (had dificulty seeing green and red), highlighting the description of task names when hovering over the tasks’ process-type coded colours.
These findings from the Think Aloud sessions provide valuable insights for improving the usability and user experience of the DISME platform. The improvement suggestions and negative feedback highlight specific areas where enhancements can be made, such as task organization, filtering options, automatic data filling, and the notification system. The positive feedback reinforces the platform’s intuitiveness, user-friendly interface, visually pleasing design, and overall ease of use, reinforcing its potential as a valuable tool in the low-code development ifeld. Incorporating these suggestions and addressing the identified issues will contribute to the ongoing development and optimization of the DISME platform, ensuring, in the future, an even better user-friendly experience, and increasing the already very high score obtained in the System Usability Scale presented in the next section.
The quantitative analysis of DISME was conducted using the System Usability Scale, which consists of ten statements that participants rated on a scale from 1 (strongly disagree) to 5 (strongly agree). The collected responses were then subjected to thorough analysis, enabling a quantitative evaluation of participants’ perceptions regarding the usability of the DISME platform.
The first statement of the SUS used to assess the usability of DISME focused on participants’ likelihood to frequently use the system. The responses from the ten participants involved in the study showed a high level of positive feedback and had an average response of 4.5, signifying a strong agreement with the statement.
The analysis of the fourth question from the SUS sheds light on participants’ perceptions regarding the need for technical support when using DISME, with an average answer of 1.8, indicating moderately low agreement with the statement. While most participants felt comfortable using the platform independently, some expressed varying degrees of agreement regarding the need for assistance, suggesting challenges or perceived complexities. Clear instructions and user-friendly interfaces are crucial to minimizing the perceived need for external support. Enhancements aimed at improving user self-suficiency and reducing potential barriers to independent usage could further enhance the platform’s overall usability.
To calculate the overall usability score, the ratings were converted into a numerical scale using the SUS equation that can be seen in Equation 1. The obtained average score for DISME was 89.25%. The maximum score obtained was 100%, while the minimum score was 80%, as can be seen in Figure 5, which indicates a favourable level of usability and user satisfaction. This score exceeded the threshold of 68%, which is typically considered above average [21]. Specific aspects of usability, such as ease of use, system complexity, and confidence in using the platform, received positive ratings from the participants.
SUS = ((Q1-1) + (5-Q2) + (Q3-1) + (5-Q4) + (Q5-1) + (5-Q6) + (Q7-1) + (5-Q8) + (Q9-1) + (5- Q10)) x 2,5 (1)
Comparing the usability scores of DISME implementing NexusBRaNT as a low-code platform with the traditionally developed NexusBRaNT system reveals intriguing findings. DISME obtained an average usability score of 89.25%, demonstrating a high level of usability and user satisfaction. The use of low-code development techniques likely contributed to its favourable score by providing a more intuitive and user-friendly experience, allowing for eficient navigation and task completion.
In contrast, the traditionally developed NexusBRaNT system achieved an impressive average usability score of 92%, ranging between 75% and 100%, as can be seen in Figure 6, with a sample of health professionals in the field of psychology with domain knowledge of the modelled processes and implemented system (N = 16, fiteen females and one male, Mdn age = 27.5, age range = 21 - 48 years) [22]. The applied method was identical to the one applied to the tests in the low-code platform, diferentiating in the task of creating clinical information that did not exist in this case. All participants also had a Human and Social Sciences background. Namely, among the participants, six are currently pursuing a Master’s Degree in Psychology, and the remaining ten are health professionals in the field of psychology. Notably, two of these health professionals are directly associated with BRaNT [22]. The obtained scores indicate a generally high level of usability, although there may be some variability in the user experience. The traditional development approach signifies a robust and well-designed system that meets the needs and expectations of its users.
The comparison suggests that both the low-code DISME platform and the traditionally developed NexusBRaNT system ofer commendable levels of usability. While the DISME platform benefits from its low-code approach in terms of user-friendliness, the traditionally developed NexusBRaNT system demonstrates its efectiveness in delivering a reliable and highly usable system. These findings highlight the potential benefits of low-code platforms in terms of ease of use and eficiency, while also acknowledging the strengths of traditional development methods in achieving high usability standards.
In this paper, we have presented an evaluation of our research eforts in developing a DEMObased Low-Code Platform, DISME, for rapid implementation of information systems. Specifically, we evaluated the usability of an Information System for Cognitive Rehabilitation, NexusBRaNT, implemented on DISME.
The usability of DISME was evaluated using both qualitative and quantitative methods. The qualitative evaluation through the Think Aloud method provided valuable feedback on various aspects of the platform, including task organization, interface design, and user experience. Participants found the platform engaging, intuitive, visually appealing, and user-friendly. Reifnements were suggested, such as task grouping, improved filtering options, and automation of data filling. The positive feedback reinforced the platform’s strengths and highlighted its potential as a valuable tool in low-code development. The quantitative evaluation using the System Usability Scale (SUS) confirmed these findings. Participants agreed that the platform was user-friendly and supported their professional activities efectively, with the overall usability score for DISME being calculated to be 89.25%.
As the main limitation, we identify the small sample size. Therefore, results must be interpreted with caution and generalizability of the findings may be limited.
In conclusion, our research has contributed to improving usability of low-code platforms by evaluating the usability of an information system for cognitive rehabilitation implemented on a DEMO-based low-code platform. The results demonstrate the potential of DISME to allow impressive gains of efectiveness in software development, while achieving high levels of usability.
As future work we foresee additional evaluations in the context of new implementations in other scenarios, continuous refinement of DISME, including support for collaborative modelling and development. With ongoing eforts, we envision DISME becoming an invaluable tool for rapidly developing efective and user-friendly information systems in various domains. Engineering XIV, Springer International Publishing, Cham, 2021, pp. 133–152. [17] D. Pacheco, D. Aveiro, D. Pinto, B. Gouveia, Towards the x-theory: An evaluation of the perceived quality and functionality of demo’s process model, in: D. Aveiro, H. A. Proper, S. Guerreiro, M. de Vries (Eds.), Advances in Enterprise Engineering XV, Springer International Publishing, Cham, 2022, pp. 129–148. [18] B. Gouveia, D. Aveiro, D. Pacheco, D. Pinto, D. Gouveia, Fact Model in DEMO - Urban Law Case and Proposal of Representation Improvements, 2021, pp. 173–190. doi:10.1007/ 978-3-030-74196-9\_10. [19] D. Pacheco, D. Aveiro, B. Gouveia, D. Pinto, Evaluation of the perceived quality and functionality of fact model diagrams in demo, in: D. Aveiro, H. A. Proper, S. Guerreiro, M. de Vries (Eds.), Advances in Enterprise Engineering XV, Springer International Publishing, Cham, 2022, pp. 114–128. [20] D. W. Eccles, G. Arsal, The think aloud method: what is it and how do i use it?, Qualitative Research in Sport, Exercise and Health 9 (2017) 514–531. doi:10.1080/2159676X.2017. 1331501. [21] J. Brooke, Sus: A quick and dirty usability scale, Usability Eval. Ind. 189 (1995). [22] E. Cunha, Nexus brant backofice para brant, 2023. URL: http://hdl.handle.net/10400.13/ 5102.