Many studies have highlighted that numerous software projects do not meet the initial expectations of end customers. In a comprehensive study cited from [1], which included specific case studies, a survey involving 800 IT (Information Technology) managers [2,3], it was revealed that 63% of software development projects failed, 49% exceeded their budget, 47% incurred higher maintenance costs than anticipated, and 41% fell short of fulfilling user and business requirements. Ibraigheeth et al. analysed various reports related to project failures, ultimately compiling a list of factors contributing to this high failure rate. These factors include setting unrealistic project objectives, incomplete requirements, insufficient stakeholder and user involvement, issues with project management and control, inadequate budget allocation, evolving or inconsistent requirements and specifications, lack of proper planning, ineffective communica-tion, and the utilization of new technologies for which software developers lacked the necessary experience and expertise [4].
On the flip side, information systems (IS) and software, in general, often exhibit rigidity and present challenges when it comes to making modifications, demanding substantial effort to align with ever-evolving requirements and regulations. Traditional software development infrastructure necessitates the use of a multitude of tools, such as Integrated Development Environments (IDEs), modelling tools, Database Management Systems (DBMS), Object-Relational (O-R) mapping frameworks, Graphical User Interface (GUI) editors, deployment and compilation utilities, and more. Moreover, there is a growing global trend towards the creation of dynamic and adaptive software that can either directly execute models or autonomously generate code based on them. Low-Code Platforms (LCPs) represent an innovative approach that consolidates these tools into a single system. This innovative method automates business operations, reducing the need to switch between different tools and, more importantly, streamlining the integration and maintenance of consistency among the implementation artefacts produced by various technologies, as referenced in [5,6,7].
An LCP serves as a versatile hub for swift application development, deployment, execution, and management. It leverages declarative, high-level programming abstractions, which encompass model-driven and metadata-based programming languages, alongside simplified one-step deployment processes. These platforms offer comprehensive support for user interfaces, business processes, and data services. They exhibit the potential to significantly reduce both the time and costs associated with establishing, implementing, and sustaining various operational processes. Furthermore, by streamlining these processes, LCPs create the opportunity to allocate more of a company's human resources to research and development tasks. This reallocation ensures that the outcomes of their work more closely align with the specific needs and objectives of the business [8,7].
With DISME (Dynamic Information System Modeller and Executor) reaching its alpha stage, we recently conducted an experiment of developing a projected information system to support cognitive rehabilitation, NexusBRaNT, in parallel, with a traditional software engineering approach and with a low-code approach, in order to formally compare the needed effort and system complexity. NexusBRaNT was designed to provide access to the BRaNT1 (Belief Revision applied to Neurorehabilitation Therapy) project's back-office and to assist healthcare professionals, specializing in cognitive rehabilitation, including psychologists, neuropsychologists, and therapists. The main features of the platform are patient management, neuropsychological testing, and cognitive training. The results, published in [9], were astonishing, as the low-code approach needed only around 5% of the time the traditional approach took, and we found a system complexity reduction of around 86% with DISME, roughly measured by comparing lines of code (LoC) of the traditional approach vs DISME's database records.
The research contributions of this paper are providing an assessment of the usability of the developed DISME system on the health domain, through the analysis of NexusBRaNT.
User testing stands out as a widely acknowledged approach for pinpointing authentic issues that could influence user performance and preference [10]. In order to evaluate the usability of NexusBRaNT integrated within DISME, we enlisted a group of healthcare professionals specialized in psychology. These experts possessed a deep understanding of the modelled processes and the implemented system.
Section 2 presents our research context, which covers the DEMO-based Low-Code Platform and the implemented Information System to support Cognitive Rehabilitation. Section 3 presents the study's context with participants' characterization and the method and procedures followed. Section 4 presents the main contribution of this paper, namely the evaluation of the usability of DISME using qualitative and quantitative methods. Conclusions and future work are found in Section 5.
This section aims to offer readers a thorough understanding of DISME. We will start with a comprehensive overview of DISME to ensure that readers are well-informed. Subsequently, we will delve into an examination of the information system utilized in this study, specifically within the health domain.
Despite the efforts made by LCPs to decrease entry barriers and facilitate application development, there are concerns regarding their restricted usability, which acts as an inhibitor to their adoption. Due to these usability constraints, novice citizen developers continue to rely on IT department developers to create their applications. Furthermore, it is widely recognized that utilizing LCPs necessitates some training, although the required amount of training is comparatively less than that of traditional development approaches [11].
DISME is an open-source low-code software platform built upon the foundation of DEMO methodology [12]. It facilitates the creation of organizational models and diagrams, enabling the specification of processes, information flow, responsibilities of both human and software elements, rules, and other organizational artefacts. DISME diverges from the usual low code approach because, instead of generating code for a static version of the organizational processes, it treats the organization as a living system, basing itself on the principles of the Adaptive Object Model [13,14]. Models can be designed, detailed, and made live immediately. To accomplish this, DISME offers two primary functional interfaces. The first is the System Modeller, which focuses on system specification through the use of various diagrams, forms, and tables. The second is the System Executor, responsible for the day-to-day execution of processes and information flow, following the specifications outlined in the System Modeller and leveraging the real-time information system of the enterprise running it. The System Executor consists of two primary components: the Dashboard, which serves as the user interface for interacting with organizational tasks; and the Execution Engine, which orchestrates the execution of rules and flow of information and processes based on the comprehensive specifications defined in the System Modeller interface. A more detailed description of DISME and its components can be found in [15].
Regarding the drivers for adopting LCPs discussed in [11], it is important to highlight that these drivers apply universally to all LCPs, including DISME, which acknowledges and addresses many of the inhibitors that hinder the adoption of LCPs. By proactively tackling these challenges, DISME aims to facilitate a seamless and efficient adoption of LCPs within the current software development process. In [15] we explore in detail how DISME addresses all inhibitors. In this paper, we underline how DISME addresses the usability inhibitor by incorporating graphical components throughout its interface to manage all model elements, including forms, drag-anddrop components, and the diagram editor. The dashboard, which serves as a central hub for users to carry out their organizational tasks, has been thoughtfully designed using modern usability patterns and technologies like Bootstrap. We aimed for DISME to offer a user-friendly interface, promoting easy navigation and seamless interaction with its various features. The study presented in this paper aims to validate and assess the usability of DISME in terms of the final user of an information system.
BRaNT is dedicated to creating technological tools that support at-home cognitive rehabilitation, harnessing the power of artificial intelligence, and is also committed to offering solutions to enhance the resilience of healthcare systems. Its primary objective is to enhance the quality of life for patients by delivering tailored and more frequent interventions.
As part of this research project, one of the tools under development involves creating a cognitive profile that utilizes artificial intelligence to optimize the customization of prescriptions. This technology offers therapists a range of tailored recommendations for highly personalized and adaptable cognitive rehabilitation, through a virtual reality-based simulation of daily life activities.
For access to the BRaNT project's back office, an online platform called NexusBRaNT was projected. This web-based system is specifically designed to serve healthcare professionals specializing in cognitive rehabilitation, including psychologists, neuropsychologists, and therapists, among others. In order to ensure that this system effectively met the fundamental requirements of these health professionals, it was necessary to clearly specify their various requirements, making it adaptable and comprehensive.
A comprehensive analysis was conducted to identify the relevant transactions within the NexusBRaNT domain. This analysis was achieved using the Transaction Description Table (TDT), as introduced in [16,17]. The TDT offers a well-structured and in-depth description of each task. It's important to note that, in this context, the terms transaction and task are used interchangeably. In the table, each task is accompanied by a detailed description, often directly sourced from the system's requirements. Additionally, the table specifies the conditions or rules that need to be verified for the execution of subsequent tasks, as well as any pertinent time constraints associated with each task, as discussed in [16,17]. An excerpt from this table is displayed in Figure 1.
The main features and tasks of this system include patient management, neuropsychological assessment management, and cognitive training management. Patient management functionalities enable health professionals to add new patients, input and browse clinical information, track patient progress, calculate absolute performance, and provide clinical interpretation. The system also facilitates the management of neuropsychological assessments, allowing users to create assessment sessions, input evaluation data from assessment instruments, view assessment results, and provide clinical interpretation based on the findings. Furthermore, the system We then conducted an extensive analysis to specifically identify the relevant fact types within the BRaNT domain, which encompass concept types and attribute types. These fact types pertain to instances regularly utilized in daily operations. This effort yielded a comprehensive and high-level perspective on all the concepts within the NexusBRaNT domain, abstracting away from their attributes. These fact types are represented in a diagram following the Concept and Relationships Diagram (CRD) proposed in [18,19]. The CRD resolves any ambiguity associated with DEMO's Fact Model keywords like entity and property by employing the more generic terms concept and attribute and treating both as facts. A portion of the resulting diagram is shown in Figure 2.
In summary, we pinpointed a total of 224 fact types within the NexusBRaNT case, which included 193 attribute types aggregated into 31 concept types. Of these 31 concept types, 20 are specifically dedicated to Neuropsychological Assessment Instruments available for registration, collectively comprising a total of 144 attributes. However, for the sake of diagram clarity, we have opted to showcase only a curated selection of 6 of these concept types. These six have been chosen to illustrate some of the supported neuropsychological assessment instruments within NexusBRaNT and their respective concept types.
Once we had established the core concepts and their relationships, the next step was to identify the relevant attributes linked to each concept. This process is facilitated through the Concept Attribute Diagram (CAD), an approach akin to, or an "expansion", of the CRD introduced earlier, as outlined in [18,19]. The CAD employs collapsible boxes to represent concepts. When you expand a box, it reveals the associated attributes, each displayed on a separate line. The left side of each line specifies the attribute's value type, while the right side presents the attribute's name. For attributes with categorical value types, a list of possible values is also provided, as described in [18,19]. To maintain clarity, Figure 3 offers a partial view of the CAD, featuring selected concepts with their expanded attributes.
In order to supplement the diagrams shown above with comprehensive textual information, we have the Fact Description Table (FDT), also proposed in [18,19]. This table mirrors each concept from the diagram and provides a detailed description of all the associated attributes.
The FDT functions as an exhaustive and trackable reference, establishing connections between attributes, their origins, and their transactional history within the system. A segment of the FDT for the NexusBRaNT case is available in Figure 4. Within this representation of the FDT, we present an in-depth examination of each concept alongside its corresponding attributes. The table offers a comprehensive overview of all attributes linked to each concept, providing details on their scope, source, concept name, value type, referenced concept or category values (depending on the value type), descriptions, and the tasks responsible for creating and modifying the concept. By structuring this information systematically, we gain valuable insights into the characteristics and interconnections of each concept within the system. This table functions as an invaluable resource for comprehending the diverse attributes associated with each concept, facilitating effective conceptual analysis.
This partial view of the complete information system model highlights the NexusBRaNT system's intricate information requirements. This complexity led us to choose it as the ideal subject for our initial usability study of DISME.
We next present a characterization of the study's participants, followed by the method and procedures for the qualitative and quantitative validations performed on the usability of DISME.
To assess the perceived functionality of the usability of NexusBRaNT implemented in DISME, we recruited a sample of health professionals in the field of psychology, with domain knowledge of the modelled processes and implemented system (N = 10, nine females and one male, Mdn age = 34, age range = 28-58 years). All participants have a Human and Social Sciences background. Namely, among the participants, two are currently pursuing a Bachelor's Degree in Psychology, one is currently pursuing a Master's Degree in Clinical, Health, and Well-Being Psychology, and the remaining seven are health professionals in the field of psychology (Scholar levels: Bachelors degree N = 2; Masters degree N = 7; and Doctoral degree N = 1). Notably, two of these health professionals are directly associated with the BRaNT project, adding valuable expertise and insights to the study.
A comprehensive study was conducted to evaluate the usability of DISME implementing Nexus-BRaNT. The study employed the Think Aloud method and the System Usability Scale (SUS) to gather qualitative and quantitative data on the platform's usability.
The Think Aloud Method is a technique used in usability testing and cognitive psychology to gain insights into a person's thought processes while performing a task. During the thinkaloud method, participants are asked to verbalize their thoughts, observations, and decisionmaking as they navigate through a task or interact with a system. They are encouraged to express their feelings, confusion, and any difficulties they encounter. This method allows researchers to understand the cognitive and perceptual processes of users in real time. By listening to participants' verbalizations, researchers can gain valuable insights into the strategies, assumptions, and mental models users employ while interacting with a system. This helps identify usability issues, comprehension problems, and areas for improvement [20].
The System Usability Scale is a cost-effective and dependable measure used for evaluating the usability of systems on a global level. It serves as a usability scale that provides reliable insights into the user experience. The SUS is designed to assess the usability of a wide range of systems, including software, websites, and various technological interfaces. It offers a standardized questionnaire that users can complete to evaluate the usability of a system. By employing the SUS, researchers, and practitioners can obtain valuable information regarding the overall usability of a system in a straightforward and efficient manner [21].
In these individual testing sessions, participants were introduced to the DISME platform's fundamentals, including a brief explanation of low-code platforms. Then, the NexusBRaNT project and its main functionalities were presented. Finally, participants were guided through DISME's Dashboard's sections, providing an explanation of each section and its relevance to the project.
After familiarizing the participants with DISME and the NexusBRaNT case and their respective functionalities, participants were initially given the opportunity for free exploration of the DISME platform, allowing them to familiarize themselves with its interface, features, and functionalities. Throughout the session, participants had the opportunity to ask questions and provide feedback to ensure their understanding and engagement with the platform. Following this exploration phase, participants were asked to perform a series of specific tasks that represented everyday activities in the NexusBRaNT domain. The tasks were Patient Registration, Clinical Information Registration, Neuropsychological Assessment Session Creation, Neuropsychological Assessment Instrument Registration, which included the automatic calculation of the patient's cognitive profile, and Cognitive Training Program Creation.
During the tasks, participants were encouraged to vocalize their thoughts and provide feedback using the Think Aloud method. This allowed researchers to gain insights into the participants' cognitive processes, challenges encountered, and perceptions of DISME's usability. Valuable feedback and suggestions were provided, highlighting both areas for improvement and the strengths of DISME.
Additionally, the System Usability Scale was administered to participants upon completion of the tasks. The SUS questionnaire, consisting of ten statements related to usability, was rated on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The responses were collected and analysed to quantitatively assess participants' perceptions of DISME's usability.
The combination of the Think Aloud method and the SUS questionnaire yielded valuable insights into the user experience of health professionals using a low-code platform for managing information in the context of cognitive rehabilitation.
In this section, we will be presenting the results of the qualitative and quantitative methods employed in this study, namely the Think Aloud and the System Usability Scale methods.
The Think Aloud sessions conducted with the participants yielded valuable feedback that emphasized several areas for improvement and positive aspects of DISME. We next present highlights of the evaluation.
Participants provided feedback on the task organization in the platform. Some suggested combining all tasks into a single section instead of separating them based on process initiation. Others preferred the separation but recommended grouping tasks initiated after a process by process type, particularly for instrument registration in assessment sessions, or by neuropsychological executive function assessed. As for the tasks' panels, there was a request to order them in chronological order according to the system's operation, as some participants found the current task order based on popularity confusing. It was also mentioned by one participant that notifications accumulated quickly, indicating the need for a more manageable notification system. Almost all participants suggested automating the filling of normative data in neuropsychological assessment instruments based on patient information, rather than manual insertion using provided images. However, DISME's implementation of NexusBRaNT strictly adhered to the traditional system's requirements, which required manual insertion. Implementing the change in DISME's NexusBRaNT implementation would be straightforward.
Concerning the positive aspects of the platform, participants found the platform highly engaging. One participant even showed interest in using it for personal projects. They praised the intuitive and visually appealing design, describing the forms as simple and easy to navigate. The choice of colours received widespread appreciation, with one participant stating they wouldn't change them and another comparing them favourably to the traditionally implemented Nexus-BRaNT with a dark colour scheme. Participants liked DISME's colour range and the ability to easily customize the Dashboard's task colours, which is done in a straightforward manner in the customization component and is directly applied to the Dashboard. The platform's intuitiveness and responsiveness were commended for making tasks manageable and enjoyable. One participant expressed satisfaction with the well-organized task sections, facilitating navigation and task completion. Regarding the graphical representation of pending and executed tasks, one participant mentioned its usefulness for colour-blind individuals, which was their case (had difficulty seeing green and red), highlighting the description of task names when hovering over the tasks' process-type coded colours.
These findings from the Think Aloud sessions provide valuable insights for improving the usability and user experience of the DISME platform. The improvement suggestions and negative feedback highlight specific areas where enhancements can be made, such as task organization, filtering options, automatic data filling, and the notification system. The positive feedback reinforces the platform's intuitiveness, user-friendly interface, visually pleasing design, and overall ease of use, reinforcing its potential as a valuable tool in the low-code development field. Incorporating these suggestions and addressing the identified issues will contribute to the ongoing development and optimization of the DISME platform, ensuring, in the future, an even better user-friendly experience, and increasing the already very high score obtained in the System Usability Scale presented in the next section.
The quantitative analysis of DISME was conducted using the System Usability Scale, which consists of ten statements that participants rated on a scale from 1 (strongly disagree) to 5 (strongly agree). The collected responses were then subjected to thorough analysis, enabling a quantitative evaluation of participants' perceptions regarding the usability of the DISME platform.
The first statement of the SUS used to assess the usability of DISME focused on participants' likelihood to frequently use the system. The responses from the ten participants involved in the study showed a high level of positive feedback and had an average response of 4.5, signifying a strong agreement with the statement.
The analysis of the fourth question from the SUS sheds light on participants' perceptions regarding the need for technical support when using DISME, with an average answer of 1.8, indicating moderately low agreement with the statement. While most participants felt comfortable using the platform independently, some expressed varying degrees of agreement regarding the need for assistance, suggesting challenges or perceived complexities. Clear instructions and user-friendly interfaces are crucial to minimizing the perceived need for external support. Enhancements aimed at improving user self-sufficiency and reducing potential barriers to independent usage could further enhance the platform's overall usability.
To calculate the overall usability score, the ratings were converted into a numerical scale using the SUS equation that can be seen in Equation 1. The obtained average score for DISME was 89.25%. The maximum score obtained was 100%, while the minimum score was 80%, as can be seen in Figure 5, which indicates a favourable level of usability and user satisfaction. This score exceeded the threshold of 68%, which is typically considered above average [21]. Specific aspects of usability, such as ease of use, system complexity, and confidence in using the platform, received positive ratings from the participants. SUS = ((Q1-1) + (5-Q2) + (Q3-1) + (5-Q4) + (Q5-1) + (5-Q6) + (Q7-1) + (5-Q8) + (Q9-1) + (5-Q10)) x 2,5
Comparing the usability scores of DISME implementing NexusBRaNT as a low-code platform with the traditionally developed NexusBRaNT system reveals intriguing findings. DISME obtained an average usability score of 89.25%, demonstrating a high level of usability and user satisfaction. The use of low-code development techniques likely contributed to its favourable score by providing a more intuitive and user-friendly experience, allowing for efficient navigation and task completion. In contrast, the traditionally developed NexusBRaNT system achieved an impressive average usability score of 92%, ranging between 75% and 100%, as can be seen in Figure 6, with a sample of health professionals in the field of psychology with domain knowledge of the modelled processes and implemented system (N = 16, fifteen females and one male, Mdn age = 27.5, age range = 21 -48 years) [22]. The applied method was identical to the one applied to the tests in the low-code platform, differentiating in the task of creating clinical information that did not exist in this case. All participants also had a Human and Social Sciences background. Namely, among the participants, six are currently pursuing a Master's Degree in Psychology, and the remaining ten are health professionals in the field of psychology. Notably, two of these health Figure 6: System Usability Scores of the traditionally developed NexusBRaNT system. [22] professionals are directly associated with BRaNT [22]. The obtained scores indicate a generally high level of usability, although there may be some variability in the user experience. The traditional development approach signifies a robust and well-designed system that meets the needs and expectations of its users.
The comparison suggests that both the low-code DISME platform and the traditionally developed NexusBRaNT system offer commendable levels of usability. While the DISME platform benefits from its low-code approach in terms of user-friendliness, the traditionally developed NexusBRaNT system demonstrates its effectiveness in delivering a reliable and highly usable system. These findings highlight the potential benefits of low-code platforms in terms of ease of use and efficiency, while also acknowledging the strengths of traditional development methods in achieving high usability standards.
In this paper, we have presented an evaluation of our research efforts in developing a DEMObased Low-Code Platform, DISME, for rapid implementation of information systems. Specifically, we evaluated the usability of an Information System for Cognitive Rehabilitation, NexusBRaNT, implemented on DISME.
The usability of DISME was evaluated using both qualitative and quantitative methods. The qualitative evaluation through the Think Aloud method provided valuable feedback on various aspects of the platform, including task organization, interface design, and user experience. Participants found the platform engaging, intuitive, visually appealing, and user-friendly. Refinements were suggested, such as task grouping, improved filtering options, and automation of data filling. The positive feedback reinforced the platform's strengths and highlighted its potential as a valuable tool in low-code development. The quantitative evaluation using the System Usability Scale (SUS) confirmed these findings. Participants agreed that the platform was user-friendly and supported their professional activities effectively, with the overall usability score for DISME being calculated to be 89.25%.
As the main limitation, we identify the small sample size. Therefore, results must be interpreted with caution and generalizability of the findings may be limited.
In conclusion, our research has contributed to improving usability of low-code platforms by evaluating the usability of an information system for cognitive rehabilitation implemented on a DEMO-based low-code platform. The results demonstrate the potential of DISME to allow impressive gains of effectiveness in software development, while achieving high levels of usability.
As future work we foresee additional evaluations in the context of new implementations in other scenarios, continuous refinement of DISME, including support for collaborative modelling and development. With ongoing efforts, we envision DISME becoming an invaluable tool for rapidly developing effective and user-friendly information systems in various domains.