-Context: Current modelling technologies, with the support of modelling frameworks, are in the base of the current adoption of Model-Driven Software development - MDD - and supporting Software Engineering phases. Problem: The focus of these tools are solely on graphical support and visual models. In fact, the chosen modelling language's concrete syntax is either graphical or textual or both. This approach is discarding the use of other senses for modelling purposes, and, for instance, the possibility of blind software engineers to take advantage of modelling and deal with the abstractions captured by those. It is necessary to improve the productivity of people with limitations or disabilities while modelling.They should not be excluded from the modelling activity. This situation of accessibility barriers starts already at education of Modelling. Method: In this paper we present a prototype of a tool that aims to take advantage of current voice recognition and speech synthesis to edit models in diverse modelling languages. The elegance of this work is the fact that, not only it is meant to make MDD accessible to a broader spectrum of practitioners, but also it is developed with an MDD approach. Results: A prototype was built, named ModelByVoice. This tool is not bound to a particular modelling language, as long as it is meta-modelled. ModelByVoice is the base for a new tool that will enable MDD highlighting the relevant human factor of accessibility via voice and audio to models. Ultimately, it aims at bringing accessibility for blind people to deal with MDD and Domain Specific (Modelling) Languages - DS(M)Ls - the same way it is already done with diagrammatic languages with the current Modelling workbenches. Index Terms-Model-Driven Software Development, Modelling Workbenches, Accessibility, Speech Generation and Synthesis, Audio Models
A research study carried out by IBM in 2004 [
According to Stack Overflows survey in 2018 [
The diagrammatic languages modelling workbenches such
as Eclipse GMF / EMF [
In this paper, we will present our tool prototype, named ModelByVoice, that allows the visually impaired people to model diagrams or systems with any modelling language (nonhardwired to a specific language), through their voice thanks to a speech recognition system implemented on the tool. The tool was created under the assumption that the supported domainspecific languages (DSL) are meta-modelled, as the current focus of the approach is on the language’s abstract syntax instead of concrete syntax.
The rest of this paper is organised as follows. In section II we discuss the current efforts that to the best of our knowledge contribute to tackling the problem towards supporting modelling in software engineering for blind people. In section III we present an overview of our prototype. In section IV we discuss a preliminary assessment of our prototype tool. Finally, in section V we conclude and discuss future challenges to address.
The current state-of-the-art does not provide tools that
make it possible for the visually impaired to model diagrams
satisfactorily. Some tools were developed to circumvent this
problem, but in the most of cases, the success rate was reduced.
As an example, we have the Technical Diagram Understanding
for the Blind, as known as TeDUB [
Another approach to this problem, named PRISCA [
According to the documentary research carried out, the
tool that most closely resembles the features and resources of
the ModelByVoice is the VoiceToModel tool [
At the architecture level, the ModelByVoice platform is implemented in three layers. The first layer is related to the voice recognition process. The platform is continuously capturing the sound generated by the surrounding environment. When a speech utterance is detected, the sound is converted to text, and the result is delivered to the second layer. The second layer then tries to match the received user command to an operation defined by the modelling language. If a valid operation is detected, this layer is responsible for the execution of the operation. The feedback mechanism is implemented in the third layer, and the platform uses speech synthesis to communicate the input commands results back to the users. In situations where an invalid command is detected, the system alerts the user, and it prompts the user to a new attempt.
To create the necessary basis to start the developing process
of the domain specific language, we used the Epsilon [
We used the Epsilon Transformation Language (ETL)
[
As said before, the ModelByVoice platform is composed
of speech recognition and synthesis system. The technological
tools used for the speech recognition system were the Sphinx-4
[
The FreeTTS tool [
An intended key aspect of this platform is that it gives the possibility to the user to model with any modelling language as long as its metamodel exists. The platform, through the reading of the meta-model of that language, allows making the association of the elements of this language, for the standard notions under described.
At the operation level, it is possible for the user to model diagrams based on the notions of node, link, compartment and attribute. A node represents a particular entity or element, a link allows to make the connection between nodes and / or compartments, a compartment represents a hierarchical set of nodes, that is, it is a node composed of other nodes, and, finally, an attribute can be associated with one or more nodes.
In the figure 2 we have represented the standard language metamodel, which covers most elements of the languages, by association, and that for most input languages, it will be the output metamodel after the conversion of the respective metamodel original input. As explained before, we have to convert the metamodel of the language that we want to model, by associating the composing elements to the notions represented on the following metamodel, which is in the most of cases, the final metamodel.
The ModelByVoice functionality is based on operations that are performed from associated voice commands. It is required that the edited languages support voice commands for CRUD (create, read, update, delete) operations on the diagram and its elements, giving the user the ability to create, listen (”read”), edit and remove the variables that compose the elements of each model. In the following figure 3, we have the supported commands of the platform.
The primary challenge found in the implementation process was the way in which the user would navigate the diagrams and perform the desired operations. The notion of ”navigation element” was used to overcome this problem. This element, which may be a node or a compartment (which is also a node), acts as a reference and guidance point for the user to explore the diagrams. The election of the navigation element is based on three possibilities: i) when a new node/compartment is created, it is assigned the role of navigation element; ii) the user by enunciating the command ”change navigation element” can designate which will be the new current navigation element through the enunciation of its name by a voice command; iii) when the navigation element is deleted, and the user loads a previously created diagram for edition, the oldest element in that diagram is called the navigation element. Therefore, this was the idealised and chosen form, to avoid that the user gets lost during the creation, exploration, and edition of the diagrams.
The diagrams are presented to the user trough the list diagram command. This command allows to list all the diagram content trough voice synthesis, or part of it, where the user have to state the range number of elements to list by the platform.
Another essential implemented function was the help mode command. This mode allows users to be guided during navigation, calculating the possible commands that the user can enunciate from the state in which the diagram is.
Each time the user announces a voice command, the operation associated with this command will be executed by the platform, and then the voice synthesis will issue a response, so that, the user will know about the success of the operation execution. The provision of feedback during the execution of the platform allows the user not to become lost during the execution of the operations.
Once the user executes all the desired operations, there is the necessity to call the save diagram command. This command will save the diagram in XMI (XML Metadata Interchange) finally from the green to the red state. After that, the subjects had to list the diagram to verify that all elements were created correctly. If all parts were created with success, they would be asked to create an attribute with the name seconds, with integer type, to represent the time that each of the semaphores would be active. In the figure 8 we have a diagrammatic representation of the process for the first task performed by the subjects. format, in the directory of the machine where the platform is located.
In the figure 4 we have represented through an activity diagram, the possible executions of the ModelByVoice and its interaction with the user. All the possible operations are represented in this activity diagram, as well as all the options that the user can take during the process of the platform.
In turn, in the figure 5 we have the internal representation of the help mode available to the user. This helpful resource is intended to calculate the possible operations to perform from a given element, and to inform the user about which commands can be executed. Figures 6 and 7 are session logs to exemplify the models creation and edition by the operator while using our tool prototype.
It was decided to perform this first task, with the purpose of the users to get to know the platform, its operation, and the commands that they can execute. The first blind subject took about 7 minutes to complete the task, the second blind subject took around 10 minutes, and the rest of the non-blind subjects, took on average, 5 minutes to complete the proposed task. A possible explanation for the deviation in the time observed in the first group is that the first blind subject took less than the second because he had academic qualifications and was more familiar with the modelling activity. Concerning the explanation for the difference between groups, the nonblind users took less time than the blind, can be explained by the fact that they have more practice and freshness in diagram modelling. Once this task was performed, it was given the opportunity to model with a modelling language of their choice, which is the second and the last task that was executed.
The first blind user opted to model with a language that he learned in his computing course, the UML class diagram language. The subject explored all the possible commands of the platform, and it took him about 10 minutes to try out all the possible executable commands. In general, all users were comfortable with the tasks involved, except the second blind subject, who was not confident in the second proposed task and preferred not to execute it because he did not know which modelling language to choose for not being familiar with Software Engineering.
With the collaboration of the Portuguese Association for Visual Impairment, including blindness and amblyopia, ACAPO - Associao dos Cegos e Amblopes de Portugal, two blind users were involved in a preliminary assessment session where they had the chance to use the prototype and answer to a questionnaire.
The profile of the subjects to perform the usability tests
of the tool was divided into two categories, blind or visually
impaired users, and users without any visual limitation.
Relatively to the first profile, one subject was MSc. in Software
Engineering with previous experience with modelling using
Graphviz [
The selection process of the participants began by defining a set of prerequisites, which the users would have to satisfy. The requirements outlined were as follows: the user must present a basic knowledge of the English language; there must be total After the experiment, all users answered a questionnaire. unfamiliarity about the platform; and finally, the need for basic The questionnaires assessed whether if they liked the experitheoretical-practical notions in the computing area. ment if ModelByVoice was challenging to use, if they would
Before the experiment, it was established that the first recommend the platform to other users, among other aspects. task would be to conduct interviews with the users. The One of the open box questions was about the strong and interviews objectives were the verification if the subjects weak points of the platform. All users highlighted the concept effectively fulfilled the prerequisites defined, and collect per- in the tool of the navigation element, also, the simplicity sonal information such as age, experience in the area and if of the commands (not hard to remember), the existence of they had previous experience with modelling tools that have a help mode and the quality of the feedback given by the integrated speech recognition and/or synthesising mechanisms. platform. Concerning the weak points, the majority of users Regarding usability tests, two tasks were outlined, both similar, reported some glitches on the speech recognition technology, but for two different languages. The first usability test task and in their opinion, this resource is quite annoying after some involved modelling with the state machine language (states time expended on modelling. Another weak point was the and transitions) because it is a simple language and involves fact that there is no interruption of the feedback during the few concepts. The problem proposed was to create a state execution, this is, the user must wait for the end of the response machine that represented all the possible states and transitions of the speech synthesiser, even if he has already heard the of a traffic light. First, the subjects had to produce a diagram information he would like to know. named Traffic Light. They had to create three state-type nodes, Their suggestions for implementing change, was the use of and assign the names red, yellow, and green. Subsequently, the keyboard, instead of the voice recognition as input rethey generated three transitions, one from the red state to the source, and the ability to start and pause the voice recognition yellow state, another from the yellow to the green state, and at any time with the keyboard as well.
The evaluation included only two blind people, which is not statistically relevant. The reduced numbers of participating subjects are a threat to the validity of the preliminary assessment. However, this reflects the difficulty of contacting software engineers that are severely sight impaired. Still thanks to ACAPO, we managed to contact the two users involved in the experiment. The number of modelling tasks performed in the exercises can also be a threat due to its simplicity.
However, as a preliminary assessment meant to guide us in future iterations of the tool, we opted to do so as the users have never used a voice recognition modelling application, and because one of them has not even experienced before the modelling activity. As expected, despite the reduced subject numbers, the performed tests showed promising and confident results and turned out to be valuable feedback for future improvements that are underway.
This work presents a prototype tool that allows people with visual impairment to create and edit diagrams for any type of modelling language as long as this language is meta-modelled.
The goal is to provide accessibility to the modelling activity that can be reached both for blind people (and with problems with vision) and users without physical limitations. With this tool, it is possible to edit and query(navigate) diagrams, via voice. We expect that this will be an enabler for the full integration of blind people into group projects involving modelling.
The preliminary usability tests or evaluations with blind subjects and non-blind, although with anecdotal figures, allowed us to get a first feedback on the previously mentioned characteristics, since the users approved the platform, and left good suggestions of change and confidence for a future continuous development work.
As mentioned in sectionIV, the main limitations pointed out
by the subjects in our preliminary experiment, were related to
technical failure (error rate) of the existing underneath speech
recognition technology, while running ModelByVoice. The
expert users (with extensive practice with software technology
and used to use the keyboard) have suggested using the
keyboard as an input component, as it may prove to be faster
and more efficient when executing commands, instead of the
speech recognition. It should be noted that this limitation in
speech recognition is due to the fact that it is limited to an
open source voice tool (Sphinx), and a voice tool (Google
Cloud Speech-to-Text) that, despite being more efficient than
the previous one, is also not 100 % reliable in translating the
voice to text. Another interesting observation, besides the need
for some improved interaction mechanisms (like pause), is the
request for introducing audio signals instead of pure voice
synthesis feed-back. We foresee that an interesting approach
to deciding for the adequate concrete syntax, in this case,
is to use an adaptation of the proposed design principles of
PoN (Physics of Notations) [
Taking into consideration the feedback and suggestions given by the users about the ModelByVoice, some features may prove to be interesting to implement in the future.
Among them we have to consider giving the users the possibility to choose the input resource that they intend to use to practise the modelling activity, that is, to allow them to select the voice recognition mechanism or the keyboard as an input resource, or even combine both input sources at the same time.
Another interesting upgrade in the tool’s architecture is the introduction of an intermediate speech recognition platform independent layer, that would support any speech recognition API. This way, the tool would be easily reconfigured to handle with any speech recognition tool.
Additionally, it would be interesting, and straight away, to create an editor that could convert the created diagrams (which are in XMI) and generate the diagram in graphical mode. This would allow the diagrams to be analysed by other entities or people who would eventually analyse or evaluate those diagrams.
Finally, as a more challenging future work, this project raises the issue of what should be the systematic approach to assess audio concrete syntax and interaction model regarding its usability. We argue that it should be a similar framework to PoN. In our perspective, an interesting line of research could be followed in this direction.
The authors would like to thank NOVA LINCS Research Laboratory (Grant: FCT/MCTES PEst UID/ CEC/04516/2013) and DSML4MAS Project (Grant: FCT/MCTES TUBITAK/0008/2014).
The authors would also like to thank ACAPO (Associac¸a˜o dos Cegos e Ambl´ıopes de Portugal) for providing us the contact of software professionals, and for their availability to test our tool.