Despite the rapid development of iStar modeling tools, drafting iStar models on paper is convenient for illustration purposes and can be inspiring during brainstorming sessions. Digitizing such hand drafts and generating corresponding model files can enable modelers to save and reuse model snippets. This paper presents a preliminary technical framework for achieving this goal. Adopting the divide and conquer strategy, the proposed framework consists of three separate recognition modules dealing with node recognition, text recognition, and element composition. Upon implementing the framework, the recognized model drafts will be transformed into iStarML files to be easily imported to iStar modeling tools. This paper focuses on discussing the feasibility of the proposed approach and presents some preliminary.
Despite the rapid development of iStar modeling tools [
One existing relevant method uses a support vector machine (SVM) to recognize
nodes [
This paper presents a preliminary framework for automatically recognizing iStar hand drafts and generating corresponding iStar model files while preserving the position * Corresponding author Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). information of all drafted elements. In particular, the paper reports on our ongoing research on the text recognition and element composition modules with an emphasis on discussing the feasibility, advantages, and disadvantages of our proposal. 2
The proposed framework for transforming iStar hand drafts to iStarML files
comprises six modules: preprocessing, node recognition, edge recognition, text
recognition, element composition, and outputting iStarML files. Some parts of the
framework are simplified based on some features of the iStar 2.0 model such as the
arrangement in position of parent and child nodes. The framework takes original
images as input and outputs corresponding iStarML files. Fig. 1 illustrates the
modules and data flows within the framework.
The preprocessing module takes an original image as input and outputs the
corresponding binarized and grayscale processed images. This module first converts
three-channel RGB images to one-channel original grayscale images. After that,
image denoising algorithms such as NL-Means Algorithm [
The node recognition module of the proposed framework takes the denoised grayscale image output by the preprocessing module as input and outputs node information comprising the category and position of the node. Fig. 2 shows the preliminary node recognition result generated by the YOLOv3-tiny network. While the preliminary result was obtained using a software draft rather than a hand draft, the network is expected to perform well in recognizing hand drafts after collecting enough handdraft samples for training the network. In the future implementation, elements containing the information of edges, such as whether the arrow is present at the tail of an edge, will be added to the node recognition module. While the module will not recognize edges as objects, it will indicate their types. 2.3
While text recognition is a well-developed area [
In the preliminary framework, all edges in an image are assumed to be solid lines. The basic idea of solving the edge recognition problem is that one node is associated with another node if the head and tail of an edge are close enough to these nodes separately. To quantify the notion of “close enough,” a concept called buffer area is introduced. The buffer area is a rectangular area around a node, which indicates the potential area where an edge starts or ends. Fig. 3 shows the buffer area around a node. The edge recognition module would generate edge recognition result comprising a pair of nodes associated together as output.
If the head and tail of an edge fall into the buffer area of two nodes separately, then these two nodes would be associated together. After detecting all nodes in an image, the edge recognition module would retrieve the position of all the nodes from node information generated by the node recognition module. The edge recognition module would iterate through all the nodes sequentially. For each node, all its pixels would be scanned through. If any pixel is 1, it would be fed as the start of the BFS process. The BFS method would search through all the adjacent black pixels set to 1. If any black pixel falls into any buffer area, then the node that buffer area belongs to would be associated with the node containing the pixel used as the start of the BFS process. Fig. 3. Buffer Area around a Node Fig. 4. Process of Associating Two Nodes illustrates the process of associating two nodes. Considering that pictures of hand drafts can be taken from different distances and angles, the buffer area may appear on a different scale, leading to different results from the same image. To solve this problem, the ratio of the node scale is set to be the criterion for determining the scale of the buffer area.
During the BFS process, if an edge overlaps with the area of any node containing the element representing edge types, then the framework would mark that edge as being of the found type. Considering the rules of refinement, edges pointing to one specific parent node should be of the same type. While the edge recognition module may fail to recognize and mark some edges correctly, the framework would still be able to mark the remaining edges pointing to a specific parent node correctly, as long as the type of any one edge pointing to that parent node is determined, which makes the proposed framework robust. To recognize the type of each edge, it is necessary to identify the parent node of a pair of nodes. To simplify this task, a parent node can be considered as a node above a child node. The framework exploits the positions of nodes output by the node recognition module to determine whether a node is s parent or child node in a pair of nodes. 2.5
The element composition module would take the edge recognition result, node information, and text recognition result as input, and output the structural representation of the identified elements. Due to the modular architecture on the proposed framework, the results necessary for generating a final iStarML file do not share a uniform representation. In the element composition module, the edge recognition result, node information, and text recognition result would be merged to a uniform and structural representation of all elements representing the entire iStar model structure in the original image. The output iStarML file module would take the structural representation of all elements as input and output an iStarML file, which could be easily imported to iStar modeling tools. 3
There are several related studies aiming at solving the problem of transforming iStar
hand drafts into machine-readable model files, e.g., iStarML files. For example,
Gilbert and Rusli [
Shi and Luo [
This paper presented a preliminary framework for recognizing iStar hand drafts and transforming them to iStarML files. The proposed framework consists of six modules, namely, preprocessing, node recognition, edge recognition, text recognition, element composition, and outputting iStarML files. We, in this paper, focus on discussing the feasibility, advantages, and disadvantages of the methods employed in the proposed framework, shedding light on plausible improvements in the future. As for our next step work, for one thing, we plan to develop a prototype tool that implements our proposed approach; for another thing, we aim to conduct case studies with iStar modelers and iteratively improve our approach based on their feedback.
This work is partially supported by the National Natural Science of Foundation of China (No.61902010), Beijing Excellent Talent Funding-Youth Project (No.2018000020124G039), and Engineering Research Center of Intelligent Perception and Autonomous Control, Ministry of Education.