Design thinking has gained popularity in recent years as a toolbox to support the development of innovative business models, product oferings and services. Applying a user-centric paradigm during the co-creation phase of stakeholders with various backgrounds, where ideas are created, prototyped, and iteratively evaluated. A challenge observed in such settings relates to how knowledge about novel ideas can be communicated and developed in collaborative settings, specifically from the perspective of processing artefacts realised. We will introduce the Scene2Model tool, developed in the OMiLAB@University of Vienna as a way to tackle this challenge. Scene2Model enables the digitalisation of design thinking artefacts, developed using haptic elements as conceptual models. The tool supports a dynamic metamodel representation and builds on training results to a) recognise the haptic elements as model concepts, b) elevate the Scene2Model metamodel continuously with domain-specific vocabulary and c) align these concepts with external definitions on type level. The digital models in Scene2Model become therefore digital representations that can be annotated and refined, queried or transformed as model value capabilities. This paper provides a demonstration of this idea.
Design Thinking - as the use of designers’ problem-solving techniques applied to innovation
processes - has gained popularity in a variety of industry sectors.Held in closed workshops, it
paves the way for early exploration and validation of design alternatives regarding innovative
services, new product (and features), processes and disruptive business models by means of
collaborative agile ideation, prototyping and testing techniques [
A prerequisite to establish location and time-independent interaction during design relates to
the digitalisation of design artefacts within the iterative innovation process. Such a digitalisation
needs to go beyond digital drawing or photos of physical design results. Instead, semantically
rich representation enable interaction on a granular level. For this purpose, conceptual models
are suggested as a technique to represent the knowledge about innovative scenarios and expose
it to the community. Conceptual models require a vocabulary that is domain-specific and
well-understood by the participants. SAP Scenes [
Scene2Model [
As already outlined in [10], domain-orientation is considered a critical aspect as it "reduces the large conceptual distance between problem-domain semantics and", generalizing Fischers statement, the intended representation of the system-under-study. Co-evolution is realized as the designer have agreed upon a common vocabulary that is adequate for the problem space and have access to relevant knowledge of various stakeholders.
The drawback of this approach is that it requires manual efort to attach QR codes to each physical object and that only the QR codes are detected and not the attached figures themselves. To overcome this drawback we developed machine learning approach for image recognition. The knowledge about the visualisation of the haptic elements, in our case paper figures, is trained and represented as a convolutional neural network to support the later recognition of elements via a video stream, observing the design thinking workshop.
Additionally, the approach applies the Agile Modelling Method Engineering (AMME) approach [11], to adapt the Scene2Model domain concepts to the needs of the concepts in the workshop. The focus hereby is on the design and formalise phase, when new concepts and their graphical notation are added. While in [12] an approach is presented that allows to extend the SAP Scenes, in this work we deal with the linkage of the physical world and the digital world.
The scope of this paper is the extended Scene2Model environment, graphically shown in Figure 1 above. Participants of a workshop utilise haptic objects, typically paper figures to describe and discuss scenarios. Scene2Model supports these workshops by a) constantly capturing the design space as a video stream and b) translating the physical scene developed into a model representation. Various approaches have been evaluated to perform this task: the initial prototype operates on QR tag recognition as introduced above. Each haptic element is extended by a tag that can be detected. The position information is then retrieved and the ID within the QR code is used to query an ontological representation of the concept. Utilising the capabilities of ADOxx [13], the information is imported in the tool as individual modelling objects that form the scene discussed. The challenge in this setting is the synchronisation of the metamodel used in Scene2Model for the realisation of processing functionality with the ontological representation and QR codes available. Extension or domain-specific adaptation implies a (re-)engineering of these artefacts.
The objective of the research presented in this paper is to combine image recognition with conceptual modelling and knowledge representation. Adding a new graphical system typically consists of two steps: First a drawing of the object is made that is adequate for the specific domain. Second, the knowledge base of conceptual elements is extended to enable a digitalisation within the tool. The second step is now supported by neural networks: the image retrieved is trained beforehand and dynamically extends the metamodel. Therefore, adaptation becomes feasible and it is not required to manually engineer the knowledge base. The recognition server is responsible to provide the interface between the image providing service (e.g. camera, video stream, mobile phone) and the Scene2Model modelling tool. This interface is established via the OLIVE micro-service framework [14].
Therefore we distinguish between a training phase which is used to prepare the domainspecific workshop vocabulary and provide the network to the recognition server. Based on the training outcomes, the metamodel is extended and the graphical elements are not only available as haptic figures but also understood by the tool.
This section introduces the object recognition architecture for conceptual models, including input to the training and executing the detection. For our implementation we use the YOLOv4tiny [15], which is based on YOLOv4 [16] and supports the requirement of real-time object detection and a fast training. Both are relevant for the Scene2Model environment as results should be available directly in the workshop situation. The initial definition used was the YOLOv4-tiny definition available online at [17].
As the preparatory activity to capture the domain (conceptual analytics), the visual elements are identified, the semantics of identified elements are elevated in a machine supported manner using open accessible dictionaries and the convolutional neural network is trained for recognition. The objective of this phase is to establish a domain-specific metamodel for Scene2Model dynamically as the vocabulary to be applied. This has an impact on the Scene2Model tool as it is dynamically adapted to support the semantics of the domain and trained concepts.
Technical, this requires the creation of a labeled dataset. Example pictures are prepared (photos and labelling) and further data augmentation is applied [18] resulting in a moderate dataset that includes artificially blurred, cut-out or distorted versions of the image. The training itself is performed using darknet [19].
As an outcome, the trained network is provided on one hand to the recognition server and on the other hand acts as input for the knowledge base represented by the metamodel in Scene2Model. The Scene2Model tool dynamically adapts to these changes and includes the new elements as a domain-specific version.
After the network is trained it is used to detect objects in the image stream from the workshop. The recognition server implemented in Python provides the necessary interfaces to perform this recognition and manages the trained networks. As an interchange format a JSON representation is returned from the recognition that is embedded in the Scene2Model tool utilising the OLIVE micro-service environment [14]. The stream is transformed into the conceptual model, and since the training network is aligned with the Scene2Model metamodel model processing capabilities can be applied.
Managing multiple networks is considered an opportunity to include pre-trained information directly, without the need to extend manually and run through the training phase.
The synchronisation of the knowledge base adapts the known concepts in Scene2Model. The knowledge base, represented as a knowledge graph, is parsed and mapped toward the abstract metamodel definition. Mechanisms in ADOxx enable that the metamodel can be update during runtime. The modelling panel therefore represents the trained concepts structurally. Syntactically properties are dynamically retrieved through openly available sources such as DBPedia.
This iteration of the Scene2Model prototype demonstrates how the metamodel is dynamically updated based on training of neural networks for object detection. The approach presented builds on pre-existing approaches in object detection and related techniques for training of neural networks and applies it to design thinking and conceptual modelling.
Further work and next steps relate to the evaluation of the approach, specifically how domainspecific libraries can support various settings in workshops and capture domain-specific aspects dynamically.
These steps are currently worked on in the context of the FAIRWork EU project [20] where Scene2Model is utilised in the requirements engineering phase.
All material provided and developed is provided via the OMiLAB.org as the community hub of the Open Models Initiative. This includes software artefacts, models created, and content discussed. This is specifically relevant for the open software “Scene2Model” used for this contribution. The artefacts are openly accessible at https://www.omilab.org/design-thinking/.
The work presented in this paper received funding from the Horizon Europe programme in the context of the FAIRWork research project (“Flexibilization of complex Ecosystems using Democratic AI based Decision Support and Recommendation Systems at Work”), Grant Agreement Number 101069499. Conference of Service Science and Innovation (ICSSI 2018) and Serviceology (ICServ 2018), 2018. [5] E.-T. Miron, C. Muck, D. Karagiannis, Transforming Haptic Storyboards into Diagrammatic Models: The Scene2Model Tool, in: 52th Hawaii International Conference on System Sciences, Hawaii International Conference on System Sciences 2019, 2019, pp. 541–550. [6] D. Karagiannis, R. A. Buchmann, X. Boucher, S. Cavalieri, A. Florea, D. Kiritsis, M. Lee, OMiLAB: A Smart Innovation Environment for Digital Engineers, in: L. M. CamarinhaMatos, H. Afsarmanesh, A. Ortiz (Eds.), Boosting Collaborative Networks 4.0, Springer International Publishing, Cham, 2020, pp. 273–282. [7] D. Bork, R. Buchmann, D. Karagiannis, M. Lee, E.-T. Miron, An Open Platform for Modeling Method Conceptualization: The OMiLAB Digital Ecosystem, Communications of the Association for Information Systems 44 (2019) 673–697. [8] OMiLAB Teamx, A Digital Innovation Environment powered by Open Models Laboratory, 2020. URL: https://zenodo.org/record/3899990. [9] R. A. Buchmann, A.-M. Ghiran, V. Döller, D. Karagiannis, Conceptual Modeling Education as a "Design Problem", Complex Syst. Informatics Model. Q. 21 (2019) 21–33. [10] G. Fischer, Domain-oriented design environments, Automated Software Engineering 1 (1994) 177–203. URL: https://doi.org/10.1007/BF00872289. doi:10.1007/BF00872289. [11] D. Karagiannis, Agile Modeling Method Engineering, in: 19th Panhellenic Conference on
Informatics (PCI 2015), 2015, pp. 5–10. [12] E. Laurenzi, K. Hinkelmann, D. Montecchiari, M. Goel, Agile Visualization in Design
Thinking, Springer International Publishing, Cham, 2021, pp. 31–47. [13] ADOxx.org, ADOxx Metamodelling Platform, 2023. URL: https://www.adoxx.org. [14] ADOxx.org, OLIVE: OMiLAB Integrated Virtual Environment, 2023. URL: https://www.
adoxx.org/live/olive. [15] Z. Jiang, L. Zhao, S. Li, Y. Jia, Real-time object detection method based on improved yolov4-tiny, arXiv preprint arXiv:2011.04244 (2020). [16] A. Bochkovskiy, C.-Y. Wang, H.-Y. M. Liao, Yolov4: Optimal speed and accuracy of object detection, arXiv preprint arXiv:2004.10934 (2020). [17] A. Bochkovskiy, YOLOv4-tiny Definition, 2020. URL: https://github.com/AlexeyAB/ darknet. [18] B. Zoph, E. D. Cubuk, G. Ghiasi, T.-Y. Lin, J. Shlens, Q. V. Le, Learning data augmentation strategies for object detection, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, pp. 566– 583. [19] J. Redmon, Darknet: Open source neural networks in c, http://pjreddie.com/darknet/, 2013–2016. [20] FAIRWork Consortium, FAIRWork: Flexibilization of complex Ecosystems using Democratic AI based Decision Support and Recommendation Systems at Work, 2023. URL: https://www.fairwork-project.eu.