Large Language Models (LLMs) recently demonstrated capabilities for generating source code in common programming languages. Additionally, commercial products such as ChatGPT 4 started to provide code interpreters, allowing for the automatic execution of generated code fragments, instant feedback, and the possibility to develop and refine in a conversational fashion. With an exploratory research approach, this paper applies code generation and interpretation to conceptual models. The concept and prototype of a conceptual model interpreter is explored, capable of rendering visual models generated in textual syntax by state-of-the-art LLMs such as Llama 2 and ChatGPT 4. In particular, these LLMs can generate textual syntax for the PlantUML and Graphviz modeling software that is automatically rendered within a conversational user interface. The first result is an architecture describing the components necessary to interact with interpreters and LLMs through APIs or locally, providing support for many commercial and open source LLMs and interpreters. Secondly, experimental results for models generated with ChatGPT 4 and Llama 2 are discussed in two cases covering UML and, on an instance level, graphs created from custom data. The results indicate the possibility of modeling iteratively in a conversational fashion.
CEUR ceur-ws.org
In recent years, Large Language Models (LLMs) have seen broad adoption due to the wide
variety of successful applications utilizing transformers [
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
execution of generated source code with a code interpreter [
In this paper, the conversational approach with instant feedback and step-wise refinement is applied to conceptual models. For exploring the potential of this approach, the research objectives are (1.) to determine whether the approach can be realized with state-of-the-art commercial or open source LLMs and interpreters, and (2.) how a possible realization could be constructed in terms of an architecture. By exploratory research, these objectives are investigated through the construction of a prototype chat application.
Potentially, conceptual models may be created within a dialogue where a modeling task is
specified fully or partially in natural language, textual syntax for models is then generated by
the LLM and automatically rendered visually by interpreters, e.g., for PlantUML or Graphviz
syntax. In addition, the interpreter is a first step towards specifying executable models for
generating software through LLMs that can be executed directly in local environments or at the
server side, e.g., generating step function models for cloud platforms and blockchains [
Prompt
Specify Modeling Task
LLM Generate
Model Sytanx
Interpreter Render Model
Syntax
Response Evaluate
Model automated iteration with refinement
Modeling with instant feedback allows for an iterative process with refinement as described in Figure 1. Such a process could be applied generally in many areas, e.g., in requirements and software engineering, where the generation of structure or behavior diagrams is above the level of source code. Further, it lowers the barrier to entry for modeling and design activities. While LLMs also lower the barrier to generating source code directly, the dificulty oftentimes lies in its evaluation; that is, to evaluate whether source code and the implied software design are suitable and fulfill the requirements.
The remainder of this paper is structured as follows. Section 2 introduces background on LLMs and related work on applying LLMs for conceptual modeling. In Section 3, the concept of a model interpreter and a possible realization architecture are outlined. Section 4 discusses experimental results of the prototype with prompts and generated models for ChatGPT 4 and Llama 2. Section 5 outlines the overall results and concludes.
Generative Pre-trained Transformer models (GPT) are a class of Large Language Models (LLMs) that apply a specific architecture on the principle of predicting or completing text, progressing on a sequence of tokens.
An input sequence is given by a prompt and the context of a dialogue preceding it, where
words are split into tokens, encoded, and represented by a vector according to an embedding in
a high-dimensional vector space in the thousands of dimensions [
LLMs notably difer in terms of their supported context size, allowing for 4096 tokens in
Llama 2 [
While prior work on LLMs and modeling applications is scarce, first publications do exist for conceptual modeling, business process management, and software modeling, in addition to related findings for programming languages.
For conceptual modeling, the application of ChatGPT has been investigated before in a
publication by Fill et al. [
examples. This paper is inspired by this investigation and seeks to integrate multiple LLMs and interpreters in a more generalized way to explore their modeling capabilities towards further systematic evaluations in the future.
For Business Process Management (BPM), Vidgof et al. [
For software modeling, Cámara et al. [
Further limitations regarding syntax and semantics aspects were investigated for
programming languages in a systematic study by Ma et al. [
This section outlines the overall concept together with requirements, leading to the discussion of the architecture for describing the system structure.
The concept of a model interpreter is first explored from a requirements perspective. This discussion aims at introducing the overall concept and stating requirements that need to be fulfilled in order to construct a corresponding architecture. 1. Conversational user interface. The application requires a conversational user interface, where a continuously ongoing dialogue is presented between the user, a LLM, and an interpreter. A user enters a textual prompt, describing a modeling task, that is appended to the dialogue and sent to a LLM. The generated LLM response in text form, potentially containing a concrete syntax of a modeling language, is appended to the dialogue. In case the response contains the concrete syntax of a known modeling language, it is sent to an interpreter. The execution result of the interpreter is appended to the dialogue in text form and, if a visual model could be rendered, in an image format. 2. LLM inference. For running LLM inference, the selection of a suitable LLM, parametrization, and the local or remote execution of the inference are required.
2.1 LLM Selection. For supporting multiple open source and commercial LLMs, possibly
difering in their capabilities of generating concrete modeling language syntax, a
LLM needs to be selected for inference.
2.2 LLM Parametrization. Depending on the selected LLM, setting hyperparameters
might be required for inference, e.g., to influence the predictability and stability
of responses for given prompts. Typically, open source LLMs provide fine-grained
control over the sampling in the sequence of tokens, often depending on temperature,
with lower values decreasing randomness when choosing the next token randomly
from the candidates, top_k, limiting the random choice of a token to the most
probable candidates, and top_p, limiting a token to be chosen from the most probable
candidates as far as they are, in sum, within probability [
3.1 Interpreter Selection. To support the concrete syntax of multiple modeling languages
in a textual format, a suitable interpreter needs to be selected. E.g., an interpreter
for PlantUML6 such as Plantweb7.
3.2 Interpreter Parametrization. Depending on the interpreter, parameters might be
required for determining the rendering layout or the output format. While a textual
syntax is assumed for the input, the output might result in vector graphics such as
SVG, raster graphics, or textual drawings [
Based on the concept and requirements, Figure 2 shows components realizing a possible architecture of a client application. In the Conversation subsystem, the main control flows are initiated through the Conversational User Interface. According to the sequences in Figure 2, the user engages in the preparation of a LLM (1.1-1.5), the preparation of an interpreter (2.1-2.4), the interaction with the LLM (3.1-3.6), and the interaction with the interpreter (4.1-4.5).
<<Subsystem>> Client Application <<Subsystem>> Conversation
Conversational User Interface Conversation
Manager 211...121::: ssseeellteeLccLtt MiLnLtpMearr↓parmeteetre↓rs ↓ 33..61:: aepnpteernpdrLoLmMptre↓sponse ↑ 2.2: set interpreter parameters ↓ 44..15:: apparpseenLdLMintreersppreotnesreo↓utput ↑ <<Subsystem>> Data Store
Data Store 1.4: create conversion↑ 1.5: set LLM configuration↑ 2.4: set interpreter configuration↑ 3.2: insert prompt↑ 43..45:: iinnsseerrtt iLnLtMerrpersepteornoseu↑tput↑ 1.3a: initialize LLM↓ 33..34aa:: rreecqeuievsetrreusnpopnrosem↑pt↓ <<Subsystem>> LLM Local 1.3b: load LLM files ↓ 33..43bb:: rreutnuprnrormesppto↓nse↑ LLM API Client
LLM Runtime 2.3: initialize interpreter↓ 4.2: run concrete syntax input↓ 4.3: return interpreter output↑ <<Subsystem>>
Interpreter Interpreter Runtime
After selecting and setting up the LLM (1.1-1.2), the Conversation Manager prepares the initiation of the conversation. Depending on the selection, either a server-side LLM is initialized through the LLM API Client (1.3a), or client-side LLM files are loaded (1.3b) with the LLM Runtime. Following the preparation of the LLM, the conversation and its configuration are recorded in the Data Store (1.4-1.5). In a similar way, an interpreter is selected and set up (2.1-2.2) with the Conversation Manager. The interpreter is initialized by the Interpreter Runtime (2.3), assumed running locally, and with its configuration recorded in the Data Store (2.4).
The user starts interacting with a LLM by entering a prompt that is sent to the Conversation Manager (3.1) and inserted in the Data Store (3.2). Either, a server-side LLM is requested to run the prompt with the LLM API Client (3.3a) and returns the received response (3.4a). Alternatively, a client-side LLM runs the prompt directly (3.3b) and returns the response (3.4b). The response is inserted in the Data Store (3.5) and appended in the Conversational User Interface (3.6).
For interacting with the interpreter, an appended response is parsed by the Conversation Manager to detect a supported modeling language syntax (4.1). The selected interpreter is executed by running it in the Interpreter Runtime (4.2) with the detected syntax as input. The output is returned to the Conversation Manager (4.3), inserted in the Data Store (4.4), and appended to the dialogue in the Conversation Manager (4.5).
This section discusses first experimental results of executing a prototype implemented for ChatGPT 4 and Llama 2 with two test cases covering the generation and interpretation of models in PlantUML and Graphviz syntax. Figure 3 shows the beginning of a dialogue in the user interface of the application, implemented as a feasibility demonstration in Python 3.118.
At the beginning of the dialogue in Figure 3, the user-provided prompt describes the test case for generating a UML class diagram according to a scenario. Without an explicit specification of the diagram, it is the task of the LLM to create classes and relationships such as orders, order items, and customers. The aim is to observe how the case is conceptualized by GPT-4 and Llama 2.
At first, GPT-4 is run using the settings visible on the left-hand side in Figure 3, resulting in the diagram in Figure 4. GPT-4 created syntactically correct PlantUML code and recognized classes, attributes, and relationships with multiplicities. All classes and relationships seem appropriate for the scenario, including the classes OrderItem, PaymentMethod, BillingAddress, and ShippingAddress, for which no specific properties or attributes were described. Possibly, BillingAddress and ShippingAddress could be further generalized. Attributes use list notation, recognizing the multiplicities correctly, and specify visibility, which is always private except for OrderHeader and OrderItem[] in the Order class for no apparent reason. While the spelling is generally appropriate for software classes, the capitalization of attributes referencing classes seems slightly out of place.
Figure 4 shows the continuation of the dialogue on the right-hand side, requesting to extend the diagram with articles and further descriptions of order items and addresses. After regenerating the PlantUML syntax and rendering, the class diagram shown in Figure 5 is the final result of the test case. A new class Article with attributes and further attributes in existing classes were created according to the description. Furthermore, GPT-4 introduced a class Address with corresponding attributes and referenced it in the classes BillingAddress and ShippingAddress by attributes. In this way, the duplication of the address attribute was prevented without a generalization, using composition rather than inheritance. Variability in the responses could be observed, e.g. distinguishing between the two types of addresses in diferent ways and using compositions instead of associations. In a few cases, additional data types and simple operations such as getter methods were additionally created.
For Llama 2, the test case is applied as before with custom settings (Appendix, Figure A.1). Also for this LLM, the result is syntactically correct PlantUML code for a class diagram, however, only classes were generated without any relationships. When asked to add relationships or associations, the LLM either did not add them or added incorrect syntax to the code.
Figure 6 shows the classes with attributes, including data types. Classes were generated similarly to GPT-4, except for the Address class which represents both billing and shipping addresses with corresponding attributes in OrderHeader and Customer. While there is no distinction on a type level and no generalization, this design also prevents the duplication of attributes. The attributes created by GPT-4 were also generated by Llama 2, representing the description correctly. However, Llama 2 also engaged in hallucination and added attributes to the OrderItem, PaymentMethod, and Address classes where the description did not indicate any specific details. Data types were correctly recognized for attributes referencing other classes and denote list types for referencing multiple classes when required by the scenario.
The dialogue is continued as before (Figure 4), still not producing relationships (Appendix, Figure A.2). An article class was added correctly, including attributes. The scenario introduced further details for addresses and order items, requiring new attributes. These were added only, it seems, if they did not conflict with existing ones. E.g., no address line and postal code attributes were added to Address which already had street and zip attributes due to prior hallucination. Variability could be observed in slight changes to the chosen attributes and data types.
In the Graphviz test case, a description for generating a directed graph from user-provided data is given to GPT-4 and Llama 2 as described in Figure 7. The aim is to observe how the LLMs construct graphs on an instance level from custom data with syntax and formatting instructions.
GPT-4 generated syntactically correct Graphviz code and created the graph as specified by the data. For the nodes, further formatting instructions for the shapes and a custom numbering format for labels were correctly implemented. As requested, edges were correctly visualized by their weight using the width; however, the weight attribute should have been specified as well. For Llama 2, Figure 8 shows the result with the settings applied before (Appendix, Figure A.1). The Graphviz code was also generated correctly, in terms of the syntax and the graph specified by the data. Compared to GPT-4, node names and labels are swapped since the specified numbering format was applied to the node names instead of the labels. Further, the shapes are missing the style attribute. Regarding the edges, the weight attributes were correctly set without the requested visualization of the width. Unnecessary edge labels denoting the graph data were additionally generated. Generally, the data and its syntax were recognized correctly by GPT-4 as well as Llama 2 with relatively low variability.
The syntax for PlantUML and Graphviz was generally created correctly by GPT-4 and Llama 2. GPT-4 utilized the syntax to a greater extent and produced more comprehensive solutions with greater complexity, also considering details recognized from the case descriptions. For Llama 2, it remains unclear whether missing elements were unknown in their syntax or not recognized, especially in the PlantUML test case where relationships had to be inferred and were missing. In this test case, the LLM was also prone to hallucination. When given a syntax description and custom data in the Graphviz test case, both Llama 2 and GPT-4 were able to recognize it and rendered graphs on an instance level. Due to the chosen phrasing and parameters, variability was generally low, however, changes as described in the results could still be observed. As expected, parameters of the sampling influenced variability, i.e., temperature, top_p, and top_k. Concerning the prototype, limitations exist in its current implementation stage, not realizing the architecture in full regarding compatibility with APIs and local LLMs beyond GPT-4 and Llama 2.
In this paper, the application of a conceptual model interpreter for LLMs was explored through the creation of an architecture and prototype using GPT-4 and Llama 2 for generating and rendering models in UML and Graphviz syntax. Results encompass (1.) the components of the architecture compatible with state-of-the-art LLMs and interpreters, and (2.) initial experimental results, demonstrating the creation of models in correct syntax for GPT-4 and Llama 2 with major advantages in correctness, recognized details, and comprehensiveness for GPT-4. Especially for GPT-4, the results show that modeling iteratively in a conversational dialogue could be practical, however, further systematic evaluations need to be conducted. Future research will apply the interpreter for these evaluations with additional commercial and open source LLMs.
This work is partially supported by the Swiss National Science Foundation project DomainSpecific Conceptual Modeling for Distributed Ledger Technologies [196889].