The role of Generative AI in computing education is one of the mostly debated issues in the last year, as evidenced by works such as [1]. Although the advantages and disadvantages of using tools such as ChatGPT 1 to support the teaching of computer-related subjects are not fully clear yet, there is a growing consensus that the technological progress of Generative AI will require an adaptation of the educational methods currently employed.
In the context of introductory and advanced programming courses, tools such as ChatGPT, which are already integrated in the most common software development tools (take as an example GitHub's Copilot) [2] are frequently exploited. These tools are designed in order to help the users to best formulate the questions submitted to the prompt.
According to the current literature on computing education, such as [1,3,4,5,6,7], the most common use of Generative AI is to generate code or to explain specific features of existing or generated code. The use of Generative AI to generate solutions to assignments and exercises is indeed a critical issue to be taken into consideration to avoid plagiarism and negative effects on student learning outcomes.
In our research, we are interested in possible applications of Generative AI to develop a Computational Thinking approach in learning to program.
Computational Thinking [8] can be defined as "The thought processes involved in formulating problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an information-processing agent. " [9] Wing emphasized that Computational Thinking is a fundamental component of Computer Science, where its abstractions can be executed by computational agents.
The shift from solving problems to effectively solving problems and making something perform computations for us distinguishes Computational Thinking from other disciplines.
One of the characteristics that defines Computational Thinking is problem decomposition, i.e., the capability of solving a complex problem by breaking it down into progressively smaller tasks.
In this paper we focus our attention on the possible use of Generative AI to develop Computational Thinking skills by supporting the process of Problem Decomposition [10] with the help of visual tools.
In particular, we examine the potential of ChatGPT to assist students in analyzing and decomposing problems into smaller subtasks, employing a top-down approach. This is followed by the generation of code for a possible implementation, using a bottom-up approach and potentially reusing the code generated in previous steps. Our approach is based a visual tool that presents the current decomposition in form of a labeled tree, in which nodes contain names and descriptions of each subtask. Starting from the root node with the description of the problem to solve, the user chooses which nodes to further decompose. The decomposition process is designed to be interactive and fully guided by the user until the solution is decomposed in a collection of subtasks that the user is capable to implement. The decomposition of a given task, i.e., a given node of the tree, can be provided manually by expanding the corresponding subtree with a set of labeled nodes inserted by the user. In addition it can also be generated via a predefined query to ChatGPT. The query formulation is based on a preliminary prompt engineering experimental design phase that seems to provide good results in the most common coding tasks taken from the literature on introductory programming courses. In our approach, the prompt engineering task is entirely hidden to the user and ChatGPT is considered as an oracle to support students during the decomposition process.
If needed, in a second phase, users can prompt Chat-GPT to generate the code of the solution in a bottom up manner. The predefined query generated by our tool ensures that functions generated for a given subtask can be reused for generating the code of the current task.
The implementation of the tool allowed us to evaluate how well ChatGPT is able to decompose a high-level task into smaller, easier-to-solve subtasks, and to implement decomposed tasks using a bottom-up approach.
In the paper we present the methodology, the proposed tasks, and the preliminary results with the help of an example.
The paper is structured as follows: In Section 2, we outline the background of our study, focusing on the effects ChatGPT has on education and computational thinking. Section 3 describes the proposed methodology and experiment setup. Section 4 reports, focusing on a case study, the results of our first experiments and discusses their limitations. Finally, in Section 5 we address conclusions and future research directions.
ChatGPT has shown remarkable capabilities in solving programming tasks across a range of different programming languages, particularly when the instructions are presented in a clear and unambiguous manner [7,6]. However, it is less effective when confronted with more complex requests, particularly when the questions are not structured in an optimal manner or when the information provided in the prompt is limited [6].
Even though the tool shows great potential in the educational domain, it is used by a limited number of students for educational purposes [11]. One of the reasons is that education regarding LLMs usage is very limited and students often don't know how to properly use these tools to support learning [11,5]. Furthermore, a number of students may prefer not to use the tool, believing it will lead to reduced learning [11].
Among the CS students who employ the tool, the most common usage is for generating code, followed by debugging. Explaining difficult concepts falls in third place [12]. Students who use ChatGPT when facing programming assignments tend to show higher efficacy and computational thinking skills [13].
Currently, ChatGPT is mainly employed to automatically provide students with feedback when programming a task. Feedback can be provided in many forms, such as coding hints for the next instruction [4] and code or error explanations [14].
When confronted with more complex problems, Chat-GPT is less adept at providing reliable solutions. As a result, the user is often required to manually break down the problem into smaller components and subsequently assemble the solution [15].
To evaluate the support offered by ChatGPT to decompose a high-level task into smaller, easier-to-solve subtasks, we designed a set of prompts for ChatGPT, as well as a custom visualization tool to display the results and allow the user to interact with the tool.
Decomposing a task into subtasks To decompose a broad task into more refined subtasks we experimented with several prompting styles.
Our initial approach was to include the current task decomposition in the prompt, so that ChatGPT would be aware of which tasks had already been decomposed and how they were decomposed at each iteration. However, this approach proved ineffective as ChatGPT struggled to understand the decomposition. Various notations were attempted, including JSON and ad hoc syntax, similar to regular tree expressions, to describe the current structure of the decomposition, but none was found to have a positive impact. Ultimately, two distinct prompts were utilized, and ChatGPT's conversational memory was relied upon to maintain the current decomposition state.
The initial prompt is used when a user first wishes to decompose a problem. It includes a user-provided problem description and detailed instructions for ChatGPT on how to accurately decompose it. The main requirements we identified are:
• Specifying that the task needs to be decomposed into a small number of subtasks. This requirement is crucial as ChatGPT has a tendency to immediately decompose the problem into approximately ten steps, with emphasis on the process rather than on the reasoning behind the task.
• Requiring the fields "name" and "description" to be similar to the ones initially provided. This is important for keeping stylistic consistency between all tasks.
• Returning the results in JSON format. This is important for using the output in our visualization tool. We also noticed that this requirement made the title and description more precise and each subtask being assigned a unique name.
• Requiring that subtasks of a given task did not contain any element of other tasks. This is needed to avoid task mix-ups.
• Ensuring there are no missing steps in the decomposition, i.e., solving all the subtasks is equivalent to solving the original problem.
We also included an instruction indicating not to decompose a task in case no reasonable decomposition can be made (for example for basic tasks). However, this proved ineffective, as ChatGPT always ended up decomposing even the simplest tasks.
A simpler prompt has been designed for decomposing subtasks. The prompt references the task by name and requests that the same process be repeated for that task. It is important to note that each task decomposed using this prompt has been originally generated by ChatGPT.
Once the user is satisfied with the current decomposition, they can start implementing the subtasks following a bottom-up approach. The prompt requires ChatGPT to implement the given task in a specific programming language, which can be selected by the user.
We identified two requirements, which are:
• Using, whenever possible, functions already generated by ChatGPT for other tasks. This requirement is crucial, since it can highlight the interaction between a task and its subtasks.
• Not writing the implementation of functions which have already been implemented, to avoid redundant code.
The tool we have created enables users to easily visualize tasks and explore their decomposition into subtasks.
Users have the option to display or hide subtasks for a given task and, if available, review its properties and implementation. Alternatively, ChatGPT can be utilized by the tool to automatically decompose or implement any task. Task decomposition is only performed upon user request, as each user's knowledge base varies and some tasks may be clear to some users but not to others. Tasks can also be marked as "solved", indicating that the user has fully understood the task and no further decomposition is needed. For the sake of usability, solved tasks are presented in a distinct color, allowing for a clear representation of the current understanding of the problem at all times.
Tasks can also be edited, created, or deleted manually. However, ChatGPT does not currently reflect these actions. Automatic decomposition or implementation of these tasks is not currently available.
The tool was employed to decompose and implement a small number of tasks. This section presents the results for the task "Write a Python program to find the most trending videos, given a CSV file containing each visualization". The task has been intentionally formulated as a broad request, to test if the program would be able to correctly identify its subtasks.
Figure 1 shows the decomposition of the task. ChatGPT decomposed the main task in the following subtasks, which is similar to how we would have manually decomposed it.
• Read CSV file: Write a Python function to read the CSV file containing video views data.
• Parse CSV file: Write a Python function to parse the data from the CSV file and extract relevant information such as video IDs and view counts.
• Calculate trending score: Write a Python function to calculate a trending score for each video based on its view count and possibly other factors such as upload date.
• Sort videos by trending score: Write a Python function to sort the videos based on their calculated trending scores in descending order.
• Retrieve top trending videos: Write a Python function to retrieve the top N videos with the highest trending scores, where N is a parameter.
We chose not to decompose basic tasks, such as reading, parsing, sorting, or displaying data, as they can be easily accomplished using existing functions in Python. Instead, we decided to further decompose the task "Calculate trending score" because its implementation was not easily understandable. ChatGPT decomposed the task into:
• Calculate view count factor: Write a Python function to calculate a factor based on the view count of a video, possibly using logarithmic scaling to give more weight to videos with higher view counts.,
• Calculate time decay factor: Write a Python function to calculate a factor to represent the time decay of a video's popularity, considering how recently it was uploaded.
• Combine factors into trending score: Write a Python function to combine the factors calculated in the previous steps into an overall trending score for each video.
which is consistent with how we would have simplified the task. Further decompositions of its subtasks, as shown in Figure 1, were also consistent with our expectations.
After obtaining the decomposition shown in Figure 1, we instructed ChatGPT to implement the tasks using a bottom-up approach. This involved starting from the smallest subtasks and subsequently progressing towards the top-level task.
For each subtask, ChatGPT created a function which solved the given problem. Each implementation also contained a docstring describing its purpose, parameters and returned value, as well as a usage example, as shown in Figure 2.
For tasks that were previously decomposed, ChatGPT was able to accurately recall and utilize the functions it had previously implemented, as shown in Figure 2b.
The main limitation of this study is the relatively small number of problems that were decomposed and implemented using our system. It is possible that some tasks may present unforeseen challenges that require further tuning of our prompts or different approaches.
A further limitation of the study is the small number of users who have tested the system. By testing the system with a larger number of users, it would be possible to ascertain whether certain tasks present issues when decomposed. Additionally, as ChatGPT is frequently updated, different prompts may be required to obtain the same output.
The developed tool demonstrates potential for decomposing tasks into smaller subtasks and implementing them using a bottom-up approach. The preliminary study decomposed a limited number of problems, but the results seem quite promising. ChatGPT correctly identified how to decompose the main problem into smaller subtasks and provided a well-documented implementation of each step.
Future directions for our work include:
• Integrating the tool with ChatGPT's APIs to fully automate decomposition and implementation functionalities;
• Enabling users to fully modify existing decompositions and implementations;
• Testing our system with a larger number of problems;
• Defining precise evaluation criteria for proper task decomposition;
• Measuring if the tool can improve computational thinking learning skills in users.
• Using the tool to identify weaknesses and misconception of students when facing a sequence of tasks that require common solving techniques.