An Automated Evaluation Approach for Jupyter Notebook Code Cell Recommender Systems Selin Aydin1,* , Dennis Mertens1 and Ouyu Xu2 1 Research Group Software Construction RWTH Aachen University, Ahornstraße 55, 52074 Aachen, Germany 2 ETH Zurich, Zurich 8092, Switzerland Abstract The reuse of code within Jupyter Notebooks is an often overlooked concept, resulting in the loss of valuable knowledge that could be retrieved from already created Notebooks. Presently, the reuse of Jupyter Notebooks is particularly impeded by the absence of suitable reuse approaches and tools. This paper presents a code cell recommender system in the form of the JupyterLab extension, designated as JupyRecSys. Moreover, we propose an automatic evaluation framework in the form of the CL-tool CelRecEval, which allows developers to evaluate and compare the performance of cell recommender systems. The evaluation capabilities of CelRecEval are demonstrated by applying it to JupyRecSys. The resulting high metric scores demonstrate that JupyRecSys can correctly recommend and rank relevant cells. Moreover, the generation of more detailed performance reports enabled the identification of specific characteristics of code cells that negatively impact the performance of the cell recommendation system. Keywords Jupyter Notebook, Recommender System, Machine Learning 1. Introduction the paper. Then, in Section 4, we present a cell recommenda- tion strategy. Next, Section 5 presents a general evaluation The Jupyter Notebook (Notebook for short) has become framework for cell code recommenders. Section 6 describes a widely utilized tool in academia and industry for pro- the implementation of the recommendation strategy and totyping Python ML solutions due to their flexibility and the evaluation framework. The setup and results of the eval- interaction. Studies have demonstrated that code from other uation are discussed in Section 7; the research questions Notebooks is frequently reused through copy-paste. In par- are answered in Section 8. After presenting the threats to ticular, code for importing packages and visualizing data validity in Section 9, Section 10 concludes the paper and is often duplicated [1]. An analysis by Källen et al. of 2.7 gives an outlook for potential future work. million Notebooks on GitHub found that 70% of the code snippets contained are identical to others, differing only in whitespace, and nearly 50% of all Notebooks contain no 2. Related Work unique code at all [2]. Currently, Notebooks lack reusability concepts. This issue has been discussed in various publi- This section reviews recent tools and methods to improve cations, and initial solutions have been proposed as best the code reusability of Notebooks and their evaluation ap- practices or JupyterLab extensions [3][4]. proaches. An initial challenge is to identify and locate relevant code JupySim, developed by Horiuchi et al., models Notebooks from previous Notebooks. While search systems can be as directed acyclic graphs to identify the relational struc- used for this purpose, they require manual intervention by tures between code, data, and outputs [5]. The system has the developer. The success of the search depends on the proven effective in identifying the most similar Notebooks keywords used. An alternative approach would be to use based on user queries. However, its complex, detailed graph- a recommender system that suggests relevant cells from based query construction may impede user adoption for previous Notebooks to match the current cell. This would regular Notebook reuse. Additionally, JupySim is currently significantly reduce the effort required, as no action by the only available as a separate web interface, suggesting the developer would be necessary. potential for a more integrated JupyterLab solution. Unfor- A Notebook cell recommendation system (cell recom- tunately, the authors do not provide an evaluation approach mender for short) implements a special form of code rec- or results. ommendation, suggesting similar code cells instead of code The Elyra code snippet JupyterLab extension represents completions. Cell recommenders have not yet been explored, a further significant addition to the Jupyter ecosystem [6]. and an evaluation method for them has not yet been devel- Users can label specific code cells within any Notebook, oped. save them to the global extension code database, and re- The paper is structured as follows: Section 2 presents an trieve them via text or label queries. Despite Elyra’s ease overview of existing tools and methods to reuse Notebooks. of integration and improved searchability, it relies heavily In Section 3, we state the research goals and contributions of on manual user intervention. The necessity for users to repeatedly identify and label reusable code snippets could QuASoQ 2024: 12th International Workshop on Quantitative Approaches potentially be a source of frustration, particularly for those to Software Quality, December 03, 2024, Chongqing China with extensive code bases. * Corresponding author. While JupySim offers a sophisticated graph-based ap- $ aydin@swc.rwth-aachen.de (S. Aydin); dennis.mertens1@rwth-aachen.de (D. Mertens); ouyuxu@ethz.ch proach to identifying similar Notebooks, its complexity and (O. Xu) standalone nature may not be optimal for everyday use. € https://github.com/d-mertens (D. Mertens) In contrast, Elyra, with its integrated JupyterLab inter-  0009-0006-1764-8091 (S. Aydin); 0009-0003-8076-0412 (D. Mertens); face, streamlines code snippet reuse but necessitates manual 0009-0006-4873-4187 (O. Xu) labeling and does not support the reuse of entire Notebooks. © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 4 A first step towards an automated solution is Typhon Embedd Cell [7], an approach for recommending Notebook code cells based on Markdown text similarity. The authors evalu- ate Typhon twofold. First, they manually request and re- Notebook Cell Tokenize view recommendations for Markdown-code-pairs contain- ing Matplotlib plot and chart code. For this, they add suitable Cell Tokens Classify Task Markdown-text to chosen code cells. The Markdown-text from these Markdown-code-pairs is then used to query rec- Generate Embedding ommendations. The authors evaluate a recommendation as correct if the recommended code is the same as in the Task Label Cell Embedding query Markdown-code pair. The authors rated the result- ing accuracy as moderate. Due to the evaluation setup, the generalizability and comparability are limited. It has to be Build Embedding Labeled Cell Embedding noted that 30,93% of public Notebooks on GitHub do not contain a single Markdown cell [3]. Further, Typhon is not yet publicly available. Figure 1: Process of embedding a Notebook cell 3. Research Goals and Contributions First, we explain how code cells are transformed into a Given the strengths and limitations of the existing ap- vector representation in a semantic vector space, resulting proaches, our current work is focused on addressing these in a cell embedding as output. Then, we present the process gaps by answering the following research questions (RQ): of transforming and loading, which stores cell embeddings in an appropriate database. Finally, we explain how recom- RQ1: What can a cell recommendation strategy look like mendations are provided using the stored cell embeddings. that makes suggestions to the developer during the programming of a cell? RQ2: How can an approach for the automatic quantitative 4.1. Embedding Cells evaluation of the performance of cell recommenders When code cells are transformed into vector representations look like? and embedded in a vector space, they are particularly close to each other if they represent similar code. To further By answering these questions, this paper makes the fol- reduce the search space around a cell embedding and get lowing contributions to improve the reusability of Note- more accurate results, each cell embedding is labeled with books: the ML task it implements. • A cell recommendation strategy specialized for The embedding process is shown in Figure 1 as a UML Python Notebooks implementing ML tasks. activity diagram. The input is a single Notebook cell. The • A general framework to quantitatively evaluate cell process consists of two parts that are executed in parallel. recommenders implementing this strategy. The specific ML task implemented in a cell is classified in • The JupyterLab extension JupyRecSys which im- the left part, returning a task label, e.g. “data preprocessing” plements the cell recommendation strategy. or “model training”. The right part of the process consists of two actions. First, the code of the cell is tokenized into • The CL-tool CelRecEval which implements this its elements (e.g., keywords, operators, identifiers), creating evaluation approach. the cell tokens. Second, in the generate embedding action, the • The results of applying CelRecEval to evaluate the syntax and the semantic relationships between the tokens performance of the cell recommender JupyRecSys. are analyzed and mapped into a vector space, returning a cell embedding. Finally, the task label and the cell embedding 4. A Cell Recommendation Strategy are composed to a labeled cell embedding. In this section, we present a recommendation strategy for 4.2. Transforming and Loading Cells cell recommenders. A cell recommender that suggests code based on similarity to a query cell is classified as a content- To recommend code from previous Notebooks, the vector based recommender. Since it only provides the top-k most representations and required metadata of code cells must be similar cells, we refer to it as a top-k cell recommender. stored in a dedicated database called CelRec-DB. A CelRec- A common approach in such recommenders is to embed DB must fulfill the following requirements: recommendation items in a semantic vector space. This has several advantages. First, it significantly reduces the com- • Storage space: the value and quality of the recommen- plexity of the items, e.g., code, syntax, semantics, variables, dations increase with the amount of available data symbols, etc. Further, a vector representation within a se- for recommendation. Thus, the CelRec-DB must mantic vector space allows the recommender to efficiently store the data in a scalable and efficient manner. determine similarities using various distance metrics, such • Data retrieval: for recommendations, it is crucial that as cosine similarity or Euclidean distance. In the vector the stored data can be accessed quickly. Otherwise, space, the proximity of vectors directly reflects the degree the value is reduced if recommendations take too of similarity between them. At the same time, the results long. remain interpretable because distances between vectors are • Data representation: the CelRec-DB has to support easy to comprehend. vectors and vector operations, ideally distance com- putations. 5 Notebook Recommend Transform & Load Notebook with Transform & k-ANN Top-k Similar Pre-Process Cells Query Cell Load Search Code Snippets <> Pre-Processed Cells CelRec-DB parallel Figure 3: Recommendation strategy for given query Notebook Embedd Cell Extract Metadata tatively evaluate the performance of cell recommenders can Labeled Cell be designed. By using this framework, we want to answer Embedding Cell Metadata the following evaluation questions (EQ): EQ1: What is the performance of the top-k cell recom- Store Cell mender? EQ2: How does its performance vary with increasing lines <> of code in a query cell? CelRec-DB The first question concerns performance in general. The Figure 2: Process of transforming and loading cells in a database second question is specific to a developer receiving recom- mendations while programming and wanting to write more lines of code in a cell. This would also show how much code is needed to obtain relevant recommendations. Figure 2 depicts the process of storing the code cells of a Notebook in a CelRec-DB. First, the Notebook is pre-processed. All code cells are 5.1. Evaluation Methodology extracted and cleaned, e.g., non-essential information like Code cell recommendation for Notebooks is a special case comments are removed. The resulting pre-processed cells are of code recommendation. Code recommenders, in general, put into an ordered list. usually suggest code for code completion. There are two Then, each pre-processed code cell undergoes two sub- main strategies for evaluating code recommenders: partial processes. Each cell is transformed into a labeled cell em- code reduction and user studies [8]. bedding by applying the embedd cell process. In parallel, In the former, code snippets are taken, and the last lines the extract metadata action returns metadata about the cell, are removed, mimicking that the developer has started typ- such as its Notebook’s name and the code it contains. Last, ing code and expects a recommendation. Then, given the the labeled cell embedding and its associated metadata are code recommendation, it is checked whether it matches stored in the CelRec-DB. the removed code. This allows for the analysis of the rec- ommender’s performance. If this is done regularly, it can 4.3. Recommending K Most Similar Cells be quickly decided if a recommendation strategy or data change has had a positive or negative impact. Given a Notebook containing one query cell, the recom- User studies, on the other hand, can verify that developers mendation process is depicted in Figure 3. First, the given perceive the recommendations as relevant. However, they Notebook goes through the transform and load process. This require much time and effort and are difficult to generalize way, the query data is represented the same way, i.e., the and replicate. query data is labeled and embedded in the same vector space Consequently, the partial code reduction evaluation is as the recommendation data. more efficient in checking whether a change in the rec- To recommend the k most similar code cells, the recom- ommendation strategy improves its performance. For this mendation strategy takes advantage of the labels and the reason, we used a partial code reduction strategy with gen- cell embeddings. First, all labeled cell embeddings in the erated evaluation data consisting of query and recommenda- database having the same label as the labeled query cell em- tion data. We added noise to the recommendation data to bedding are filtered to reduce the search space. Second, an ”confuse“ the recommendation strategy and thereby test its approximate nearest neighbor (k-ANN) search using cosine robustness. similarity is performed to obtain the k most similar labeled cell embeddings to the labeled query cell embedding. Cosine similarity was chosen among other distance metrics because 5.2. Evaluation Data it provides a better-standardized comparability of vectors. Before recommendations can be requested, the database Since the code snippet associated with a labeled cell embed- must be filled with recommendation data and query data ding is stored as metadata, the associated code snippets of for which recommendations will be given. the k most similar cells are returned as recommendations. The conceptual idea of generating the query and recom- mendation datasets is illustrated in Figure 4. In the follow- ing, we will describe the generation process for each dataset 5. A Framework for the Quantitative in detail. Evaluation of Cell Recommenders Query Dataset: Since we want developers to receive recommendations as they type, query cells with different After presenting the strategy for generating recommenda- numbers of lines of code per cell should be contained in the tions, we present in this section how a framework to quanti- query dataset. 6 Relevance slled yreuq etareneg relevant / similar not relevant / not similar sCOL gnicuder yb recommended in top-k True Positive (TP) False Positive (FP) Recommen- apply mutation operators dation m1 m 1, m 2 m1 .. mk not recommended False Negative (FN) True Negative (TN) Seed Cell in top-k Query Recommendation Dataset Dataset Figure 5: Relevance confusion matrix for cell recommendations Figure 4: Generation of the query and recommendation datasets based on seed cells Relevance Classification Figure 5 shows a confusion matrix representing the classifi- cation schema for recommendations. It defines the classes To this end, the generation starts with a set of unique True Positive (TP), False Positive (FP), False Negative (FN), Notebook code cells S, referred to as seed cells. and True Negative (TN). The number of lines in each seed cell is gradually reduced For each top-k recommendation for a query cell, its rel- in the generation process, starting with the last line until evance class must be decided according to our hypothesis, only the first line of code is left. Empty lines are ignored. leading to the following classification rules: Hence, for a seed cell with n lines of code, n query cells are generated. For each seed cell 𝑠, the set 𝑄𝐶𝑠 , consisting of 1. If a recommendation is a mutated version of the seed the seed cells and all generated query cells, is added to the cell, it is considered relevant and classified as TP. query dataset QD. 2. If a mutated version of the seed cell is not listed in Recommendation Dataset: Since a top-k cell recom- the top-k recommendations, then the missing rec- mender suggests the top-k cells that are most similar, the ommendation is classified as FN. recommendation dataset must have k recommendation cells 3. If the top-k recommendation list contains a recom- with different degrees of similarity for each seed cell. mendation cell not generated by applying mutation The process to generate these recommendation cells has operators to the seed cell, it is not considered rele- to ensure that the different degrees of similarity conform vant and is classified as FP. to the expected order of the top-k recommendations. To 4. If a recommendation is not generated from the seed generate similar recommendation cells from a given seed cell and does not appear in the top-k recommenda- cell 𝑠 ∈ S, mutation operators are applied. Therefore, a set tions, it is classified as TN. of k ordered mutation operators {𝑚1 ..𝑚𝑘 } must be defined. Using these mutation operators, the generation process is Relevance Rating as follows: To rate the relevance of a recommendation 𝑟, a rating func- • Step 1: apply the mutation operator 𝑚1 to the seed tion needs to be defined that maps 𝑟 to the values of a scale. cell 𝑠. This rating scale must specify values for each relevant (mu- • Step i, 2 ≤ 𝑖 ≤ 𝑘: apply the mutation operator 𝑚𝑖 tated) recommendation and include at least one value for on the recommendation cell generated in step i-1. irrelevant recommendations. The values must be different for each recommendation resulting from a different number Since each generation step applies one more mutation of applied mutation operators so that the ranking order of operator on the original seed cell, the similarity of the gen- the recommendations can be considered. erated recommendation cell decreases step by step. A top-k cell recommender requires a rating scale with at least k+1 values where k values are above the rating 5.3. Assessment of Recommendations threshold. The following section describes the selected performance Our assessment approach is based on the following hypoth- metrics based on this classification scheme and relevance esis: a perfect top-k cell recommender recommends the rating. mutated recommendation cells according to the number of mutation operators applied. Thus, the first recommendation 5.4. Selected Performance Metrics would be the recommendation cell with one mutation oper- ator applied, the second recommendation with two, and so Commonly used metrics for top-k recommender are preci- on. sion@k, recall@k, AP@k and nDCG@k [9]. In this section, Therefore, a recommendation to a query cell is considered we present the adoption of these metrics to evaluate a top-k relevant if it results from applying mutation operators to cell recommender. the original seed cell. precision@k: measures how many recommendations Metrics are used to evaluate a top-k cell recommender’s of the top-k positions are relevant. For a query cell 𝑞 ∈ QD performance. Some metrics use a relevance classification of and the set 𝑇 𝑃𝑞 consisting of the relevant cells in the top- the given recommendations. Others use a rating score and a k recommendations for this query cell, is it calculated as rating threshold. follows: |𝑇 𝑃𝑞 | 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑘𝑞 = 𝑘 7 recall@k: measures the share of relevant cells that are recommendations 𝑟 ∈ 𝑅𝑞 such that they are in a descending also contained in the top-k recommendations. Its value is ranking order based on their assigned rating 𝑟𝑎𝑡𝑖𝑛𝑔(𝑟)𝑞 for calculated as follows: the query cell 𝑞. 𝑛𝐷𝐶𝐺@𝑘𝑞 results in the highest score (1) for a query |𝑇 𝑃𝑞 | 𝑟𝑒𝑐𝑎𝑙𝑙@𝑘𝑞 = cell 𝑞 if the recommendations are in the ranking order of |𝑇 𝑃𝑞 ∪ 𝐹 𝑁𝑞 | their relevance based on the assigned ratings. where the set 𝑇 𝑃𝑞 ∪ 𝐹 𝑁𝑞 consists of recommendations Mean: Let M = {precision@k, recall@k, AP@k, considered relevant for the query cell 𝑞 ∈ QD. nDCG@k } be the set of the selected performance met- F1@k: combines the metrics precision@k and recall@k rics. This metric computes the overall mean for each used in one metric and weights them equally. It is calculated this performance metric 𝑚 ∈ M for a given set of query cells way: 𝑄 ⊆ QD as follows: 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑘𝑞 × 𝑟𝑒𝑐𝑎𝑙𝑙@𝑘𝑞 1 ∑︁ 𝐹 1@𝑘𝑞 = 2 × 𝑚𝑒𝑎𝑛𝑚 @𝑘(𝑄) = 𝑚@𝑘𝑞 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑘𝑞 + 𝑟𝑒𝑐𝑎𝑙𝑙@𝑘𝑞 |𝑄| 𝑞∈𝑄 AP@k: measures the average of the precision values 5.5. Procedure at different cut-off points in the top-k recommendations. It evaluates the accuracy of the ranking order of the top-k To calculate and present these metrics, the following steps recommendations for a query cell 𝑞 ∈ QD. are performed, given a set S of seed cells: It is calculated as follows: 1. Generate the query dataset QD and the recommendation ∑︀𝑘 dataset based on S. 𝑖=1 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑖𝑞 × 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑐𝑒(𝑟𝑖 ) 2. For each query cell 𝑞 ∈ QD: 𝐴𝑃 @𝑘𝑞 = 𝑇 𝑃𝑞 a) Request recommendations for 𝑞 . b) Classify the recommendations according to the where the function 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑐𝑒(𝑟𝑖 ) for the recommenda- classification rules. tion 𝑟𝑖 at rank 𝑖 is defined as follows: c) Compute all metrics 𝑚 ∈ M for 𝑞 . 3. For each metric 𝑚 ∈ M: a) Compute 𝑚𝑒𝑎𝑛𝑚 @𝑘(QD) on all values of 𝑚. {︃ 1, if 𝑟𝑎𝑡𝑖𝑛𝑔(𝑟𝑖 )𝑞 ≥ 𝑧. 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑐𝑒(𝑟𝑖 ) = 4. For each seed cell 𝑠 ∈ S and for each 𝑚 ∈ M: 0, otherwise. a) Compute 𝑚𝑒𝑎𝑛𝑚 @𝑘(𝑄𝐶𝑠 ) on all values of 𝑚. 5. Create the performance report. The rating function 𝑟𝑎𝑡𝑖𝑛𝑔(𝑟𝑖 )𝑞 of a recommendation 𝑟𝑖 has to be defined as described in Section 5.3. If the resulting rating score is above the defined rating threshold 𝑧, the 6. Implementing the Strategy and recommendation is considered relevant. 𝐴𝑃 @𝑘𝑞 results in the highest score (1) if relevant rec- the Evaluation Framework ommendations (i.e., the mutations) are ranked higher than To demonstrate the application of the evaluation framework irrelevant recommendations in the top-k positions. How- to a top-k cell recommender, we developed JupyRecSys, im- ever, the ranking order of the relevant recommendations plementing the cell recommendation strategy presented in does not matter. The score is the lowest (0) if no relevant Section 4 and the CelRecEval tool to automate the pre- recommendations are shown in the top-k positions. sented evaluation framework. The implementations and nDCG@k: In addition to the AP@k, the Normalized Dis- adjustments to the recommendation strategy and the evalu- counted Cumulative Gain also evaluates the ranking order ation framework are described below. of the recommendations based on a non-binary relevance rating. This also enables an evaluation of the ranking order of the relevant recommendations (i.e., recommendations 𝑟 6.1. JupyRecSys - A Top-3 Cell with 𝑟𝑎𝑡𝑖𝑛𝑔(𝑟)𝑞 ≥ 𝑧). Recommender Given a list of recommendations 𝑅𝑞 for a query cell 𝑞 ∈ The top-3 cell recommender JupyRecSys is implemented QD, DCG is calculated based on the defined rating function as a Python library, which is integrated into a JupyterLab 𝑟𝑎𝑡𝑖𝑛𝑔(𝑟𝑖 )𝑞 for each recommendation 𝑟𝑖 ∈ 𝑅𝑞 for a rank extension. It consists of a back- and frontend; its user inter- 𝑖 ∈ [1, ..., |𝑅𝑞 | = 𝑘] as follows: face is shown in Figure 6. The backend provides REST end- 𝑘 points to upload Notebooks to the CelRec-DB or to request ∑︁ 2𝑟𝑎𝑡𝑖𝑛𝑔(𝑟𝑖 )𝑞 − 1 recommendations. To access the CelRec-DB, a dedicated 𝐷𝐶𝐺@𝑘𝑞 = 𝑖=1 𝑙𝑜𝑔2 (𝑖 + 1) database service is provided, which offers all most often needed queries to facilitate the search for similar cells. Each DCG@k value per query cell is normalized. This We used existing tools and technologies to implement allows us to compare the accuracy between different recom- some steps in the recommendation strategy. To implement menders, even returning recommendation lists of different the embedd cell process, we applied our cell labeling tool lengths: JupyLabel [10] to get the cell’s task label. Further, we used 𝐷𝐶𝐺@𝑘𝑞 the pre-trained CodeBERT model [11] to analyze the cell 𝑛𝐷𝐶𝐺@𝑘𝑞 = tokens and map them into a vector space in the implementa- 𝑚𝑎𝑥𝜋 𝐷𝐶𝐺𝜋 @𝑘𝑞 tion of the generate embedding action. Finally, we used the where 𝑚𝑎𝑥𝜋 𝐷𝐶𝐺𝜋 @𝑘𝑞 represents the DCG for an opti- specialized vector database Milvus DB [12] to set up the mal recommendation list 𝑅𝑞+ for the query cell 𝑞 ∈ QD. 𝑅𝑞+ CelRec-DB and applied its optimized k-ANN search method is created from 𝑅𝑞 with a permutation 𝜋 that reorders the that supports cosine similarity as a standard feature. 8 CelRecEval implements the cell relevance classification rules based on the presented confusion matrix (Section 5.3). As relevance rating, the following scores are assigned to the top-3 recommendations, based on how the recommendation is related to the original seed cell of the query cell: 5 (M1 applied), 4 (M1 & M2 applied), 3 (M1, M2 & M3 applied) and 1 (not related to query cell). Consequently, the used relevance threshold is 𝑧 = 3. Further, CelRecEval implements the selected performance metrics accordingly. The parameters must be changed to apply CelRecEval to other top-k cell recommenders. Further, the mutation oper- ators and the REST endpoints for uploading query and rec- ommendation data and requesting recommendations must be adapted accordingly. If a different CelRec-DB is used, an adapter for deleting the data has to be provided since each evaluation has to start with an empty database. Figure 6: Screenshot of JupyRecSys UI with provided code cell recommendations 7. Evaluation For the evaluation, we selected 114 unique seed cells from 6.2. CelRecEval - A Cell Recommender the Notebooks provided by KGTorrent [13]. Using CelRec- Eval, a query dataset with 810 cells and a recommendation Evaluation Tool dataset with 342 cells were generated. The 810 recommen- CelRecEval implements the cell recommender evaluation dation lists returned by JupyRecSys were evaluated, and framework, especially for JupyRecSys. It is provided as the performance report was generated. Two sets of mean a Python CL-tool, allowing the user to configure various performance scores (MPS) were calculated: parameters such as the number of recommended cells (top- MPS1: All query cells’ mean scores of all performance met- k), the number of recommended cells per seed cell, or the rics 𝑚 ∈ M were computed. metric selection. In addition, the report’s output format, e.g., 𝑚𝑒𝑎𝑛𝑚 @𝑘(QD) an Excel spreadsheet, can be configured. MPS2: For each seed cell 𝑠 ∈ S, the mean performance As the generation of the evaluation data depends on the scores of all performance metrics 𝑚 ∈ M for all its cell recommender’s k-value and we wanted to evaluate the generated query cells (𝑄𝐶𝑠 ) were computed. top-3 cell recommender JupyRecSys, three mutation oper- ∀𝑠 ∈ S : 𝑚𝑒𝑎𝑛𝑚 @𝑘(𝑄𝐶𝑠 ) ators for the generation of the recommendation data are implemented: This way, we can gain insights into the overall perfor- mance, analyze performance across different code cell types, M1: Only the variable names are changed. This is a com- and identify difficulties with specific seed cells. mon case when rewriting similar code. In the following, we interpret the obtained performance M2: The variable names are changed, and comment lines metric scores according to the evaluation questions EQ1 and separate the lines of the query cell. This case is es- EQ2 presented in Section 5. pecially interesting for recommendation strategies, which also consider comments. 7.1. EQ1: Performance of the M3: The variable names are changed, comment lines sep- arate the lines of the query cell, and the code lines Recommendation Strategy of the query cell are duplicated and inverted. Table 7.1 depicts the mean performance scores of all query cells (MPS1). The resulting cell mutations, especially the ones gener- The overall mean precision@3, recall@3, and F1@3 ated by the M3 oerator, which introduces a lot of noise by scores are all equal, with a value of 0.8697. This is due to generating nonsense code, allow us to make statements the specific characteristics of the evaluation setup. As each about the robustness of the recommendation strategy. recommendation list contains exactly three recommenda- The following code illustrates the application of the mu- tions, the number of potentially relevant recommendations tation operators on an example seed cell: is also 3. Therefore, precision@3 and recall@3 have the 1 lr = LogisticRegression() same denominator (|𝑇 𝑃𝑞 ∪ 𝐹 𝑁𝑞 | = 3) and also share the 2 lr.fit(X_train,y_train) same numerator (|𝑇 𝑃 |). If precision@3 and recall@3 are Listing 1: Example seed cell equal, their harmonic mean is also equal, resulting in the same value for the F1@3 score. After applying all three mutation operators, the following Moreover, the mean AP@3 achieves an even higher, re- code cell is generated as one of the recommendation cells: markable value of 0.9579. This highlights the strategy’s 1 new_lr = new_LogisticRegression() effectiveness in prioritizing relevant recommendations at 2 noissergeRcitsigoL_wen = rl_wen the top of the recommendation lists. In this evaluation, a 3 # Additional comment line query cell’s mutated and relevant versions are consistently 4 new_lr.fit(new_X_train, new_y_train) ranked higher than other irrelevant cells. 5 )niart_y_wen ,niart_X_wen(tif.rl_wen The higher AP@3 score indicates that while the recom- Listing 2: Recommendation cell with M1-M3 applied mendations are correctly ranked, some relevant cells are 9 MPS1: Mean Performance Scores precision@3 recall@3 F1@3 AP@3 nDCG@3 0.8697 0.8697 0.8697 0.9579 0.817 Table 1 Mean performance scores of all query cells (∀𝑚 ∈ M : 𝑚𝑒𝑎𝑛𝑚 @𝑘(QD)) not always included in the recommendation lists. By ana- cell task classifier could further reduce the search space lyzing the MPS2 scores (see Table 7.2), we observe that the within the semantic vector space. The evaluation results following cell characteristics negatively impact the recom- in Section 7 highlight the effectiveness of the proposed cell mendation strategy’s performance : recommendation strategy. To answer RQ2, we designed and implemented an evalu- • For-loops: The three seed cells with the lowest ation framework in the CL-tool CelRecEval. precision@3, recall@3 and F1@3 scores are the cells We could have generated the evaluation data with iden- 𝑠76 , 𝑠4 , and 𝑠40 . All three cells contain a for-loop tical query-recommendation pairs. However, this would with data operations. In particular, if a query cell have only shown that the similarity calculation works and only contains the first line of a for-loop, this leads would be a rather simplistic evaluation for a top-k cell rec- to matches with all cells that contain a for-loop, as ommender, where ranking also plays a role. This would not these usually only differ in the variable names. reflect real-world usage. • Multiple tasks: Seed cell 𝑠8 achieved only moder- In practice, developers would receive recommendations ate results. A closer look revealed that this cell im- as they type rather than when they have finished writing plements multiple tasks: dependency import, model a code cell. By introducing cells with different numbers of prediction, and model evaluation. Considering only lines of code and using mutation operators, we were able the first few code lines, the recommender suggests to identify weaknesses such as for-loops, multiple tasks, and other dependency import code cells. many lines of code. Without automated evaluation, it would • Many code lines: It could be observed that the have been necessary to manually check all query cells to strategy’s performance is lower for seed cells having identify these issues. many lines of code, e.g., 𝑆𝐿𝑂𝐶𝑠4 = 12. Furthermore, CelRecEval can be re-executed with the same setup so that we can evaluate in the future whether changes to the implemented recommendation strategy elim- 7.2. EQ2: Impact of SLOCs to Performance inate the weaknesses mentioned above. Scores Consequently, CelRecEval enables developers to identify If a seed cell contains precisely one code line, one query cell the strengths and weaknesses of cell recommenders and and three mutated recommendation cells are created. For compare different ones. this query cell, the recommendations given are perfect, and the performance scores are optimal. This was the case for 14 out of 114 seed cells. 9. Threats to Validity Our analysis of the performance results indicates a sig- Internal Validity nificant correlation between the number of code lines and the strategy’s performance. The generated query cells, hav- In this paper, we automatically evaluated the performance ing fewer lines than their common seed cell, sometimes of a cell recommendation strategy using our definition of exhibit higher similarity with recommendation cells with relevancy of a recommendation. When evaluating recom- similar numbers of lines instead of the recommendation menders, users usually evaluate a recommendation’s rele- cells that would ultimately be relevant to the developer. vancy in a so-called online evaluation. However, the impact This phenomenon can lead to erroneous recommendations, of this threat should be limited, as we consider the code’s lowering the average metric result for the specific seed cell. similarity and the performed task implemented in a cell as Therefore, it can be concluded that the more lines a devel- criteria for recommendation. oper enters, the more accurate the recommendations will become. External Validity Nevertheless, even the lowest-performing seed cell The evaluation framework can be adapted with minor adjust- achieves an nDCG@3 value of at least 0.5, indicating that ments to other top-k cell recommenders. It allows the auto- approximately half of the relevant recommendations are matic generation of evaluation data suitable for the selected still recommended in the correct ranking order within the k. Currently, the evaluation performance results are used top-3 ranks. to compare different versions of JupyRecSys against each other. Different values for k and different sets of mutation 8. Discussion operators may hinder comparability among recommenders. In response to research question RQ1, we proposed a cell recommendation strategy, which we implemented in the 10. Conclusion and Future Work JupyterLab extension JupyRecSys. The transformation of This paper has two main contributions. First, we present a code cells into a semantic vector space enables the efficient strategy for Jupyter Notebook code cell recommendation. computation of similarities between a query cell and a large number of recommendation cells. Applying an automatic 10 MPS2: Mean Performance Scores for selected Seed Cells (sorted by nDCG@3) Seed Cell ID SLOC precision@3 recall@3 F1@3 AP@3 nDCG@3 48 1 1 1 1 1 1 70 1 1 1 1 1 1 103 1 1 1 1 1 1 ... ... ... ... ... ... ... 8 6 0.7143 0.7143 0.7143 0.7143 0.6514 ... ... ... ... ... ... ... 76 20 0.5333 0.5333 0.5333 0.7833 0.6443 4 12 0.5385 0.5385 0.5385 0.7436 0.6406 40 2 0.6667 0.6667 0.6667 0.75 0.5001 Table 2 Mean performance scores of all query cells 𝑄𝐶𝑠 generated from all seed cells (∀𝑄𝐶𝑠 𝑠 ∈ S, ∀𝑚 ∈ M : 𝑚𝑒𝑎𝑛𝑚 @𝑘(𝑄𝐶𝑠 )) Second, we present a framework for automatically evaluat- ducibility of Jupyter Notebooks, Empirical Software ing such cell recommenders. The framework allows users Engineering 26 (2021) 65. to customize the generation of evaluation data, select per- [4] A. Rule, I. Drosos, A. Tabard, J. D. Hollan, Aiding formance metrics, and access external tools, such as the cell Collaborative Reuse of Computational Notebooks with recommender or its database. Both the recommendation Annotated Cell Folding, Proc. ACM Hum.-Comput. strategy and the evaluation framework are implemented as Interact. 2 (2018). tools: the strategy as the JupyterLab extension JupyRec- [5] M. Horiuchi, Y. Sasaki, C. Xiao, M. Onizuka, JupySim: Sys cell recommender and the evaluation framework as the Jupyter Notebook Similarity Search System., in: EDBT, Python CL-tool CelRecEval. 2022, pp. 2–554. To demonstrate the application of CelRecEval to a cell [6] Elyra Team, Code Snippets - Elyra 3.15.0 documen- recommender, the framework was adapted to JupyRecSys. tation, https://elyra.readthedocs.io/en/v3.15.0/user_ The evaluation results demonstrate that JupyRecSys exhibits guide/code-snippets.html, 2022. [Acc. 19-Sep-2024]. high performance across all metrics, effectively delivering [7] C. Ragkhitwetsagul, V. Prasertpol, N. Ritta, P. Sae- relevant code cells in a near-optimal order as the developer Wong, T. Noraset, M. Choetkiertikul, Typhon: Au- types code into the cell. Given its high metric scores, it is tomatic Recommendation of Relevant Code Cells in particularly well-suited to enhancing the reusability of Note- Jupyter Notebooks, 2024. books. In addition to evaluating general performance, we [8] S. Proksch, S. Amann, S. Nadi, M. Mezini, Evaluating can also make statements about how the cell recommender the evaluations of code recommender systems: A real- performs while a developer is typing into a code cell. Fur- ity check, in: 2016 31st IEEE/ACM International Con- ther, CelRecEval allows us to identify some strengths and ference on Automated Software Engineering (ASE), weaknesses of JupyRecSys. 2016, pp. 111–121. As part of our future research, we aim to extend CelRec- [9] Y.-M. Tamm, R. Damdinov, A. Vasilev, Quality Metrics Eval with more performance evaluation metrics to consider in Recommender Systems: Do We Calculate Metrics more recommender quality attributes, such as diversity or Consistently?, in: Proceedings of the 15th ACM Con- confidence. ference on Recommender Systems, RecSys ’21, Associ- Furthermore, the recommendation strategy of JupyRec- ation for Computing Machinery, New York, NY, USA, Sys could be improved by including Markdown text in the 2021, p. 708–713. similarity analysis. An online evaluation would give a more [10] M. Perez, S. Aydin, H. Lichter, A Flexible Cell accurate picture of the performance of the cell recommender, Classification for ML Projects in Jupyter Note- as users would evaluate the relevance of the recommenda- books, 2024. URL: https://arxiv.org/abs/2403.07562. tions in a real-world setting. A user study with developers arXiv:2403.07562. could also provide essential insights into the usability of [11] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, this recommender. L. Shou, B. Qin, T. Liu, D. Jiang, M. Zhou, CodeBERT: A All software artifacts, including JupyRecSys, CelRecEval Pre-Trained Model for Programming and Natural Lan- and the evaluation data are available on Zenodo [14]. guages, 2020. URL: https://arxiv.org/abs/2002.08155. arXiv:2002.08155. [12] Milvus, The High-Performance Vector Database Built References for Scale, https://milvus.io/, 2024. [Acc. 13-Sep-2024]. [13] L. Quaranta, F. Calefato, F. Lanubile, KGTorrent: A [1] N. Ritta, T. Settewong, R. G. Kula, C. Ragkhitwetsagul, Dataset of Python Jupyter Notebooks from Kaggle, T. Sunetnanta, K. Matsumoto, Reusing My Own Code: in: 2021 IEEE/ACM 18th International Conference on Preliminary Results for Competitive Coding in Jupyter Mining Software Repositories (MSR), IEEE, 2021. Notebooks, in: 2022 29th Asia-Pacific Software Engi- [14] S. Aydin, D. Mertens, O. Xu, Zenodo: An Automated neering Conference (APSEC), 2022, pp. 457–461. Evaluation Approach for Jupyter Notebook Code Cell [2] M. Källén, T. Wrigstad, Jupyter Notebooks on GitHub: Recommender Systems - Software Artifacts, https:// Characteristics and Code Clones, The Art, Science, doi.org/10.5281/zenodo.13836922, 2024. and Engineering of Programming 5 (2021). [3] J. F. Pimentel, L. Murta, V. Braganholo, J. Freire, Un- derstanding and Improving the Quality and Repro- 11