The reuse of code within Jupyter Notebooks is an often overlooked concept, resulting in the loss of valuable knowledge that could be retrieved from already created Notebooks. Presently, the reuse of Jupyter Notebooks is particularly impeded by the absence of suitable reuse approaches and tools. This paper presents a code cell recommender system in the form of the JupyterLab extension, designated as JupyRecSys. Moreover, we propose an automatic evaluation framework in the form of the CL-tool CelRecEval, which allows developers to evaluate and compare the performance of cell recommender systems. The evaluation capabilities of CelRecEval are demonstrated by applying it to JupyRecSys. The resulting high metric scores demonstrate that JupyRecSys can correctly recommend and rank relevant cells. Moreover, the generation of more detailed performance reports enabled the identification of specific characteristics of code cells that negatively impact the performance of the cell recommendation system.
The Jupyter Notebook (Notebook for short) has become
a widely utilized tool in academia and industry for
prototyping Python ML solutions due to their flexibility and
interaction. Studies have demonstrated that code from other
Notebooks is frequently reused through copy-paste. In
particular, code for importing packages and visualizing data
is often duplicated [
An initial challenge is to identify and locate relevant code from previous Notebooks. While search systems can be used for this purpose, they require manual intervention by the developer. The success of the search depends on the keywords used. An alternative approach would be to use a recommender system that suggests relevant cells from previous Notebooks to match the current cell. This would significantly reduce the efort required, as no action by the developer would be necessary.
A Notebook cell recommendation system (cell recommender for short) implements a special form of code recommendation, suggesting similar code cells instead of code completions. Cell recommenders have not yet been explored, and an evaluation method for them has not yet been developed.
The paper is structured as follows: Section 2 presents an overview of existing tools and methods to reuse Notebooks. In Section 3, we state the research goals and contributions of the paper. Then, in Section 4, we present a cell recommendation strategy. Next, Section 5 presents a general evaluation framework for cell code recommenders. Section 6 describes the implementation of the recommendation strategy and the evaluation framework. The setup and results of the evaluation are discussed in Section 7; the research questions are answered in Section 8. After presenting the threats to validity in Section 9, Section 10 concludes the paper and gives an outlook for potential future work. This section reviews recent tools and methods to improve the code reusability of Notebooks and their evaluation approaches.
JupySim, developed by Horiuchi et al., models Notebooks
as directed acyclic graphs to identify the relational
structures between code, data, and outputs [
The Elyra code snippet JupyterLab extension represents
a further significant addition to the Jupyter ecosystem [
While JupySim ofers a sophisticated graph-based approach to identifying similar Notebooks, its complexity and standalone nature may not be optimal for everyday use. In contrast, Elyra, with its integrated JupyterLab interface, streamlines code snippet reuse but necessitates manual labeling and does not support the reuse of entire Notebooks.
A first step towards an automated solution is Typhon
[
Given the strengths and limitations of the existing approaches, our current work is focused on addressing these gaps by answering the following research questions (RQ): RQ1: What can a cell recommendation strategy look like that makes suggestions to the developer during the programming of a cell? RQ2: How can an approach for the automatic quantitative evaluation of the performance of cell recommenders look like?
By answering these questions, this paper makes the following contributions to improve the reusability of Notebooks: • A cell recommendation strategy specialized for
Python Notebooks implementing ML tasks. • A general framework to quantitatively evaluate cell recommenders implementing this strategy. • The JupyterLab extension JupyRecSys which implements the cell recommendation strategy. • The CL-tool CelRecEval which implements this evaluation approach. • The results of applying CelRecEval to evaluate the performance of the cell recommender JupyRecSys.
In this section, we present a recommendation strategy for cell recommenders. A cell recommender that suggests code based on similarity to a query cell is classified as a contentbased recommender. Since it only provides the top-k most similar cells, we refer to it as a top-k cell recommender.
A common approach in such recommenders is to embed recommendation items in a semantic vector space. This has several advantages. First, it significantly reduces the complexity of the items, e.g., code, syntax, semantics, variables, symbols, etc. Further, a vector representation within a semantic vector space allows the recommender to eficiently determine similarities using various distance metrics, such as cosine similarity or Euclidean distance. In the vector space, the proximity of vectors directly reflects the degree of similarity between them. At the same time, the results remain interpretable because distances between vectors are easy to comprehend. Notebook Cell
Classify Task
Tokenize Cell Tokens Generate
Embedding Task Label
Cell Embedding Build Embedding
Labeled Cell Embedding
First, we explain how code cells are transformed into a vector representation in a semantic vector space, resulting in a cell embedding as output. Then, we present the process of transforming and loading, which stores cell embeddings in an appropriate database. Finally, we explain how recommendations are provided using the stored cell embeddings.
When code cells are transformed into vector representations and embedded in a vector space, they are particularly close to each other if they represent similar code. To further reduce the search space around a cell embedding and get more accurate results, each cell embedding is labeled with the ML task it implements.
The embedding process is shown in Figure 1 as a UML activity diagram. The input is a single Notebook cell. The process consists of two parts that are executed in parallel. The specific ML task implemented in a cell is classified in the left part, returning a task label, e.g. “data preprocessing” or “model training”. The right part of the process consists of two actions. First, the code of the cell is tokenized into its elements (e.g., keywords, operators, identifiers), creating the cell tokens. Second, in the generate embedding action, the syntax and the semantic relationships between the tokens are analyzed and mapped into a vector space, returning a cell embedding. Finally, the task label and the cell embedding are composed to a labeled cell embedding.
To recommend code from previous Notebooks, the vector representations and required metadata of code cells must be stored in a dedicated database called CelRec-DB. A CelRecDB must fulfill the following requirements: • Storage space: the value and quality of the recommendations increase with the amount of available data for recommendation. Thus, the CelRec-DB must store the data in a scalable and eficient manner. • Data retrieval: for recommendations, it is crucial that the stored data can be accessed quickly. Otherwise, the value is reduced if recommendations take too long. • Data representation: the CelRec-DB has to support vectors and vector operations, ideally distance computations. Pre-Process Cells Pre-Processed
Cells
Given a Notebook containing one query cell, the recommendation process is depicted in Figure 3. First, the given Notebook goes through the transform and load process. This way, the query data is represented the same way, i.e., the query data is labeled and embedded in the same vector space as the recommendation data.
To recommend the k most similar code cells, the recommendation strategy takes advantage of the labels and the cell embeddings. First, all labeled cell embeddings in the database having the same label as the labeled query cell embedding are filtered to reduce the search space. Second, an approximate nearest neighbor (k-ANN) search using cosine similarity is performed to obtain the k most similar labeled cell embeddings to the labeled query cell embedding. Cosine similarity was chosen among other distance metrics because it provides a better-standardized comparability of vectors. Since the code snippet associated with a labeled cell embedding is stored as metadata, the associated code snippets of the k most similar cells are returned as recommendations.
After presenting the strategy for generating recommendations, we present in this section how a framework to quanti
Recommend Notebook with Query Cell
Transform &
Load k-ANN
Search <<datastore>> CelRec-DB
Top-k Similar Code Snippets tatively evaluate the performance of cell recommenders can be designed. By using this framework, we want to answer the following evaluation questions (EQ): EQ1: What is the performance of the top-k cell recommender? EQ2: How does its performance vary with increasing lines of code in a query cell?
The first question concerns performance in general. The second question is specific to a developer receiving recommendations while programming and wanting to write more lines of code in a cell. This would also show how much code is needed to obtain relevant recommendations.
Code cell recommendation for Notebooks is a special case
of code recommendation. Code recommenders, in general,
usually suggest code for code completion. There are two
main strategies for evaluating code recommenders: partial
code reduction and user studies [
In the former, code snippets are taken, and the last lines are removed, mimicking that the developer has started typing code and expects a recommendation. Then, given the code recommendation, it is checked whether it matches the removed code. This allows for the analysis of the recommender’s performance. If this is done regularly, it can be quickly decided if a recommendation strategy or data change has had a positive or negative impact.
User studies, on the other hand, can verify that developers perceive the recommendations as relevant. However, they require much time and efort and are dificult to generalize and replicate.
Consequently, the partial code reduction evaluation is more eficient in checking whether a change in the recommendation strategy improves its performance. For this reason, we used a partial code reduction strategy with generated evaluation data consisting of query and recommendation data. We added noise to the recommendation data to ”confuse“ the recommendation strategy and thereby test its robustness.
Before recommendations can be requested, the database must be filled with recommendation data and query data for which recommendations will be given.
The conceptual idea of generating the query and recommendation datasets is illustrated in Figure 4. In the following, we will describe the generation process for each dataset in detail.
Query Dataset: Since we want developers to receive recommendations as they type, query cells with diferent numbers of lines of code per cell should be contained in the query dataset. llrsyed sLCO e g qu icn te u a de re r gen yb Seed Cell m1 apply mutation operators m1, m2
To this end, the generation starts with a set of unique Notebook code cells S, referred to as seed cells.
The number of lines in each seed cell is gradually reduced in the generation process, starting with the last line until only the first line of code is left. Empty lines are ignored. Hence, for a seed cell with n lines of code, n query cells are generated. For each seed cell , the set , consisting of the seed cells and all generated query cells, is added to the query dataset QD.
Recommendation Dataset: Since a top-k cell recommender suggests the top-k cells that are most similar, the recommendation dataset must have k recommendation cells with diferent degrees of similarity for each seed cell.
The process to generate these recommendation cells has to ensure that the diferent degrees of similarity conform to the expected order of the top-k recommendations. To generate similar recommendation cells from a given seed cell ∈ S, mutation operators are applied. Therefore, a set of k ordered mutation operators {1..} must be defined. Using these mutation operators, the generation process is as follows: • Step 1: apply the mutation operator 1 to the seed cell . • Step i, 2 ≤ ≤ : apply the mutation operator on the recommendation cell generated in step i-1.
Since each generation step applies one more mutation operator on the original seed cell, the similarity of the generated recommendation cell decreases step by step.
Our assessment approach is based on the following hypothesis: a perfect top-k cell recommender recommends the mutated recommendation cells according to the number of mutation operators applied. Thus, the first recommendation would be the recommendation cell with one mutation operator applied, the second recommendation with two, and so on.
Therefore, a recommendation to a query cell is considered relevant if it results from applying mutation operators to the original seed cell.
Metrics are used to evaluate a top-k cell recommender’s performance. Some metrics use a relevance classification of the given recommendations. Others use a rating score and a rating threshold. recommended
in top-k n relevant / similar
not relevant / not similar True Positive (TP)
False Positive (FP) False Negative (FN)
True Negative (TN) To rate the relevance of a recommendation , a rating function needs to be defined that maps to the values of a scale. This rating scale must specify values for each relevant (mutated) recommendation and include at least one value for irrelevant recommendations. The values must be diferent for each recommendation resulting from a diferent number of applied mutation operators so that the ranking order of the recommendations can be considered.
A top-k cell recommender requires a rating scale with at least k+1 values where k values are above the rating threshold.
The following section describes the selected performance metrics based on this classification scheme and relevance rating.
Commonly used metrics for top-k recommender are
precision@k, recall@k, AP@k and nDCG@k [
precision@k: measures how many recommendations of the top-k positions are relevant. For a query cell ∈ QD and the set consisting of the relevant cells in the topk recommendations for this query cell, is it calculated as follows: @ = | | recall@k: measures the share of relevant cells that are also contained in the top-k recommendations. Its value is calculated as follows:
| | | ∪ | where the set ∪ consists of recommendations considered relevant for the query cell ∈ QD.
F1@k: combines the metrics precision@k and recall@k in one metric and weights them equally. It is calculated this way:
@ × @ × @ + @
AP@k: measures the average of the precision values at diferent cut-of points in the top-k recommendations. It evaluates the accuracy of the ranking order of the top-k recommendations for a query cell ∈ QD.
It is calculated as follows: ∑︀ =1 @ × () where the function () for the recommendation at rank is defined as follows: () = {︃1, if () ≥ .
0, otherwise.
The rating function () of a recommendation has to be defined as described in Section 5.3. If the resulting rating score is above the defined rating threshold , the recommendation is considered relevant.
@ results in the highest score (1) if relevant recommendations (i.e., the mutations) are ranked higher than irrelevant recommendations in the top-k positions. However, the ranking order of the relevant recommendations does not matter. The score is the lowest (0) if no relevant recommendations are shown in the top-k positions.
nDCG@k: In addition to the AP@k, the Normalized Discounted Cumulative Gain also evaluates the ranking order of the recommendations based on a non-binary relevance rating. This also enables an evaluation of the ranking order of the relevant recommendations (i.e., recommendations with () ≥ ).
Given a list of recommendations for a query cell ∈ QD, DCG is calculated based on the defined rating function () for each recommendation ∈ for a rank ∈ [1, ..., || = ] as follows:
@ = ∑︁ 2() − 1
2( + 1) =1
Each DCG@k value per query cell is normalized. This allows us to compare the accuracy between diferent recommenders, even returning recommendation lists of diferent lengths: where @ represents the DCG for an optimal recommendation list + for the query cell ∈ QD. + is created from with a permutation that reorders the recommendations ∈ such that they are in a descending ranking order based on their assigned rating () for the query cell .
@ results in the highest score (1) for a query cell if the recommendations are in the ranking order of their relevance based on the assigned ratings.
Mean: Let M = {precision@k, recall@k, AP@k, nDCG@k } be the set of the selected performance metrics. This metric computes the overall mean for each used performance metric ∈ M for a given set of query cells ⊆ QD as follows: 1
∑︁ @ || ∈
To calculate and present these metrics, the following steps are performed, given a set S of seed cells: 1. Generate the query dataset QD and the recommendation dataset based on S. 2. For each query cell ∈ QD: a) Request recommendations for . b) Classify the recommendations according to the classification rules.
c) Compute all metrics ∈ M for . 3. For each metric ∈ M:
a) Compute @(QD) on all values of . 4. For each seed cell ∈ S and for each ∈ M:
a) Compute @() on all values of .
5. Create the performance report.
To demonstrate the application of the evaluation framework to a top-k cell recommender, we developed JupyRecSys, implementing the cell recommendation strategy presented in Section 4 and the CelRecEval tool to automate the presented evaluation framework. The implementations and adjustments to the recommendation strategy and the evaluation framework are described below.
The top-3 cell recommender JupyRecSys is implemented as a Python library, which is integrated into a JupyterLab extension. It consists of a back- and frontend; its user interface is shown in Figure 6. The backend provides REST endpoints to upload Notebooks to the CelRec-DB or to request recommendations. To access the CelRec-DB, a dedicated database service is provided, which ofers all most often needed queries to facilitate the search for similar cells.
We used existing tools and technologies to implement
some steps in the recommendation strategy. To implement
the embedd cell process, we applied our cell labeling tool
JupyLabel [
CelRecEval implements the cell relevance classification rules based on the presented confusion matrix (Section 5.3). As relevance rating, the following scores are assigned to the top-3 recommendations, based on how the recommendation is related to the original seed cell of the query cell: 5 (M1 applied), 4 (M1 & M2 applied), 3 (M1, M2 & M3 applied) and 1 (not related to query cell). Consequently, the used relevance threshold is = 3. Further, CelRecEval implements the selected performance metrics accordingly.
The parameters must be changed to apply CelRecEval to other top-k cell recommenders. Further, the mutation operators and the REST endpoints for uploading query and recommendation data and requesting recommendations must be adapted accordingly. If a diferent CelRec-DB is used, an adapter for deleting the data has to be provided since each evaluation has to start with an empty database.
CelRecEval implements the cell recommender evaluation framework, especially for JupyRecSys. It is provided as a Python CL-tool, allowing the user to configure various parameters such as the number of recommended cells (topk), the number of recommended cells per seed cell, or the metric selection. In addition, the report’s output format, e.g., an Excel spreadsheet, can be configured.
As the generation of the evaluation data depends on the cell recommender’s k-value and we wanted to evaluate the top-3 cell recommender JupyRecSys, three mutation operators for the generation of the recommendation data are implemented:
M1: Only the variable names are changed. This is a common case when rewriting similar code.
M2: The variable names are changed, and comment lines separate the lines of the query cell. This case is especially interesting for recommendation strategies, which also consider comments.
M3: The variable names are changed, comment lines separate the lines of the query cell, and the code lines of the query cell are duplicated and inverted.
The resulting cell mutations, especially the ones generated by the M3 oerator, which introduces a lot of noise by generating nonsense code, allow us to make statements about the robustness of the recommendation strategy.
The following code illustrates the application of the mutation operators on an example seed cell: 1 lr = LogisticRegression() 2 lr.fit(X_train,y_train)
Listing 1: Example seed cell
After applying all three mutation operators, the following code cell is generated as one of the recommendation cells: 1 new_lr = new_LogisticRegression() 2 noissergeRcitsigoL_wen = rl_wen 3 # Additional comment line 4 new_lr.fit(new_X_train, new_y_train) 5 )niart_y_wen ,niart_X_wen(tif.rl_wen
Listing 2: Recommendation cell with M1-M3 applied
For the evaluation, we selected 114 unique seed cells from
the Notebooks provided by KGTorrent [
@(QD) MPS2: For each seed cell ∈ S, the mean performance scores of all performance metrics ∈ M for all its generated query cells () were computed.
∀ ∈ S : @()
This way, we can gain insights into the overall performance, analyze performance across diferent code cell types, and identify dificulties with specific seed cells.
In the following, we interpret the obtained performance metric scores according to the evaluation questions EQ1 and EQ2 presented in Section 5.
recall@3 F1@3 0.8697 0.8697 not always included in the recommendation lists. By analyzing the MPS2 scores (see Table 7.2), we observe that the following cell characteristics negatively impact the recommendation strategy’s performance : • For-loops: The three seed cells with the lowest precision@3, recall@3 and F1@3 scores are the cells 76, 4, and 40. All three cells contain a for-loop with data operations. In particular, if a query cell only contains the first line of a for-loop, this leads to matches with all cells that contain a for-loop, as these usually only difer in the variable names. • Multiple tasks: Seed cell 8 achieved only moderate results. A closer look revealed that this cell implements multiple tasks: dependency import, model prediction, and model evaluation. Considering only the first few code lines, the recommender suggests other dependency import code cells. • Many code lines: It could be observed that the strategy’s performance is lower for seed cells having many lines of code, e.g., 4 = 12.
If a seed cell contains precisely one code line, one query cell and three mutated recommendation cells are created. For this query cell, the recommendations given are perfect, and the performance scores are optimal. This was the case for 14 out of 114 seed cells.
Our analysis of the performance results indicates a significant correlation between the number of code lines and the strategy’s performance. The generated query cells, having fewer lines than their common seed cell, sometimes exhibit higher similarity with recommendation cells with similar numbers of lines instead of the recommendation cells that would ultimately be relevant to the developer. This phenomenon can lead to erroneous recommendations, lowering the average metric result for the specific seed cell. Therefore, it can be concluded that the more lines a developer enters, the more accurate the recommendations will become.
Nevertheless, even the lowest-performing seed cell achieves an nDCG@3 value of at least 0.5, indicating that approximately half of the relevant recommendations are still recommended in the correct ranking order within the top-3 ranks.
In response to research question RQ1, we proposed a cell recommendation strategy, which we implemented in the JupyterLab extension JupyRecSys. The transformation of code cells into a semantic vector space enables the eficient computation of similarities between a query cell and a large number of recommendation cells. Applying an automatic cell task classifier could further reduce the search space within the semantic vector space. The evaluation results in Section 7 highlight the efectiveness of the proposed cell recommendation strategy.
To answer RQ2, we designed and implemented an evaluation framework in the CL-tool CelRecEval.
We could have generated the evaluation data with identical query-recommendation pairs. However, this would have only shown that the similarity calculation works and would be a rather simplistic evaluation for a top-k cell recommender, where ranking also plays a role. This would not reflect real-world usage.
In practice, developers would receive recommendations as they type rather than when they have finished writing a code cell. By introducing cells with diferent numbers of lines of code and using mutation operators, we were able to identify weaknesses such as for-loops, multiple tasks, and many lines of code. Without automated evaluation, it would have been necessary to manually check all query cells to identify these issues.
Furthermore, CelRecEval can be re-executed with the same setup so that we can evaluate in the future whether changes to the implemented recommendation strategy eliminate the weaknesses mentioned above.
Consequently, CelRecEval enables developers to identify the strengths and weaknesses of cell recommenders and compare diferent ones.
In this paper, we automatically evaluated the performance of a cell recommendation strategy using our definition of relevancy of a recommendation. When evaluating recommenders, users usually evaluate a recommendation’s relevancy in a so-called online evaluation. However, the impact of this threat should be limited, as we consider the code’s similarity and the performed task implemented in a cell as criteria for recommendation.
The evaluation framework can be adapted with minor adjustments to other top-k cell recommenders. It allows the automatic generation of evaluation data suitable for the selected k. Currently, the evaluation performance results are used to compare diferent versions of JupyRecSys against each other. Diferent values for k and diferent sets of mutation operators may hinder comparability among recommenders. 10. Conclusion and Future Work This paper has two main contributions. First, we present a strategy for Jupyter Notebook code cell recommendation.
MPS2: Mean Performance Scores for selected Seed Cells (sorted by nDCG@3) SLOC precision@3 recall@3 F1@3 AP@3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ... ... ... ... ... 6 0.7143 0.7143 0.7143 0.7143 ... ... ... ... ... 20 0.5333 0.5333 0.5333 0.7833 12 0.5385 0.5385 0.5385 0.7436 2 0.6667 0.6667 0.6667 0.75 Second, we present a framework for automatically evaluating such cell recommenders. The framework allows users to customize the generation of evaluation data, select performance metrics, and access external tools, such as the cell recommender or its database. Both the recommendation strategy and the evaluation framework are implemented as tools: the strategy as the JupyterLab extension JupyRecSys cell recommender and the evaluation framework as the Python CL-tool CelRecEval.
To demonstrate the application of CelRecEval to a cell recommender, the framework was adapted to JupyRecSys.
The evaluation results demonstrate that JupyRecSys exhibits high performance across all metrics, efectively delivering relevant code cells in a near-optimal order as the developer types code into the cell. Given its high metric scores, it is particularly well-suited to enhancing the reusability of Notebooks. In addition to evaluating general performance, we can also make statements about how the cell recommender performs while a developer is typing into a code cell. Further, CelRecEval allows us to identify some strengths and weaknesses of JupyRecSys.
As part of our future research, we aim to extend CelRecEval with more performance evaluation metrics to consider more recommender quality attributes, such as diversity or confidence.
Furthermore, the recommendation strategy of JupyRecSys could be improved by including Markdown text in the similarity analysis. An online evaluation would give a more accurate picture of the performance of the cell recommender, as users would evaluate the relevance of the recommendations in a real-world setting. A user study with developers could also provide essential insights into the usability of this recommender.
All software artifacts, including JupyRecSys, CelRecEval
and the evaluation data are available on Zenodo [