Clustering Students’ Short Text Reflections: A Software Engineering Course Case Study Mohsen Dorodchi Alexandria Benedict Erfan Al-Hossami University of North Carolina at University of North Carolina at University of North Carolina at Charlotte Charlotte Charlotte Charlotte, NC 28223 Charlotte, NC 28223 Charlotte, NC 28223 Mohsen.Dorodchi@uncc.edu abenedi4@uncc.edu ealhossa@uncc.edu Andrew Quinn Sandra Wiktor Aileen Benedict University of North Carolina at University of North Carolina at University of North Carolina at Charlotte Charlotte Charlotte Charlotte, NC 28223 Charlotte, NC 28223 Charlotte, NC 28223 aquinn16@uncc.edu swiktor@uncc.edu abenedi3@uncc.edu Mohammadali Fallahian University of North Carolina at Charlotte Charlotte, NC 28223 mfallahi@uncc.edu ABSTRACT instructor, especially in large classroom settings, where timely Student reflections can provide instructors with beneficial feedback is needed to address students’ possible concerns. knowledge regarding their progress in the course, what chal- Machine learning and knowledge discovery-based methods lenges they are facing, and how the instructor can provide have been used to assist educators in understanding and more effectively to the students’ needs. Reading every stu- helping students [14, 1, 20]. Unsupervised methods in natu- dent reflection, however, can be a time-consuming task that ral language processing (NLP) such as topic modeling, have may affect the instructor’s ability to efficiently address stu- been used to automatically extract topics from student re- dent needs in a timely manner. In this research, we ex- flective journals [5]. However, they fall short when it comes plore the use of clustering and sorting of student reflections to short text, typically around a sentence in length, such to shorten reading time while maintaining a comprehensive as tweets. Recent research has utilized K-means cluster- understanding of the reflection content. We obtain student ing along with transformer-based sentence-embeddings to reflections from a software engineering course. Next, we automatically extract topics from tweets [12, 2]. K-means generate transformer-based sentence embeddings and then clustering is often supplemented with a representation of cluster the reflections using K-Means. Lastly, we sort the text. Representations can include statistical-learnt repre- reflections based on the distance of each reflection from its sentations such as term frequency-inverse document (TF- cluster center. We conduct a small-scale user study with the IDF) [13], neural-learnt representations also known as word course’s Teaching Assistants and provide promising prelim- embeddings (e.g. Word2Vec [19], Glove [22]), and more re- inary results showing a significant increase in reading time cently representations computed from large pretrained trans- efficiency without sacrificing understanding. former deep learning models. Keywords Transformers are deep learning models following the archi- Natural Language Processing, Student Reflections, Cluster- tecture proposed by Vaswani et al. [27]. These models often ing undergo an unsupervised pretraining on a massive text cor- pus to create an initial version of the network later fine-tuned for more specific tasks in process called transfer learning. 1. INTRODUCTION Pretrained transformers such as BERT [6], RoBERTa [16], Reflections are an effective way for instructors to detect and GPT-3 [4], have achieved state of the art in many nat- what their students may be struggling with throughout their ural language processing tasks. Some of these tasks include courses, gain a perspective on students’ impressions of course detecting positive and uplifting discussion on social media content, and track their overall progress [9]. However, in or- (e.g. [17]), determining answers to questions given a passage der to utilize these benefits to the fullest, instructors would of text (e.g. [28]), summarizing text (e.g. [15]), and estimat- need to manually read through each individual reflection. ing semantic similarity between sentences. For this reason, Manually analyzing reflections can be overwhelming for an we select a transformer-based language model to create a semantic representation of student responses. Copyright ©2021 for this paper by its authors. Use permitted under Cre- ative Commons License Attribution 4.0 International (CC BY 4.0) In this research, we implement an approach using k-means clustering from the scikit-learn library and utilize transformer- 2. What was your biggest challenge this past week? This based sentence embeddings. We evaluate our approach in a can include in-class activities, assignments, prep work, preliminary user study, observing the time taken for teach- studying, time management, motivation, and so on. ing assistants to read and analyze student reflections. 3. How can you address the challenge you mentioned above? What can you do to overcome this challenge for next 2. DATASET time? Course. The data used in our research was collected from an undergraduate software engineering course based on the active learning course model proposed in [7]. The number of For the purpose of this research, we focused solely on the students enrolled in the course was 108 students. Modules students’ responses to question 2, as this question was free- are organized based on the concepts being taught and typ- response and would provide unique responses for the clus- ically spanned across approximately one week. The course tering process. contained 11 modules in total with the topics listed in Ta- ble 1. Following the active learning course model presented Dataset Statistics. We used two different module reflections from the software engineering course throughout this study: Table 1: Concepts taught within each module of the software Module 7 reflections and Module 8 reflections. Table 2 show- engineering course. cases our descriptive statistics of our collected student reflec- tion responses corpora. The selected module reflections were comparable in size. Firstly, the response rates to the Module Module Topic(s) 7 and Module 8 reflections are 94 or 87.0% and 89 or 82.4% Introduction to Software Engineering and responses out of 108 total students after preprocessing re- 1 Agile Development Methods spectively. Moreover, the total word counts were 1866, 1390 2 Introduction to Requirements and Modeling for Module 7 and Module 8 reflections respectively. We also observe that most student reflections contained between a 3 Requirement Analysis and Modeling sentence or two on average in both module reflections. Fur- 4 Architecture and Modeling thermore, we note that most reflections in our corpus were around a sentence in length. Data Flow Diagrams, Context Diagrams, and 5 UML Diagrams Table 2: Descriptive statistics of the Module 7 and Module Use Case Diagrams & Extracting 8 reflections collected from the undergraduate software engi- 6 neering course. Requirements Cloud based Software engineering, Testing, 7 Module Reflection Reflection 7 Reflection 8 Object-oriented Design Pattern 8 Microservices, feasibility Responses (%) 94 (87.0%) 89 (82.4%) Avg. Word Count 19.4 15.4 9 Reliable programming Avg. Number of 1.4 1.3 10 Final exam Sentences Avg. Words per 11 Final project 14.0 11.5 Sentence Total Words 1866 1390 in Dorodchi et al. [10, 7], each module is typically divided into multiple scaffolds: prep-work to complete before class including reading assignments and videos to watch, in-class 3. APPROACH activities, post-lecture activities, including assignments and Our overall approach is illustrated in Figure 1. First we col- labs, and a reflection at the end of the module. Labs are lect data from an undergraduate course with 108 students. more challenging assignments provided to students which This is described in more detail in section 2. Then, we require hands-on coding. These lab activities are typically perform preprocessing on the data using natural language divided into multiple parts. There are a total of 4 labs in processing (section 3.1). Next, we generate sentence embed- this course, with the first lab beginning in Module 2 and the dings (section 3.2), cluster those embeddings (section 3.3), last lab being introduced in Module 8. and sort the reflections based on clusters for TA’s to view (section 3.4). Data Collection. A survey questionnaire was provided to students within Canvas, the University’s Learning Manage- 3.1 Preprocessing ment System (LMS), at the end of each module to allow stu- Before we generate sentence embeddings from our reflec- dents to reflect on their learning and challenges. We refer to tions dataset, we first preprocess the data by removing any student responses of this questionnaire as student reflections blank, or null, student responses, and also removing any throughout this work. The questions asked of students were: non-breaking spaces which appear in the text. Next, the student responses are compiled and provided to the model for generating sentence embeddings. 1. On a scale of 1 to 5, with 5 being Very Active and 1 being Not Active, how engaged would you rate your group this week? 3.2 Sentence Transformers Figure 1: Illustration of our clustering and sorting approach of short student reflections. Background. Transformer architectures can be computation- reflection 7 and 8 clusters for module reflection 8. ally inefficient when trying to find the most semantically similar pair in sizable collection of sentences. To address 3.4 Sorting of Student Reflections this issue, sentence transformers were developed. Sentence After each student reflection is assigned a cluster, the re- transformers utilize mean pooling which computes the av- flections undergo a sorting process. The goal of the sorting erage of all the word-level vectors in the inputted sentence. process is to group reflections from most similar to least Pooling helps sentence transformers maintain a fixed size similar to assist in the reading process. Cluster distances vector as their output. Sentence transformers then undergo were calculated using the scikit-learn library fit_transform a fine-tuning training process using the SNLI dataset [3] function which computes and transforms the sentence em- containing over 570,000 annotated sentence pairs. The fine- beddings to cluster-distance space. This function uses the tuning process Siamese and triplet networks [26] are utilized euclidean distance formula for calculating the distance be- to compute weights during fine-tuning so that sentence em- tween a student reflection response r and its assigned cluster beddings are optimizing for meaningfulness and can be com- center rc , as follows: pared with cosine-similarity. Working with sentence-level representations make it easier and more efficient for tasks p distance(e(r), rc ) = e(r) · e(r) − (2 ∗ e(r) · rc ) + rc · rc such as computing the semantic similarity of 2 sentences. Sentence transformers reduce computation time of finding where e(r) represents a student response r embedded us- the most similar Quora question from over 50 hours to a ing sentence-transformers into a vector of 768 elements. rc few milliseconds using Transformer architectures [23]. Fur- represents the computed cluster center assigned to r. thermore, Sentence transformers outperform regular trans- formers on several semantic textual similarity tasks [23]. After computation, we sort the reflections using the assigned cluster number to group reflections within the same clus- Approach. We use the sentence-transformers package [23]. ter together. Lastly, we sort the reflections within the same We particularly select the DistilRoBERTa-base-cased cluster using the distance metric in descending order as well. model to get our sentence embeddings. DistilRoBERTa- This way reflections are sorted by most semantically simi- base-cased is a RoBERTa transformer model [16], dis- lar to the cluster center to least semantically similar to the tilled using [25]. The dimension of the embeddings is 768. cluster center. Next, we explore our user study set up and In the embedding process, we take each student response evaluate how well this approach assists in the reading pro- which is typically a sentence in length, and convert it into a cess. vector of 768 floats representing the sentence. These embed- dings are then used to cluster the reflections as described in the next subsection. 4. RESULTS 4.1 Experimental Setup In order to measure the efficacy of clustering in the knowl- 3.3 Clustering edge extraction process, we developed a user study which Our earlier step yields a set of embedded student responses compares the time efficiency of reading through and extract- one set for module 7 reflections and another for module re- ing topics from student reflections in two formats: flection 8. For each set of embedded student responses from our earlier step, we use K-means clustering using the scikit- learn machine learning library [21]. We compute the clus- 1. Unsorted student reflections exported directly from the ter centers for each cluster using the embedded student re- LMS. sponses, hence cluster centers are represented by an embed- 2. Sorted student reflections sorted based on cluster dis- ding vector of the same shape. We also assign each response tances. to a cluster based on the nearest cluster center. The number of clusters was determined using the Silhouette First, the method of the user study will be described, and method [24] for finding the optimal number of clusters. Us- then a summary of the results. Our hypothesis when con- ing the Silhouette method, we generate 4 clusters for module ducting this study was that clustering can help reduce the cognitive load and increase effectiveness and efficiency of Table 4: Normalized time taken to fully extract knowledge knowledge extraction. from all student responses per module reflection. In this user study, four teaching assistants were selected to Module Reflection Unsorted Sorted N read through the student reflections of a Software Engineer- Reflection 7 90.0 15.0 94 ing course. The module 7 and 8 reflections were chosen as minutes minutes the corpora to extract knowledge from, as the TAs had not Reflection 8 121.4 20.9 89 yet read these in particular. minutes minutes Each TA was assigned a reflection and a format. For exam- ple, TA 1 would read and extract topics from Reflection 7 remained consistent, with a slight improvement in compar- unsorted, TA 2 would read and extract topics from Reflec- ison to the unsorted format. Following the portion of the tion 7 clustered/sorted, and so on, as illustrated in Table user study which required TAs to individually extract top- 3. For the TAs which were assigned the clustered/sorted ics from the reflections, they then met afterwards to discuss format, they individually ran the K-Means clustering algo- their similarities and differences in topics. The TAs who an- rithm first without reading any responses before beginning alyzed Reflection 7 extracted the same topics from the stu- the process. dent responses with no differences. During the discussion, the Reflection 7 TAs took turns sharing the topics they had Table 3: Assignment of TAs to specified reflection and format extracted during the user study, and concluded that they type for the knowledge extraction process. were in 100% agreement with the topics coded. Reflection 8, however, had one topic which was extracted in the clustered Module Reflection Unsorted Sorted and sorted reflections and not in the unclustered/unsorted Reflection 7 TA 1 TA 2 reflections. The TAs assigned with Reflection 8 noted that Reflection 8 TA 3 TA 4 this was most likely due to a lack of time to completely an- alyze all unsorted student reflections, hence displaying how time efficiency can also be beneficial to improving the ac- The free-response question used in particular for this study curacy of knowledge extraction if given a time-constraint. was: Despite the improved time efficiency of the clustered and sorted reflection format, no topics were missed. “What was your biggest challenge this past week? We utilize the dimension reduction algorithm UMAP [18] This can include in-class activities, assignments, to visualize the resulting clusters of student reflections as prep work, studying, time management, motiva- shown in Figure 2. The student reflections for Module 7 tion, and so on.” resulted in 4 clusters with 4 major topics including manag- ing workload, motivation and time management, lab work, Each TA individually read through each student’s reflection and group work. The Module 8 student reflections resulted response for this question, extracted any new topics men- in 8 clusters with each cluster containing a challenge in at tioned in the student response, and timed themselves ac- least one of the following categories: Lab work, time man- cordingly for the duration of the process. Once all TAs had agement, studying, motivation, group work, and some reflec- collectively finished, they then met to discuss what topics tions mentioned no challenges whatsoever. Managing work- they found, and compared times and results. load, motivation, studying, and time management relate to the student’s own discerned ability to handle the course- work in general. Lab work and group work were challenges 4.2 Evaluation in which students related their troubles more specifically to After comparing results of this study, we derive that by pro- difficult topics being covered, confusions about instructions, viding instructors with student reflections in a clustered and or trouble with communicating among their groups to com- sorted format, the time needed for knowledge extraction de- plete activities. Students who were in the category of “no creases while maintaining the accuracy of identifying top- challenges” noted that they did not have any difficulties or ics. Reflection 7, with a total of 94 student responses, took confusion during the span of that module. As displayed in 90 minutes to completely read through and extract topics these scatter plots and the major topics described, there are on the unsorted responses, while only requiring 15 minutes overlaps among several of the clusters. This overlap is cre- in the sorted and clustered format. Reflection 8 had simi- ated by the similarities in the students’ wordings. For exam- lar results in which efficiency increased, with a total of 89 ple, two student responses within the “Managing Workload” responses taking approximately 121.4 minutes on the un- cluster of the Module 7 reflection were: sorted format and 20.9 minutes on the clustered and sorted responses. It is important to note that the TA extract- ing knowledge from Reflection 8 unsorted did not complete 1. “My biggest challenge has been not procrastinating my within a 90 minute time frame, thus their results were nor- work.” malized based on how many reflections they did complete. 2. “The biggest challenge this week was working with the These results are provided in Table 4. dash and the dashboard framework.” In addition to the increased efficiency of knowledge extrac- tion with a clustered and sorted format, the topics extracted The first student response was the cluster center with a dis- (a) Module 7 Reflection clusters based on question 2: student chal- (b) Module 8 Reflection clusters based on question 2: student chal- lenges. lenges. Figure 2: UMAP scatter plots visualizing student reflection K-Mean clusters tance of 3.12, and the second student response was one of and cluster short text student reflections and we conduct the farthest points from the cluster center, with a distance an educator-centered evaluation where we assess the direct of 7.05. Therefore, clusters still maintain semantic simi- impact of our approach on teaching assistants’ reading and larities to many of the responses with smaller intracluster analysis time. distances, but contain outliers due to the overlap caused by similar word usages. 6. DISCUSSION & FUTURE WORK In our research, we implement an approach using k-means clustering and sentence-transformers on student reflections 5. RELATED WORK to aid in reducing the labor and time-consumption of man- Reflections are a necessary component in active learning ually analyzing reflections. Our study presents promising courses, as it allows the instructor to track students’ im- preliminary results showing that by clustering student re- pression on the course, activities, and social learning aspects flections based on semantic similarities and sorting by intr- [9]. In Dorodchi et al. [8], student reflections are used in an acluster distance, instructors are able to decrease the time introductory computer science (CS1) course to test its effi- needed to extract topics from the student corpora. How- cacy as a feature to predict early on which students may be ever, our study suffers from several limitations. Firstly, our at-risk of failing. By including student reflection data as a sample size for the user study is very small (N = 4) and feature in a temporal data model, referred to as the student our results may not generalize to different classes, or re- sequence model, the authors were able to increase the accu- flection corpora. Furthermore, teaching assistants read at racy of predicting student outcomes of pass or fail [8]. De- different paces. Our results may not generalize to different spite the advantages of integrating student reflections into a teaching assistants. To address these limitations we intend course model, these benefits require the time-consuming pro- to conduct a user study with a significantly larger pool of cess of manually reading through individual reflections and participants, module reflections, and in multiple courses. In extracting common themes. For this reason, creating an au- addition, we are planning to utilize fuzzy clustering [11] in tomated process to assist instructors is similarly explored in the future version as well. [5]. Chen et al. [5] presents positive results in exploring the usage of topic modeling for analyzing and extracting knowl- Reflections are fundamental for enhancing learning in class- edge from student reflections. In this particular study, the rooms [9], and provides the instructor with instant feedback MALLET toolkit was utilized for the topic modeling pro- on student progress. This study focuses on exploring the cess, and the number of clusters K was manually selected. impact of clustering on student reflections to assist instruc- These methods of knowledge extraction are not only effec- tors in reducing time costs of analysis. In our future work, tive in an academic environment, but is also used in other we plan to integrate our k-means clustering algorithm into applications such as social media mining for COVID-19 re- a dashboard tool for instructors and conduct an expanded lated information. Comparatively to the time-sensitive task user study to further evaluate our approach. The dashboard of analyzing student reflections, clustering can also be used will provide instructors and TAs the functionality to cluster to discover new information from relevant tweets to assist student reflections from the LMS and be guided through the in the decision-making steps that may follow [12]. For this responses. task, Ito et al. [12] and Asgari et al. [2] implement algo- rithms using K-means clustering and sentence embeddings, which both provide positive results in topic extraction. Our study is distinguished from prior works in that, we collect 7. ADDITIONAL AUTHORS [13] K. S. Jones. A statistical interpretation of term 8. REFERENCES specificity and its application in retrieval. Journal of [1] A. Al-Doulat, N. Nur, A. Karduni, A. Benedict, documentation, 1972. E. Al-Hossami, M. L. Maher, W. Dou, M. Dorodchi, [14] H. Li, W. Ding, S. Yang, and Z. Liu. Identifying and X. Niu. Making sense of student success and risk at-risk k-12 students in multimodal online through unsupervised machine learning and interactive environments: A machine learning approach. arXiv storytelling. In International Conference on Artificial preprint arXiv:2003.09670, 2020. Intelligence in Education, pages 3–15. Springer, 2020. [15] Y. Liu and M. Lapata. Text summarization with [2] M. Asgari-Chenaghlu, N. Nikzad-Khasmakhi, and pretrained encoders. arXiv preprint arXiv:1908.08345, S. Minaee. Covid-transformer: Detecting covid-19 2019. trending topics on twitter using universal sentence [16] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, encoder. O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. [3] S. R. Bowman, G. Angeli, C. Potts, and C. D. Roberta: A robustly optimized bert pretraining Manning. A large annotated corpus for learning approach. arXiv preprint arXiv:1907.11692, 2019. natural language inference. In Proceedings of the 2015 [17] K. Mahajan, E. Al-Hossami, and S. Shaikh. Conference on Empirical Methods in Natural Language TeamUNCC@LT-EDI-EACL2021: Hope speech Processing, pages 632–642, Lisbon, Portugal, Sept. detection using transfer learning with transformers. In 2015. Association for Computational Linguistics. Proceedings of the First Workshop on Language [4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, Technology for Equality, Diversity and Inclusion, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, pages 136–142, Kyiv, Apr. 2021. Association for G. Sastry, A. Askell, et al. Language models are Computational Linguistics. few-shot learners. arXiv preprint arXiv:2005.14165, [18] L. McInnes, J. Healy, and J. Melville. Umap: Uniform 2020. manifold approximation and projection for dimension [5] Y. Chen, B. Yu, X. Zhang, and Y. Yu. Topic modeling reduction. arXiv preprint arXiv:1802.03426, 2018. for evaluating students’ reflective writing: A case [19] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and study of pre-service teachers’ journals. In Proceedings J. Dean. Distributed representations of words and of the sixth international conference on learning phrases and their compositionality. arXiv preprint analytics & knowledge, pages 1–5, 2016. arXiv:1310.4546, 2013. [6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. [20] N. Nur, N. Park, M. Dorodchi, W. Dou, M. J. Bert: Pre-training of deep bidirectional transformers Mahzoon, X. Niu, and M. L. Maher. Student network for language understanding. arXiv preprint analysis: a novel way to predict delayed graduation in arXiv:1810.04805, 2018. higher education. In International Conference on [7] M. Dorodchi, E. Al-Hossami, M. Nagahisarchoghaei, Artificial Intelligence in Education, pages 370–382. R. S. Diwadkar, and A. Benedict. Teaching an Springer, 2019. undergraduate software engineering course using [21] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, active learning and open source projects. In 2019 B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, IEEE Frontiers in Education Conference (FIE), pages R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, 1–5. IEEE, 2019. D. Cournapeau, M. Brucher, M. Perrot, and [8] M. Dorodchi, A. Benedict, D. Desai, M. J. Mahzoon, E. Duchesnay. Scikit-learn: Machine learning in S. Macneil, and N. Dehbozorgi. Design and Python. Journal of Machine Learning Research, implementation of an activity-based introductory 12:2825–2830, 2011. computer science course (cs1) with periodic reflections [22] J. Pennington, R. Socher, and C. D. Manning. Glove: validated by learning analytics. 12 2018. Global vectors for word representation. In Proceedings [9] M. Dorodchi, L. Powell, N. Dehbozorgi, and of the 2014 conference on empirical methods in natural A. Benedict. Strategies to Incorporate Active Learning language processing (EMNLP), pages 1532–1543, 2014. Practice in Introductory Courses, pages 20–37. 04 [23] N. Reimers and I. Gurevych. Sentence-bert: Sentence 2020. embeddings using siamese bert-networks. In [10] M. M. Dorodchi, N. Dehbozorgi, A. Benedict, Proceedings of the 2019 Conference on Empirical E. Al-Hossami, and A. Benedict. Scaffolding a Methods in Natural Language Processing. Association team-based active learning course to engage students: for Computational Linguistics, 11 2019. A multidimensional approach. In 2020 ASEE Virtual [24] P. J. Rousseeuw. Silhouettes: A graphical aid to the Annual Conference Content Access. ASEE interpretation and validation of cluster analysis. Conferences, Virtual On line, 2020. Journal of Computational and Applied Mathematics, [11] M. Doroodchi and A. Reza. Implementation of fuzzy 20:53–65, 1987. cluster filter for nonlinear signal and image processing. [25] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. In Proceedings of IEEE 5th International Fuzzy Distilbert, a distilled version of bert: smaller, faster, Systems, volume 3, pages 2117–2122 vol.3, 1996. cheaper and lighter. arXiv preprint arXiv:1910.01108, [12] H. Ito and B. Chakraborty. Social media mining with 2019. dynamic clustering: A case study by covid-19 tweets. [26] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: In 2020 11th International Conference on Awareness A unified embedding for face recognition and Science and Technology (iCAST), pages 1–6. IEEE, clustering. In Proceedings of the IEEE conference on 2020. computer vision and pattern recognition, pages 815–823, 2015. [27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. [28] Z. Zhang, J. Yang, and H. Zhao. Retrospective reader for machine reading comprehension. arXiv preprint arXiv:2001.09694, 2020.