Plagiarism Detection in Algorithms - a Case Study Using Algorithmi António Manso Célio Gonçalo Marques Paulo Santos Techn&Art - Instituto Politécnico de Techn&Art - Instituto Politécnico de Techn&Art - Instituto Politécnico de Tomar, Portugal Tomar, Portugal Tomar, Portugal manso@ipt.pt celiomarques@ipt.pt psantos@ipt.pt Vitor Alencar Degree Programme in Computer Engineering Instituto Politécnico de Tomar, Portugal student 21737@ipt.pt Abstract—Learning to program is crucial in computer Several studies indicate that academic misconduct, and in science degree programmes. For students to gain this skill, they particular plagiarism, is particularly problematic in the need to practise a lot, since programming is a difficult and courses of programming [4]. complex process and practice improves it. The courses Plagiarism is defined as the action of copying someone "Algorithmics" and "Introduction to programming" have else's work or ideas without giving them credit for it. therefore become almost insurmountable barriers with high failure rates. To overcome these barriers, researchers have The existence of the Internet with original contents indexed, sought to find new ways of teaching. For their part, students searchable and cloneable makes plagiarism accessible at the have looked for ways to succeed with less effort by repeatedly distance of a click. On the other hand, the existence of resorting to plagiarism. The need for classes to be taught equipment and instantaneous communication applications remotely due to the pandemic has further aggravated this such as smartphones facilitates the plagiarism of exercises problem. To fight it, we created the Algorithmi tool and with the transmission of answers to questions between equipped it with a plagiarism detection module. This tool students in an irregular way. In a regular classroom context, promotes stand-alone study by allowing self-correction of the these factors can be eliminated or at least diminished through exercises whilst validating authorship. This paper presents a the presence and control of the lecturers. When their presence case study on the application of the plagiarism module to the exercise resolutions submitted by the students in their personal is not possible (as was the case during the confinement repository. Through the analysis of the results obtained we can induced by the COVID-19 outbreak), these factors contribute conclude that there was a high rate of plagiarism. The current to the increase of plagiarised works from the original sources version of the tool needs to be extended with new features, not and among the students. only to become more accurate, but above all to prevent Intensive practice is the best way for students to learn how plagiarism from happening. to program correctly because it improves abstract thinking for complex problem solving. The need to perform many Keywords: Algorithmi, Learning Systems, Programming exercises in conjunction with overcrowded classes means that Languages, Algorithms, tutor, assessment, plagiarism detection validation of the authorship of the exercises cannot be done I. INTRODUCTION manually by the lecturers in charge. This makes plagiarism the fastest and easiest way for students to succeed. Programming is the art of making computers to solve The pandemic outbreak caused by COVID-19 and the problems automatically. For this to happen the programmer need for classes to be held remotely has further aggravated this needs to have knowledge of problem-solving and situation, leading to the need for a tool that allows stand-alone computational logic and master the programming language. A coursework and detects plagiarism. It is in this scenario that task which is “considered difficult, complex and categorised the Algorithmi's plagiarism detection module appears. as part of the seven grand challenges in computing education” This paper presents a case study of the use of the [1]. We are therefore facing a challenge that is not met by most Algorithmi tool in students' answers to programming students in the introductory programming modules, leading to exercises in the course “Introduction to Programming” of the high failure rates [2]. A situation that affects student school year 2019/2020. willingness to participate and makes class management We start by addressing the issue of plagiarism, and then increasingly difficult. Introductory programming modules explain how Algorithmi detects plagiarism. The case study already have many students due to the demand for and the research conclusions are described below. professionals in these areas and school failure further aggravates the problem. II. PLAGIARISM This scenario has led many researchers to look for new According to [5], there are four categories of plagiarism: methodologies, strategies and tools to overcome the problem accidental, where there is no knowledge both of the attitudes [3]. characterised as plagiarism and the understanding of Simultaneously, students also try to find other ways to practices of citation and/or reference practised by the succeed in the module, including plagiarism. different educational institutions; unintentional, where the great amount of information available can, in a certain way, "influence thoughts and the same ideas can arise through Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). spoken or written expressions"; intentional, where the A. Plagiarism Detection in Source Code complete work or part of the work of another person is As already mentioned, plagiarism does not occur only in intentionally copied without mentioning the original author textual documents, or image files, but also in source codes. and; autoplagiarism, where the author uses his/her own According to [10], beginner programmers often send complete work or part of it already published for other plagiarised codes in practical tasks during the introductory purposes without citing the original. programming modules. There are several implications regarding the practice of The possible reasons for the practice of plagiarism as plagiarism, some of these implications in the academic field, described by [11] and [12] are: 1) difficulty level of the according to [6] are: reputation of faculty members and exercise higher than the student's programming level; 2) students, legal issues, financial implications and plagiarised interest of the student in minimising a large amount of the research. In other words, in general, the act of plagiarism is task by plagiarising the task of the colleagues; 3) insufficient considered illegal and can result in severe punishment, such time for the completion of the exercises; 4) fear of failure, 5) as the imposition of fines, restriction of professional practice lack of interest on the part of the student regarding practical in a certain position and even imprisonment. tasks; 6) lack of merit/recognition and 7) inadequate The project Impact of Policies for Plagiarism in Higher resources such as hardware, software, library, teaching staff, Education Across Europe (IPPHEAE), which took place among others. from October 2010 to September 2013, makes a comparison There are several approaches in the literature regarding of policies for academic integrity in Higher Education in the the detection of plagiarism, varying in implementation style European Union [7]. In this study, a "maturity level" was and source code. Above all, it is important to emphasise calculated for each country using an Academic Integrity initially the difference between detecting plagiarism in text Maturity Model (AIMM). Portugal was ranked 12.79/36, files and in files derived from codes and algorithms. 18th out of the 27 countries involved in the study. In addition, According to [13] the verification in text files consists it was found that: 1) Many of the faculty members and essentially in looking for direct similarities, verifying the students who responded to the survey did not understand very similarity "line by line" in relation to some other text. well what is considered plagiarism; 2) few Portuguese Text file detectors are not effective for verifying Universities use software to verify the originality of the work similarities between source files because this approach submitted by students; and 3) 77% of the faculty members cannot verify changes made to the copied code such as and 79% of the students interviewed showed interest in changing variables and possible changes to the structure of having more training and information to avoid plagiarism and the document [13]. When checking for plagiarism in files academic fraud. defined as "source code", which are described by languages The scenario where students and faculty have no and code structures, they are notoriously different and need knowledge about conduct that may constitute plagiarism or to undergo a modelling process in order to be able to compare fraud is worrying, because in addition to the legal them with other source files. implications, plagiarism can also affect student learning, This modelling process usually follows some pre-defined considering that the simple fact of "copying and pasting" steps that are established based on each objective and need. inhibits critical thinking and the development of logical These factors may vary depending on the wide variety of reasoning. According to [8], in students' perception, the plagiarism detectors available on the market. The most practice of plagiarism occurs because: they think that their common practices for plagiarising source code files imply practice of plagiarism will not be detected; it is very easy to changing the code. Such practices range from changing the copy and paste information from the Internet; ignorance name of variables to inserting redundant code snippets. about how to quote and provide references, the objective is to Figure 1 illustrates some of the most common techniques in complete the coursework rather than focusing on learning; source code plagiarism. In literature, these practices are not being able to express other people's ideas and other characterised by "blinding methods", where the plagiarist coursework in their own words; not being able to deal with aims to hide code snippets that are originally from another the workload; not being aware of the penalties related to person. plagiarism; not having control on the part of the lecturer regarding the detection of plagiarism; among other causes. In addition, [9] identified possible negative emotions that can predict plagiarism conduct such as stress, depression or anxiety. The most common methods of plagiarism according to [5], are: copy and paste a certain content word by word, plagiarise an idea, reaffirm the same content using synonyms or changing the order of sentences, artistic plagiarism, not citing or referencing the original work, not using quotation marks (“ ”) when the content used is exactly the same as the original content, use of incorrect or non-existent sources and references, translation of content without reference to the Figure 1 - Source code obfuscation techniques original work and, finally, plagiarism of source code, the main focus of this work. Determining the level of similarity between two source codes is a difficult task and several automatic tools have been developed to help lecturers in the arduous task of detecting The IDE not only shows and allows editing the cases of plagiarism. Examples of these software tools are flowchart and pseudocode, but also allows the display in MOSS (Measure of Software Similarity) [14] and Jplag [15]. several programming languages (Python, Java, C, C++, C#, These tools use advanced techniques to compare small PHP and JavaScript), Figure 4. fragments of source code at the syntactic level, calculating The server component is available through registration the fingerprint of a code segment, using the winnowing and authentication in the same domain. The server used was algorithm. These tools implement mechanisms that allow the implemented with open-source technologies Apache, detection of plagiarism using the obfuscation techniques MySQL and PHP that allow the provision of various features presented in Figure 1. to the information system. This server provides services that access a common database: Backoffice - a web application III. PLAGIARISM DETECTION IN ALGORITHMI used by faculty, Frontoffice - a web application where Algorithmi is an information system to support students can find solved problems and their score, and an API learning programming consisting of a multi-platform - a set of REST web services ttha includes plagiarism application (IDE) designed to help students take their first checker. steps in programming and a server which provides various services that can be accessed directly from a web browser, Figure 2. Figure 2 - Information system architecture of Algorithmi Figure 4 – Automatic translation of the algorithm to Java language and Javascript language The application is available for free at www.algorithmi.ipt.pt, and allows students to encode the The backoffice component is used by faculty to create and algorithms using flowcharts or pseudocodes. The application make available programming exercises that can be accessed internally uses a marker-based language, GAL (Generic through the IDE, provided that the student is authenticated in Algorithm Language) [16], which is translated into the the information system. The programming exercises can be student's natural language for editing and viewing. This way corrected automatically and, in case of errors, the IDE the algorithm can be viewed and edited in different natural provides a description of the errors made by students. This languages such as Portuguese, English or Chinese. feature is particularly important in order to promote The GAL language supports the definition of simple and autonomous learning by students, as the IDE helps students indexed variables of several types: integer, real, logical and verify that their algorithms are correct, and provides clues for text; it has I/O console instructions; it allows to control the their resolution. execution flow through decisions and iterations and supports After solving the exercises, the students submit them to the definition of subalgorithms which can be recursive. This the server that stores them in the student's personal set of instructions allows coding complex algorithms through repository. The student can visualise the resolution of their a set of tools designed for teaching and learning algorithms. exercises in a web environment using the same tools as the The algorithm can be edited in both languages and its IDE: flowchart, pseudocodes or programming languages; and visualisation can be displayed simultaneously as shown in can also execute them by translating them into Javascript Figure 3. embedded in a web page. Figure 3 - Simultaneous editing and display of an algorithm. exercise and with aggregate information for each worksheet and for each student. IV. CASE STUDY This section analyses the application of the plagiarism module to the exercises submitted in the course “Introduction to Programming” of the CteSP programme in Information Systems Technologies and Programming offered at the Escola Superior de Tecnologia de Tomar - Instituto Politécnico de Tomar. The course covers the basic programming concepts: variables, operators, functions and control flow. One of the assessment components of the course is made based on the portfolio of programming exercises submitted by students. The exercises are divided into 15 worksheets with a total of 480 different exercises which cover a) b) the various subject contents. In each worksheet, the exercises have difficulty levels: Figure 5 - Programming exercise: a) Problem o Demonstration - Solved exercises and problems that Statement; b) Automatic correction of the exercise by the are provided to students to demonstrate a new concept. IDE. These exercises are crucial for stand-alone learning and students can edit them in order to check the A. Plagiarism control in Algorithmi outcome of these changes. Algorithmi encourages stand-alone programming through o Easy - Exercises that are easy to solve and are usually the exercise assessment tool and the tutorial system that variants of demonstration exercises. identifies errors. Problem repository is one of the forms of o Normal - Medium-difficulty exercises assessment that can be used to assign a score to students, and o Difficult - High-difficulty exercises where algorithmic the provision of a roll of honour listing the names of the techniques are explored, especially in the limit cases. students with the highest scores. Gamification is a key o Challenging - Challenging exercises whose resolution component in learning environments to maintain students’ depends on the combination of several complex motivation to solve more complex exercises [16] [17]. algorithmic techniques The pressure to get a good score in the repository and the The greater the difficulty of resolution, the more points opportunity to belong to the roll of honour make students the student earns, improving his ongoing assessment plagiarise some coursework. The IDE provides some tools component and consequent rise in the roll of honour position. that limit source code plagiarism. The exercises provided by Plagiarism in demonstration exercises is almost 100%. The the server to the IDE are marked with the username which lecturer-in-charge provides the resolution, the students can prevents other students from taking them as their own. The then execute and submit it and consequently the resolutions. exercise delivery and assessment module is only available for Figure 6 gives the plagiarism prevalence rate of an easy authenticated users and the programs of that user. The other exercise solved by 18 identified students from S01 to S018, tool that prevents plagiarism is the cut and paste tool. Pasting and whose plagiarism indexes were provided by the from software programmes within and outside Algorithmi are plagiarism detection module. The numerical information is only allowed within software programmes of the same user. presented in tabular form with the plagiarism prevalence rate This way we prevent the source code from being shared between each of the submissions for the same exercise, between users. together with a colour that ranges between green and red for With the help of these two features we can ensure that the easier reading of the results. By the empirical analysis of the algorithm was built by the authenticated user, but we can not preliminary results we consider that two algorithms are ensure that it was not copied from another user. plagiarised if their similarity is equal to or higher than 90%. In order to inform the lecturer of this type of plagiarism, This information is presented in the last row and in the last we introduced the plagiarism detection module in the column of Figure 6 and is useful to verify the degree of information system to compare the similarity between the originality of the resolution of the exercise. In Figure 6 we resolutions of the students submitted in their repository. The can see that there are 10 resolutions that the system identifies module was developed using JPlag (sources available at as plagiarised and one that differs from all the others. This https://github.com/jplag/jplag) which communicates with the plagiarism prevalence rate is natural, since it is an easy information system through a web service. Due to the exercise based on a similar demonstration problem. Algorithmi's ability to convert the algorithm into various programming languages, we use the Java language to compare source codes and detect plagiarism. The service receives as passing parameter two programmes in Java and returns a number in the interval [0.100] that represents the plagiarism prevalence rate. The results are presented through the backoffice with the results of the plagiarism prevalence rate for each programming Alunos S01 S02 S03 S04 S05 S06 S07 S08 S09 S10 S11 S12 S13 S14 S15 S16 S17 S18 >=90% S01 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 Alunos S01 S02 S03 S04 S05 S06 S07 S08 S09 S10 >=90% S02 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S03 76 76 100 76 100 76 69 76 100 76 76 98 100 98 76 76 100 76 7 S01 100 51 70 60 51 52 84 51 100 41 2 S04 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S05 76 76 100 76 100 76 69 76 100 76 76 98 100 98 76 76 100 76 7 S02 51 100 51 46 100 81 49 60 51 61 2 S06 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S07 72 72 69 72 69 72 100 72 69 72 72 68 69 68 72 72 69 72 1 S03 70 51 100 60 51 52 84 51 100 41 2 S08 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S09 7 76 76 100 76 100 76 69 76 100 76 76 98 100 98 76 76 100 76 S04 60 46 60 100 46 47 58 46 60 53 1 S10 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S11 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S12 74 74 98 74 98 74 68 74 98 74 74 100 98 100 74 74 98 74 7 S05 51 100 51 46 100 81 49 100 51 61 3 S13 76 76 100 76 100 76 69 76 100 76 76 98 100 98 76 76 100 76 7 S14 74 74 98 74 98 74 68 74 98 74 74 100 98 100 74 74 98 74 7 S06 52 81 52 47 81 100 50 81 52 63 1 S15 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S16 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S07 84 49 84 58 49 50 100 49 84 39 1 S17 76 76 100 76 100 76 69 76 100 76 76 98 100 98 76 76 100 76 7 S18 100 100 76 100 76 100 72 100 76 100 100 74 76 74 100 100 76 100 10 S08 51 60 51 46 100 81 49 100 51 61 2 >=90% 10 10 7 10 7 10 1 10 7 10 10 7 7 7 10 10 7 10 8,3 S09 3 Figure 6 – Plagiarism detection results in an easy exercise 100 51 100 60 51 52 84 51 100 41 S10 41 61 41 53 61 63 39 61 41 100 1 Alunos S01 S02 S03 S04 S05 S06 S07 S08 S09 S10 S11 S12 S13 S14 S15 S16 S17 S18 >=90% S01 100 83 90 75 75 64 49 99 68 100 83 92 93 83 83 100 41 54 7 >=90% 2 2 2 1 3 1 1 2 3 1 1,8 S02 83 100 63 69 69 54 47 52 41 83 100 91 72 100 100 83 59 59 5 S03 90 63 100 58 58 50 50 89 68 90 63 71 72 63 63 90 41 55 4 Figure 8 – Plagiarism detection results in an assessed S04 75 69 58 100 100 64 38 74 50 75 69 76 73 69 69 75 41 42 2 S05 75 69 58 100 100 64 38 74 50 75 69 76 73 69 69 75 41 42 2 practical test. S06 64 78 50 64 64 100 69 74 46 64 78 78 72 78 78 64 57 49 1 S07 49 47 50 38 38 69 100 52 67 49 47 46 47 47 47 49 63 77 1 S08 99 52 89 74 74 74 52 100 69 99 52 89 92 52 52 99 44 52 5 Student portfolios consists of more than 7,000 exercises S09 68 41 68 50 50 46 67 69 100 68 41 47 64 41 41 68 53 91 2 S10 100 83 90 75 75 64 49 99 68 100 83 92 93 83 83 100 41 54 7 evaluated by the Algorithmi module to obtain a grade and by S11 5 S12 83 92 100 91 63 71 69 76 69 76 54 78 47 46 52 89 41 47 83 92 100 91 91 100 72 81 100 100 91 91 83 92 59 61 59 66 8 the plagiarism module to obtain the similarity index. S13 93 72 72 73 73 72 47 92 64 93 72 81 100 72 72 93 39 52 5 Figure 9 gives an overview of average plagiarism in each S14 83 100 63 69 69 54 47 52 41 83 100 91 72 100 100 83 59 59 5 S15 83 100 63 69 69 54 47 52 41 83 100 91 72 100 100 83 59 59 5 student's exercises and the percentage of exercises solved. S16 100 83 90 75 75 64 49 99 68 100 83 92 93 83 83 100 41 54 7 S17 41 59 41 41 41 57 63 44 53 41 59 61 39 59 59 41 100 55 1 The overwhelming majority of students have solved more S18 >=90% 7 54 59 5 55 4 42 2 42 2 49 1 77 1 52 5 91 2 54 7 59 5 66 8 52 5 59 5 59 5 54 7 55 1 100 2 2 4,1 than 50% of the proposed exercises and more than half have Figure 7 – Plagiarism detection results in a difficult solved more than 90%. This statistic shows the work and exercise commitment of the students in solving the exercises and Figure 7 gives plagiarism detection results in a difficult because "you only learn to program by programming" the exercise. Due to the complexity of the algorithm that solves result was reflected in the score of the repository. the exercise, the cases identified as plagiarism will be really Another important conclusion is that the plagiarism plagiarism. In that figure we can identify a set of students who prevalence rate varies according to the number of exercises have a degree of similarity of 100% and a set of unique solved by the students. Students with fewer solved exercises resolutions. tend to plagiarise more, due to their learning difficulties. The previous figures present various information about the rate of plagiarism of worksheet exercises. The course has Since Algorithmi doesn't let you make direct copies and a strong component of stand-alone work and the great deliver them as your own, plagiarising takes work and majority of the exercises are completed outside the contact students replace the work of thinking and creating with the hours, however the students continue to maintain contact with work of copyists that is far less rewarding. each other. Since Algorithmi does not allow you to make full copies of exercises, even if students exchange solutions with Average levels of plagiarism have always remained high, in each other, they will have to edit the algorithms themselves. the order of 80%, which suggests a high rate of plagiarism. For these reasons, the plagiarism rates of the exercises in the worksheets are high. Figure 8 gives the plagiarism prevalence rate in an assessed practical test. Assessed practical tests are done individually during the contact hours and in the presence of the lecturer. The class is divided in shifts to increase the physical distance between students and the time for solving the exercises is limited. From the results presented in Figure 8 we can see that the plagiarism prevalence rate has decreased, however we can see that there are at least three resolutions that the system has identified as 100% plagiarism. Figure 9 – Student portfolios: percentage of exercises solved and rate of plagiarism. V. CONCLUSIONS This paper presents the plagiarism detection module introduced in the information system of Algorithmi. This work was carried out with the support of a student of the Computer Engineering degree as part of his final project and was designed to increase the effectiveness of this system for [4] Jian, H. L., Sandnes, F. E., Huang, Y. P., Cai, L., & Law, K. M. Y. (2008). On students' strategy-preferences for managing difficult course work. IEEE teaching/learning computer programming. Transactions on Education, 51(2), 157-165. This module was applied to the course which was taught in the first semester of the school year 2019/2020 to [5] H. Maurer, F. Kappe e B. Zaka, Plagiarism - A Survey, Journal of Universal Computer Science, 2006, 12(8), pp. 1050-1084. Available: check the rates of plagiarism in the students repository. http://jucs.org/jucs_12_8/plagiarism_a_survey/jucs_12_08_1050_1084_ma It was found that the rates of plagiarism are high, urer.pdf especially in the worksheets completed outside contact hours. [6] O. P. Rajasree, A. H. Mangala e U. T. Sunil Kumar, An Overview of In assessed practical exercises, whose objective is to assess Plagiarism and Intellectual Property Issues, National Conference on knowledge, the plagiarism rates are significantly lower. This Academic Libraries in E-learning Environment: Role and Prospect, 2020, is because the exercises are more difficult, the students have ISSN: 0474-9030, 68. a limited time for their resolution and were solved during [7] I. Glendinning, T. Foltynek, C. Demoliou, K. Joswik, L. Stabingis e A. contact hours. Nevertheless, the system detected several Stulginkis, Comparison of policies for Academic Integrity in Higher cases of plagiarism. Education across the European Union, 2013. With the COVID-19 forced confinement of students and [8] I. Glendinning, K. Joswik e A. Michałowska-Dutkiewicz, Plagiarism staff, in which the majority of teaching was delivered Policies in Portugal, full report , 2014. remotely, the module now developed has assumed greater [9] I. K. Tindall e G. J, Curtis, Negative Emotionality Predicts Attitudes significance. The work developed corresponded to Toward Plagiarism, Journal of Academic Ethics, 2020, 18, pp. 89-102. expectations and above all opened new horizons that will be [10] Vogts, D. Plagiarising of source code by novice programmers a “cry for explored in future versions. help”? 2009 Annual Research Conference of the South African Institute of In the future, we intend to apply the online module where Computer Scientists and Information Technologists, 141-149. the system registers and informs the student about the [11] Vamplew, P., & Dermoudy, J. An anti-plagiarism editor for software originality of the submitted code. In the offline version, development curses. 7th Australasian Computing Education Conference. whose results we present here, we only know which exercises Retrieved March 21, 2007, from http:// are plagiarised, but we do not know the original source. With crpit.com/confpapers/CRPITV42Vamplew.pdf the online version, the first exercise to be submitted has a [12] A. G. Liaqat & A. Ahmad. Plagiarism Detection in Java Code. Linnaeus maximum degree of originality while copies submitted later University. School of Computer Science, Physics and Mathematics. Retrieved June 26, 2011. will see their originality diminished. This way we intend to stimulate the development of different algorithms for the [13] Sheard, Judy & Dick, Martin & Markham, Selby & MacDonald, Ian & same problem, increasing the gamification already present in Walsh, Meaghan. (2002). Cheating and plagiarism: Perceptions and practices of first year IT students. ACM Sigcse Bulletin. 34. 183-187. Algorithmi. 10.1145/637610.544468. The fact that students realise that the system has a system that detects plagiarism may serve as a deterrent for them to [14] Schleimer ,S.; Wilkerson,S., Aiken, A. - Winnowing: local algorithms for document Fingerprinting. In Proceedings of the 2003 ACM SIGMOD avoid less ethical behaviour and channel their energy to the international conference on Management of data. ACM, 76–85, 2003 development of algorithmic reasoning, because in Algorithmi [15] Prechelt, L, Malpohl, G, Philippsen , M. Finding plagiarisms among a plagiarism also takes work. set of programs with JPlag. J. UCS 8, 11 (2002), 1016, . 2002 [16] Manso, A., Marques, C.G., Santos, P., Lopes, L. & Guedes, R. (2019). REFERENCES Algorithmi: Bridging the Algorithms to Natural and Programming Languages., in 15th China-Europe International Symposium on Software [1] Derus, S., & Ali, A. Z. M. (2012). Difficulties in learning programming: Engineering Education. Lisbon, Portugal, May 30-31, 2019 Views of Students, In 1st International Conference on Current Issues in Education (ICCIE 2012). [16] Manso, A., Marques, C.G., Santos, P. (2019). Algorithmi IDE – Integrated learning environment for the teaching and learning algorithms, in [2] Butler, M. & Morgan, M, (2007). “Learning challenges faced by novice 21º International Symposium on Computers in Education (SIIE). Tomar, programming students studying high level and low feedback concepts”, Portugal, 2019 ASCILATE 2007 Singapore, pp. 99-107. [17] Manso, A., Marques, C.G., Santos, P. (2019). Teaching and Learning [3] Tan, P.-H., Ting, C-Y, Ling, S-W (2009). Learning Difficulties in How to Program with Algorithmi., in 21º International Symposium on Programming Courses: Undergraduates' Perspective and Perception, In Computers in Education (SIIE). Tomar, Portugal, 2019 Computer Technology and Development (ICCTD '09).