Teaching image processing algorithms is a widely interesting and easily accepted topic by students. The mentioned algorithms offer initial visual results, a lot of space for improvements, personal initiative, and the opportunity for continuous work. This paper presents our experience in teaching image processing algorithm implementation with different approaches, emphasizing concepts of resource awareness. The programming routines explained to the students start from a managed environment with and without dedicated classes and packages, through the unsafe code execution up to native code integration. The paper presents the comparison of time utilization, programming effort, and the level of the concept adoption by the students. The analysis was based on the student projects uploaded to the learning management system in the period from 2018 to 2024. The results are used to adjust the course structure and better adopt resource awareness-related concepts. The percentage of projects entirely based on more effective approaches rose from 44% in 2018 to 80% in 2023 and 2024. During the monitored period, the overall average execution time of the benchmark algorithms was reduced closely to one-third compared to the results from 2018, which follows the shift towards more effective approaches. In this way, we tend to say that it is important to point out all the technology benefits and shortcomings and to encourage students to try to find more effective ways to solve time and resource-consuming problems.
Implementation of image processing algorithms is considered a remarkably interesting part of computer science [1]. Many students and programmers genuinely like this topic since it gives results that can be verified quickly and visually.
To learn about image processing, a variety of sources are available through different books, papers, and tutorials. Looking at Google Scholar, it could be realized that standard learning resources like [2] and [3] are cited several thousand times. Furthermore, the topic offers particularly good ground for students’ further development and research.
Teaching image processing itself, is also an interesting challenge. It is important to find a suitable approach in terms of the used technology, offered frameworks, and teaching methods. All this should consider previous students’ knowledge and experience. The result should be a well-defined engineering course and a properly suited set of requirements for the projects that should be implemented by the students.
One of the well-known approaches, that we used as the starting point, is to create an environment with basic methods and interfaces that will allow students to implement their algorithms as plugins [4]. This methodology typically favors technologies that enable rapid implementation, such as Java or C#, with the objective being to grasp complex algorithms. In our scenario, we present students with a project that includes fundamental functionalities and an application framework, yet they retain the liberty to develop their applications from the start.
Since we teach the course in the eighth (final) semester of bachelor studies, we tend to move the focus of the subject to more efficient programming, while basing the course outline on the standard image processing algorithms that could be practically used, as 0000-0003-4998-2036 (P. Rajković); 0000-0003-1198-0174 (D .Janković);
Copyright © 2024 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
in [
On the other hand, the image processing algorithms
are interesting from the point of view of execution
optimization and general resource awareness, using
different techniques and approaches [
In this study, emphasis is placed on examining the
students’ execution of image processing algorithms and
their propensity to adopt various resource-aware
strategies to enhance performance in their projects.
Besides the execution of complex algorithms constitutes
a component of the curriculum at the bachelor’s,
master’s, and doctoral study levels. This analysis
concentrates on a cohort of final-year bachelor’s degree
students who undertake image processing algorithm
implementation as part of their Multimedia Systems
course. [
To implement some operations on digital images, they must be stored in the form of a matrix of pixels [1]. The operation should run through the matrix and perform some calculations for each pixel. To describe the importance of resource awareness in image processing, let us analyze the processing requirements for the simple invert operation [3]. Invert operation is replacing every byte in the picture with its complementary value. I.e., if one byte in the original picture has a value of 30, in the inverted picture it will have a value of 225. The inverted value is calculated as the difference between the maximal value for a byte and the original value.
If the image, where we perform the invert operation,
has a resolution of 1000x1000 pixels, and three bytes per
pixel (standard 24bRGB format [
If we observe today's image resolution, which is far more than one million pixels, it is obvious that image processing must be implemented carefully and effectively. Looking at the side of the story related to the execution speed, the obvious direction is to go towards the implementation in the programming environment closest to the processor and operation system’s core.
Unfortunately, such environments are not usually user-friendly and not convenient for the one-semester course teaching where students must finish their tasks and learn as much as possible. On the other hand, the environments based on the execution of virtual machines, such are Java and .NET offer high programming flexibility and user-friendly interface which makes the work on the project much faster. Unfortunately, the execution speed in such environments is much slower than with native code.
For this reason, we choose one possible approach
that could bring the best of both worlds, with the
technology that offers a combination of both execution
speed and fast developments, in the parts where each of
the approaches has its value [
At the same time, we do not post any limitations to the students for the technology choice, but advocate implementation approaches that result in the code that executes faster. However, in this paper, four different implementation approaches, which are presented to the students during the course, are compared by the execution and development speed, with a set of guidelines on how to use them.
The mentioned approaches are used by the students for the implementation of various categories of imageprocessing algorithms, as part of their projects. Data collected over several years are examined to identify the student’s responses to the use of the different implementation approaches.
The study shows that students are willing to adopt implementation approaches that are more complex, from the point of view of the implementation, up to some point, for the benefit of faster code execution.
It should be noted that each student that attended our course, hails from the Department of Computer Science and is presently in their final semester. Throughout their academic journey, they have uniformly completed a foundational curriculum encompassing algorithms, mathematics, as well as programming environments and technologies. Consequently, there exists no subset of students possessing a disproportionate advantage in terms of prior knowledge. Prior to commencing our course, they have garnered experience with programming languages such as C++, Java/C#, in addition to various Web and Mobile technologies.
The example project environment is set up to
Microsoft .NET Windows Forms application [
Like Java, .NET is the environment that runs on a
process virtual machine [
To overcome this issue, the application requires a
higher execution speed, the .NET framework offers the
concept of unsafe code execution [
Also, .NET offers the possibility to directly include libraries written in native code, as in any other programming environment, which is supposed to run as fast as possible. To support lectures, the following implementation modes are presented to the students. Namely:
unsafe block, which runs through the managed environment.
Processing based purely on a native code, written, and evaluated in an external execution environment, and then brought to the managed environment using COM (Component Object Model) interface and runtime-capable wrapper.
As presented in Figure 2, the source code, crafted in .NET languages such as C#, is initially translated into intermediate or bytecode. Subsequently, this bytecode is compiled into the final executable. In contrast, unsafe code bypasses the intermediate code stage and is compiled directly into the executable. Additionally, integrated calls to native libraries are executed directly in the executable code, leveraging the capabilities provided by the COM interface.
The curriculum is delivered over the spring semester, spanning 14 working weeks, and encompasses a variety of topics related to image processing. In the initial stages, students become acquainted with various color models, image file formats, and techniques for reducing image size, including downscaling and compression methods.
Subsequently, the course curriculum introduces
students to a range of image processing algorithms,
including fundamental filters (such as brightness,
grayscale, contrast, gamma correction, and others),
dithering techniques, convolution, and displacement
filters. Furthermore, advanced subjects are explored,
encompassing histogram-based methods and intricate
filters like the Kuwahara filter. [
When teaching image processing algorithms we focus on three major points – algorithm construction/correctness, variants and similar filters, and execution efficiency. In most cases, execution efficiency is more important than memory use, since many algorithms are built around additional data structures which cannot be avoided during the implementation.
As mentioned, the focus of our course is execution analysis. Since we expect that students implement algorithms efficiently, from the algorithm flow’s point of view, we display the differences in the execution depending on the chosen technology.
The students are then able to measure in the demo filters differences in the execution speed between the implementation in different technologies and directly experience trade-offs between comfort programming environments (such as C# and Java), and execution speed in native code implementation (C++).
Since the execution time is one of the most important factors in the eventual grade (14 out of 30 points), students could decide whether pays off if they invest more time in a faster solution (Table 1).
Over the years, we slightly adapted the project structure, taking into consideration results from the previous years. The current project structure consists of: Image processing application that can load standard file formats and display changes after applying image transformation. Defining own file format by implementing down sampling and compression Implementation of one basic filter Implementation of one dithering method (like convolution filters) Implementation of one displacement filter Additional advanced filter implementation During this time, we kept the project complexity requirements at the same level. The basic requirement is to implement a user-friendly application that will display, at any point, original and changed pictures, and to allow the user to save the effect of transformation either to some standard or to a custom file format.
From year to year, we are changing requirements in terms of classes of implemented algorithms as well as the definition of the file format. The projects from one year exclude combinations of algorithms implemented in previous years, but the general requirements remain at the same level. For example, this year (2024) the students had to convert RGB pictures to YUV models and then apply some downsample scheme as in JPEG (4:2:0, or 4:1:1, etc.). After downsample, they had to implement some of the dictionary based lossless compression algorithms like Huffman or Shannon-Fano. Table 2 shows the requirements for filters in three different years.
This year students had to implement contrast as the basic filter, where different variants will be applied to different group of students. Similarly, for the convolution style filter, dithering was the theme for 2024, where each group should implement its specific variant (like Jarvis, Stucki, Sierra, etc.).
When we look in the Table 1, each project element comes in two to five variants, giving us more combinations than the students, which means that every student will have a unique project. I.e., in 2024, we had 4 downsampling variants, 2 compression algorithms, 3 contrast variants, 4 dithering filters, 3 variants of displacement filter, and 2 different advanced topics, which makes 576 different variants for less than 100 students.
The students upload their projects in Moodle (until 2020) and Teams (2021 onwards). After that, we execute them in the referent environment and with a referent set of images and assign points for each project item. We evaluate both algorithm correctness and execution speed. In all these years evaluation has been done only by the authors of these papers, which, we believe, ensured the same level of assessment. In this work, we are focused only on execution time. Since the student applications execute a single algorithm at a time, memory usage and scalability are out of the scope of their projects. In the same semester, they have different course (Distributed systems) where the main aim is on scalability and robustness).
Raw data processing is the basic way to process images (Figure 3). The programmer should load an array of bytes from the file and transform them into a meaningful data structure. The complexity of this task depends on the file type. For example, uncompressed files like bitmaps could be directly loaded and converted, while compressed files, like JPEGs and PNGs, must be significantly processed.
For bitmaps [
The implementation based on raw data processing is suitable only for plain bitmap files since they contain directly stored pixel-related data. This approach requires no additional data structures and classes and could be implemented using the programming language basics only. In this sense, it requires a bit more organization of the processing and it could be suitable for the filters with simpler implementation. This approach could be used in any programming language, and if properly implemented, could give excellent execution performance.
For the most effective use, the framework classes could
be utilized. The most important class for image
processing is the class Bitmap inherited from the more
common class Image [
This class offers the easiest, from the programming point of view, approach. The programmer loads a picture of any supported type using the Load method. Once, the image is loaded and converted to a bitmap object, each pixel can be accessed by GetPixel method.
This method returns an instance of type Color with Red, Blue, Green, and Alpha components. These values could be then processed, and new color values now could be written back to the Bitmap object. From the logical point of view, the programmer must focus only on the implementation of processing algorithms, without the need to take care or pay attention to any other task in the scope of image processing.
On the other hand, this approach runs fully under managed code, and it is the slowest way of implementation. One can say that the programming comfort is paid by the slowest runtime.
Development using the provided framework classes
[
The calls of the methods that should get or set the pixel value must go through the execution virtual machine and must ensure necessary locking mechanisms each time. The recommendation for this approach is to be used in the initial stage of the projects where the students are setting up their environment and learning the process of image processing itself. This approach could be used if the expected size of the picture will not exceed certain limits, and if it will be used for previews since they could be easily integrated into Web routines and technologies such as Blazor.
There are two options to use unsafe/native/nonmanaged code. First, the programmer could write a method in the native language library and then include the library in the project (Figure 5). The problem with this approach is that debugging of such code should be done in some programming tool, different from the environment where the main application is written.
It is important to point out that such an approach is not a problem per se during the development of real software systems, but for student projects whose duration is very limited, and where students have to cope concurrently with the additional tasks, learning to use multiple environments could be a challenge up to some point.
To have the best of both worlds – managed and
unmanaged code, the students are encouraged to use
unsafe blocks in the Windows Forms application [
Working in the unsafe block is close, if not equal to writing code in the native, C++ environment. The programmers have under their disposal, complete pointer arithmetic with the additional requirements to transform data types from .NET to native classes, and to take care of garbage collection.
The implementation of the invert operation in an
unsafe environment is displayed in Figure 6. To support
the coding, programmers could use BitmapData [
The approach with an unsafe code execution is the fastest way to perform image processing from the managed code environment. It is up to some percent slower combined with direct native applications, but the comfort of the integrated development seems justified as the acceptable price.
As can be seen in the previous section .NET environment offers adequate transformation classes that help in moving the execution context from managed to unmanaged. Keeping in mind that the application development, in the case of the student project, relates to tight deadlines such an approach proved its value.
As it could be seen each of the approaches has its benefits and drawbacks. While raw data processing requires no additional libraries and frameworks, it requires more attention to organize code, and it is not useful for complex image types.
Relying only on the framework based on managed code will speed up the development process, at the price of the slowest execution. The integration of externally developed algorithms brought the fastest execution, but the development must be split between multiple projects, development tools, and programming languages. It requires the highest amount of time for development.
The approach based on the unsafe code seems like a
promising approach for student projects focused on
image processing algorithms. It offers performance close
to the native code, seamless integration of the native
code in the managed environment, and higher memory
efficiency since it enables precise memory management.
The drawbacks of this approach are the same as with the
native code execution - higher potential for bugs and
higher complexity for maintenance [
For the application of the mentioned approaches, the students have the task of implementing several image filters from the various categories. To make the most convenient test cases, three diverse types of filters are checked:
managed to native code.
Convolution filters – the filters where the
values of neighboring pixels affect the
currently processed pixel. Compared to basic
filters they require significantly more
processing, but a similar amount of memory
Displacement filters – require additional data
structure which needs a comparable amount of
memory as the processed image. Compared to
basic filters they require more memory and
more processing operations
As has been mentioned, basic filters are those that
require one pass through the matrix of pixels with an
optional parameter transformation. For our evaluation,
we choose a contrast filter (Figure 8), which is an
example of a common filter with many specific variants
depending on the area of the application [
Unlike the basic filters, the convolution filters use the auxiliary matrix structure, usually called the kernel. The kernel is a matrix of small size, in most cases 3x3, and is set up with predefined parameters. In addition, the convolution filter has two more values - factor and offset. Some of the example of convolution matrices are presented in Figure 9.
The processing works in a way that the submatrix, of the same size as a kernel, from the picture, should be extracted. The value of each pixel from the extracted matrix is multiplied by the corresponding parameter in the kernel. All these products are then summed up together, divided by the value provided for factor, and on top of this value is added offset.
The values in the matrix are important for the filtering operation itself, while the values for factor and offset are used to normalize the sum of products into the required value range. In our case, these are the values that could be stored in one byte (0 to 255).
The number of operations needed to perform the convolution filter is significantly higher than with basic filters. We can assume that during basic filter execution, the number of performed steps is the multiplication of the image resolution, for the convolution filters it is a multiplication of the image resolution and number of elementary operations connected with a chosen kernel matrix. For example, the execution of a sharpened filter (Figure 10) will execute ten times (nine multiplication and division with the factor value) more operations than the execution of a basic invert.
Displacement filters are based on a pre-built transformation matrix which is the same size as the targeting picture (Figure 11). Their execution is split into setup and execution phases. In the setup phase, the transformation matrix is created using some algorithm, and in the execution phase, the values from the transformation matrix are combined with the pixel values from the picture to achieve the desired effect.
We have been teaching courses that partly cover image processing for longer than a decade and a half. Image processing was taught as a part of courses such are Algorithm Complexity, Multimedia Systems, Secure Software Design, Medical Imaging, and Medical Informatics. These courses were conducted on diverse levels of studies with different complexity and structure of the student projects.
Depending on the subject, we used to point out different elements of the image processing. Somewhere the focus is on the algorithm design, somewhere on the integration in large-scale systems, somewhere on a minimal alteration of the existing algorithms with a focus on the execution performance. For the evaluation of different programming approaches, we used the subject of Multimedia Software Systems – an electoral course in the fourth year of bachelor study. In the evaluated period, the number of students was in the range from 32 to 70 (2018 57, 2019 35, 2020 44, 2021 47, 2022 50, 2023 49, 2024 70).
The image processing software project is one of the assignments in the course, and the requirement is to implement several image processing algorithms of different complexity with minimal use of processing power and memory. For the test, the students must create an application that is comparable to Windows Forms in a .NET environment which can immediately display the result after processing is finished.
The only limitation that students have is related to the user interface which has to be fast and responsive and allow the display of the processed image. There is no specific request for a certain technology, besides, we advocate Microsoft .NET. On the other hand, there are no limitations to the implementation of image processing algorithms themselves.
Table 3 shows the number of enrolled students per year together with the number of students who successfully finished the entire project. Excluding the years 2019 and 2020, where the lectures were conducted in online mode, the percentage of the students who finished the projects was above 80%. During the period of online classes, this percentage dropped to less than two-thirds.
In the previous year (2023) we made a slight change in course organization by moving the image processing project to be the last in the row, as we considered it more complex. This gave the students the possibility to work on it in a period where they had fewer overall tasks during the school year. This resulted in the highest percentage of successfully finished projects at 90%. This year, we will follow the same approach, and, at the beginning of June, we will have complete data for 2024.
Data in Table 2 and Table 3 show the project distribution per technology. The projects, by the implementation technology of image processing algorithms, could be categorized into four major categories – managed, native, unsafe, and other. Projects developed in .NET and Java are considered managed code-based projects.
The projects marked as “native” are those whose algorithms are developed in various C++ environments, regardless of the technology used for the front end. In the category unsafe are these that follow the suggestion to include unsafe code blocks in managed projects. Category other is for the projects implemented in various Web technologies with different approaches considering implementations of algorithms both in the front and back end, using technologies such are various JavaScript-based frameworks.
After the students uploaded their projects in the collaborative learning platform, initially it was Moodle and later switched to Microsoft Teams, the projects were checked for performance in the demo machine to verify against the same conditions. The demo machine is an Intel-based i7-8550U running at 1.8 GHz with 4 cores and 8 logical processors, supported by 16GB of RAM. It is important to point out that all projects from the year 2018 until now are run and evaluated under the same conditions.
The Intel processors which mark ends by the sign U are not designed primarily for speed, but rather for energy efficiency. In that sense, such a computer is the perfect environment for the execution demo since the differences are better displayed.
The obvious benefit of using native and unsafe approaches can be seen in Table 6. Comparing execution time between native and unsafe approaches shows that native code runs 10 to 20 percent faster. The exact time varies depending on which moment of implementation students brought unsafe mechanisms to the project. For those that start with unsafe functions during data loading, the results are better than those that use unsafe mechanisms only for the algorithm execution.
The significant difference is between unsafe and managed code. The difference is in dozens of multiplications. Figure 12 shows that the difference is such, that the logarithmic scale could be easily employed to display the difference. Other approaches, based on different Web technologies demonstrate the worst results in the sense of the execution time. Based on Web technology, the disadvantage is that considerable time is required to upload the source picture and then to download the results, which, makes the situation worse.
What could be seen, during the years, is that students accept the resource awareness narrative at a high percent. Besides the higher popularity of Web applications and JavaScript-based frameworks, this approach was not the dominant choice to manage image processing problems. 2023 2024 8 7 12 13 9 8 8 9
Student projects are set up for three weeks, which makes 15 days the maximal amount of time needed for the execution. During the project, students must implement several filters that belong to various categories. The filters that should be implemented are changed each year. Besides that, the overall project complexity tends to be kept, in the teachers’ opinion, on the same level.
Table 7 and Figure 13 show an insight into how many days students were active on the project. These numbers represent the difference between the dates when students downloaded the assignment and the dates when the solutions were submitted. This is not the best conceivable way, since there is no exact way to prove the correctness of this approach, but, on the other hand, there is no morally acceptable way to measure how much time students spend working on their projects. Optionally the survey could be created, but the answers could be disputable either.
Projects predicated on managed code predominated in 2019 and 2020, coinciding with the period when lectures were delivered online. With the resumption of conventional in-person lectures and the enhancement of interactive, hands-on laboratory demonstrations, there was a discernible shift in preference towards approaches that are more efficient in terms of performance.
Nevertheless, several observations can be discerned. Primarily, students who commence their assignments late tend to opt for a managed solution or a JavaScriptbased framework, as these approaches necessitate less time investment. This trend was particularly pronounced during the years 2019 and 2020 when instruction was conducted in an online format.
Most unsuccessful solutions were developed using the native code, which is logical given its demand for more time and advanced programming expertise. Additionally, it is noteworthy that the greatest failure rate was observed in the “other” category. This may be attributed to students’ lack of sufficient familiarity with technologies that appeared promising, yet they lacked adequate experience to utilize them effectively.
An unsafe approach seemed like a good balance. It offers the possibility to start with the managed approach, reach some point in implementation, make the proof concept of all the algorithms, and then convert only parts of the project to the native code. Besides the approach being heavily technology-dependent and even not easily portable, it is a good example of the hybrid approach in development and how this kind of approach could bring overall benefits.
For an average of 15% more time than needed than for the projects based on managed code, the output runs only 10% slower compared to the projects in which image processing algorithms are entirely built in the native code. Ignoring years 2019 and 2020, the average time needed for the native approach is around 50% more than with the managed code.
Bringing the concepts of general resource awareness in programming is yet again important. Besides the rising popularity of fast-to-build and nice-to-look frameworks, followed by the constant increase of processing power and memory volume of all computational devices, one cannot entirely rely on the easiest solutions.
As could be seen with a managed approach in the languages based on the execution virtual machine, the two-line implementation could save time while programming, but it will result in a fifty times slower overall execution.
The use of the native environment will, of course, bring the best possible results, but the price will be significantly longer development phase. In that sense, the technology-specific solutions, like unsafe, could be an extremely good compromise between approaches. Getting 10% worse results, compared to native code, with spending around 15% more time compared to a completely managed approach could be in most cases optimal way.
However, each of the presented approaches has its pros and contras, and depending on the type of the projects, domain of usage, and the expectations regarding performance and memory usage, could be voted as optimal. In any case, future programmers must have a good overview of all the options and be aware of the appropriate use for the most valuable resource they choose – either development time or the use of the execution resources.
In this context, we posit that the incorporation of resource-awareness principles into the educational curriculum is crucial, particularly in the concluding year of study. Our research advocates for an instructional approach in image-processing education that utilizes managed code examples instead of pseudocode for algorithmic elucidation during lectures. Concurrently, laboratory exercises should employ unsafe and unmanaged code to elucidate the disparities in execution velocity, which can be significant in certain instances.
Through this paper, our objective is to convey that while numerous contemporary technologies offer considerable programming convenience and efficient development at the business layer, they should not be indiscriminately adopted as a panacea for all programming challenges. Instead, prospective developers must meticulously analyze the specific problem domain and judiciously select appropriate technologies for each component therein.
The Ministry of Science, Technological Development and Innovation of the Republic of Serbia supported this work [grant number 451-03-65/2024-03/200102]. This work is partially supported by the cost action CA 19135 CERCIRAS (Connecting Education and Research Communities for an Innovative Resource Aware Society).