Processing pipeline for automated data mining of the single astronomical objects from blurred CCD frames

Abstract

In this paper the authors presented a sophisticated data mining pipeline, which was designed for restoring the high-quality images from the blurred frames made by the Charge-Coupled Device (CCD) cameras. The developed data mining pipeline leverages the modern informational technologies for the horizontal and vertical scalability. The core methodology integrates the following mathematical methods and algorithms: an inverse median filtration method for the noise reduction and the Lucy-Richardson algorithm for deblurring. The inverse median filtration effectively reduces impulsive noise while preserving edges, and the Lucy-Richardson algorithm iteratively refines the image by correcting for blurring effects encoded in the point spread function (PSF). The proposed system's architecture of the processing pipeline for automated data mining of the single astronomical objects from blurred CCD frames utilizes the following modern technologies: Python programming language, Redis, FastAPI, React, Docker, and Caddy to ensure high performance, scalability, and ease of deployment. This integrated approach significantly enhances the accuracy of astronomical observations, facilitating more precise studies of celestial objects. The proposed pipeline addresses the unique challenges of astronomical image processing, offering a robust solution for automated data mining of single astronomical objects. Our work demonstrates the potential to advance astronomical research by improving image clarity and reliability, contributing to various fields within astronomy. The combination of effective noise reduction and deblurring techniques, along with a scalable and high-performance system architecture, provides a comprehensive solution to the challenges faced in processing astronomical images.

Keywords

• Data mining
• automated pipeline
• CCD frame
• astronomical image
• blurred image
• image processing
• object detection
• point spread function
• Lucy-Richardson algorithm
• noise reduction
• inverse median filter
• deconvolution1

1. Introduction

Astronomical imaging has significantly advanced our understanding of the universe [ 1 ]. However, capturing high-resolution images of celestial objects presents various challenges, one of the most prominent being image blur. This blur can obscure critical details necessary for astronomical research [ 2 ], thus necessitating sophisticated deblurring techniques.

In the quest to observe and understand celestial phenomena, astronomers rely on highly sensitive imaging devices. Charge-Coupled Device (CCD) cameras [3] have become the cornerstone of modern astronomical research due to their superior sensitivity to light and ability to produce high-quality images with fine detail and low noise. These attributes make CCD cameras indispensable for capturing faint celestial objects and critical details necessary for astronomical observations [ 4 ].

Despite the advantages of CCD cameras, various factors contribute to the blurring of astronomical images. These include atmospheric conditions, optical imperfections, mechanical issues, and intrinsic properties of light. Understanding these causes is essential for developing effective deblurring techniques [ 5 ].

Astronomical image blur primarily results from atmospheric turbulence, where heterogeneities in atmospheric density and temperature cause differential refraction of celestial light, leading to distortions and the twinkling effect observed in stars. This phenomenon, known as "seeing," significantly impacts the clarity of astronomical observations [ 6 ]. Optical aberrations in telescopes, arising from imperfections in the design or misalignment of optical components, introduce further degradation. Aberrations such as spherical, chromatic, and astigmatic distortions compromise image fidelity, even in high-quality telescopes.

Examples of blurry astronomical objects shown in the Figure 1.

Without effective mathematical methods [ 7 ] to counteract blur, identifying features of distant galaxies, studying nebulae structures, and detecting exoplanets become exceedingly difficult, often leading to incorrect interpretations and conclusions. Thus, developing various approaches to mitigate blur is critical for advancing astronomical research [ 8 ].

In this context, our work focuses on the implementation of the information system based on the processing pipelines for automated data mining [ 9 ] of single astronomical objects from blurred CCD frames. Our system is built upon cloud technologies, allowing us to scale the system efficiently, which is crucial when handling the large volumes of data common in astronomical research. The core of our methodology leverages the Lucy-Richardson algorithm [ 10 ], powerful deconvolution technique, to restore the original, unblurred images.

The significance of our work lies in its potential to enhance the accuracy of astronomical observations and interpretations. By effectively mitigating the effects of blur, our system facilitates more precise studies of celestial objects, contributing to advancements in various fields of astronomy [ 11 ], from galaxy formation and evolution to the search for exoplanets.

2. Related Works

Data mining in astronomical image processing [ 12 ] is a critical area of research, focusing on extracting valuable information from the vast amounts of data generated by modern telescopes and imaging devices. Despite significant advancements in technology, numerous challenges impede the effectiveness of current data mining techniques. These challenges range from image quality issues to algorithmic limitations, making it difficult to achieve accurate and reliable results.

Common approaches to handle astronomical image blurring include machine learning algorithms, deconvolution techniques, and various image processing methods. Machine learning algorithms, particularly deep learning, have shown promise in image restoration tasks. Convolutional neural networks (CNNs) [ 13 ] are widely used for their ability to learn complex patterns and features from large datasets. However, these models require extensive training data, which is often limited in astronomy, and are computationally intensive, posing challenges for realtime processing.

Deconvolution techniques [ 14 ] are another prevalent approach. These methods iteratively restore images by reversing the effects of blurring. While effective, they rely heavily on accurate point spread function (PSF) estimates, which can be difficult to obtain. Misestimation of the PSF can lead to artifacts and suboptimal image restoration.

Other methods include wavelet-based techniques [ 15 ] and matched filtration methods [ 16 ]. Wavelet-based approaches can effectively denoise and deblur images by decomposing them into different frequency components. However, these methods may struggle with the multi-scale nature of astronomical data, requiring additional techniques to enhance performance of the short time series [17]. Matched filtration methods [18], which utilize pre-defined filter shapes to enhance signal detection, can be useful but depend on precise knowledge of the blurring characteristics, limiting their flexibility in varying conditions. In the papers focusing on computer and machine vision [19], researchers developed foundational algorithms but lacked specific adaptations for astronomical image processing. The general algorithms discussed often fall short when dealing with the high noise levels and specific distortions found in astronomical images. This highlights the need for specialized techniques to handle unique challenges, such as cosmic ray hits and varying illumination.

Further studies suggest using the different image processing algorithms [20] including Sobel filter [21] for astronomical image recognition, which is effective in edge detection but struggles with the high levels of noise and blur typical in astronomical images. The Sobel filter [22], designed for general edge detection, fails to adequately enhance the fine details necessary for accurate astronomical analysis, potentially leading to misidentifications of celestial objects. Moreover, image recognition [23] indicates that processing speed decreases significantly as the size of the image frames increases, thereby limiting their applicability for high-speed processing tasks required in astronomical observations.

In a paper [24] suggested approach has the requirement for a stable and controlled environment for accurate measurements. This dependency may limit the practical application of the approach in more variable or field conditions, where maintaining such controlled conditions is challenging. Additionally, the approach relies on selecting and observing multiple markers in different positions, which introduces the risk of incorrect marker selection or observation errors. If the markers are not placed or observed correctly, it can lead to significant inaccuracies in the reference point determination, further complicating the process and reducing the reliability of the results.

3. Astronomical objects data mining pipeline

The proposed astronomical objects data mining pipeline is designed to address the challenges posed by blurred CCD frames in astronomical imaging and astronomical big data analysis [25]. The following sections provide a detailed description of each component of the designed pipeline and its role in restoring high-quality astronomical images including photometry [26]. Implemented pipeline have been integrated inside the fully functional information system with the web-based interface which allows to automate the astronomical objects data mining process.

3.1. Data mining pipeline description

The given pipeline integrates the median filter [27] and the Lucy-Richardson algorithm [28] in order to enhance the quality of astronomical images affected by blurring. This integration leverages the strengths of both techniques to effectively reduce noise and recover fine details, ultimately improving the accuracy of data mining processes for single astronomical objects considering the different typical forms [29]. The whole pipeline is shown in the Figure 2:

The initial phase of our data mining [30] pipeline entails the application of a median filter [31], a sophisticated non-linear digital filtering technique renowned for its efficacy in noise reduction. The median filter operates by traversing the image pixel by pixel, substituting each pixel value with the median value derived from the surrounding neighborhood of pixels. This method is particularly adept at mitigating impulsive noise, such as salt-and-pepper noise, while preserving the integrity of edges and fine details, making it an indispensable tool for pre-processing astronomical images prior to deblurring. Median filtering can be described using the following formula: ( , ) = ( , )− ( , ), (1) where ( , ) represents the output pixel value at coordinates (m, n) after applying the median filter; ( , ) denotes the input pixel value at coordinates (m, n) in the original image; ( , ) is the median value of the pixel values within the neighborhood centered around (m, n).

The resulting image with equalized background brightness may have pixels with a negative value, so an additional correction should be performed and the minimum value between all pixels should be subtracted from each pixel: ( , ) = ( , )− where ( , ) represents the output pixel value at coordinates (m, n) after applying the correction; ( , ) denotes the input pixel value at coordinates (m, n) in the original image; the minimal value of the pixel in the image.

The median filter is uniquely advantageous in its ability to preserve edge sharpness, which is crucial in astronomical imaging where the accurate delineation of celestial bodies is paramount. Traditional linear filters, like the mean filter, tend to blur edges along with noise reduction, leading to a loss of critical information.

In the context of our pipeline, the use of the median filter is a critical pre-processing step. Astronomical images often suffer from various types of noise introduced during the capture process by CCD cameras, atmospheric conditions, or electronic interference. By applying the median filter, we can significantly enhance the quality of the raw images, thereby facilitating more accurate subsequent deblurring using the Lucy-Richardson algorithm. Furthermore, the median filter's robustness against noise and its edge-preserving properties makes it highly suitable for astronomical applications, where precision and clarity are essential.

After noise reduction via the median filter, the deblurring process is executed using the LucyRichardson algorithm [32]. This algorithm is specifically tailored to recover a latent image that has the output pixel i; following equation: , , ℎ

′ ℎ where ( − ) is the point spread function;

is the element i, j in the transition matrix p.

The iterative nature of the Lucy-Richardson algorithm allows for progressive refinement of the image, with each iteration enhancing the clarity and detail by correcting for the blurring effects encoded in the PSF. It operates by maximizing the likelihood that the observed blurred image could be obtained from the deblurred image when convolved with the PSF. The Lucy-Richardson method on each iteration can be described using following equation: been subjected to blurring by a known point spread function (PSF). The PSF characterizes the response of the imaging system to a point source or a point object, encapsulating the spread of the point source's light due to factors such as atmospheric turbulence, motion, or lens aberrations. The observed image can be decomposed as a sum of individual points and represented through a transition matrix: = ∑ , ,

, = ( − ), (3) (4) (5)

One of the key strengths of the Lucy-Richardson algorithm is its efficacy in restoring images degraded by various forms of blur, including motion blur, out-of-focus blur, and atmospheric distortion. Its robustness is further underscored by its ability to produce high-quality deblurred images even when the PSF is not perfectly known, leveraging iterative refinements to converge towards an accurate representation of the latent image.

Overall, the combination of the median filter for noise reduction and the Lucy-Richardson algorithm for deblurring forms a powerful pipeline for enhancing the quality of astronomical images. This pipeline is particularly advantageous in the context of processing blurred CCD frames, where high precision and clarity are paramount for accurate data mining and analysis of single astronomical objects.

3.2. System design and architecture

The system architecture for the astronomical objects data mining pipeline is constructed using a combination of modern technologies and frameworks to ensure high performance, scalability, and ease of deployment. The architecture is built upon Python, Redis, FastAPI, React, PostgreSQL, Docker, DockerCompose, and Caddy, each playing a critical role in the functionality and efficiency of the pipeline. The suggested architecture is provided in the Figure 3:

Python [33] serves as the orchestrator for the data mining pipeline, overseeing the coordination and execution of tasks. However, the computationally intensive parts of the pipeline, including the implementation of the median filter and the Lucy-Richardson algorithm, are developed as precompiled binary files to maximize performance and efficiency. Redis is utilized as a task queue [34], effectively managing the distribution and scheduling of tasks within the system. This ensures that the processing of data is both streamlined and efficient, reducing latency and optimizing resource usage.

FastAPI functions as the backend framework, providing a high-performance, scalable API for handling client requests and managing the data mining pipeline. The asynchronous capabilities of FastAPI significantly enhance performance by enabling the concurrent handling of multiple requests. Additionally, FastAPI auto-generates interactive API documentation using Swagger UI, facilitating ease of use and integration.

React is employed to develop the web-based interface of the information system. This interface allows users to interact with the pipeline, upload images, and visualize the processed results. React's component-based architecture ensures a modular and maintainable codebase, while libraries such as Redux efficiently manage application state, enhancing the user experience. PostgreSQL is used as the primary database for storing and managing the metadata associated with the images and processed results. PostgreSQL's robustness [35], support for complex queries, and ACID compliance make it an ideal choice for handling the relational data required by the system.

Caddy is utilized as the web server and reverse proxy, offering several advantages, including automatic HTTPS for secure communication with automated TLS certificate management, and simplified setup and configuration compared to traditional web servers.

For deployment, Docker-Compose [36] is employed to set up a local development environment, ensuring that all services run seamlessly together. In production, the application is deployed on a cloud platform using Docker containers, with Caddy managing secure HTTP traffic and load balancing.

In summary, the described system architecture leverages a sophisticated list of technologies to construct an efficient and scalable data mining pipeline for astronomical images. By integrating Python for task orchestration, precompiled binaries for computationally intensive processing, Redis as a task queue, FastAPI for backend services, React for the frontend interface, and Docker with Caddy for deployment, the pipeline achieves high performance and user-friendliness. This architecture not only enhances the quality of astronomical images but also ensures that the entire process, from data ingestion to result retrieval, is seamless and efficient.

3.3. System interface

The data flow within the system initiates with users uploading blurred CCD images through the React-based web interface as a task (see Figure 4). can see task information which extended with details about data processing time.

Finally the user can download the results by pressing the download button shown in the Figure 6. This action triggers the download of an archive file, which, upon extraction, reveals two main directories: one labeled "input" and the other "output."

Within the "input" folder, users will find a comprehensive collection of images that were originally uploaded or processed by the system. These images are cataloged and match the listings displayed in Figure 6. Similarly, the "output" folder contains the resultant images generated or modified during the processing stage. The organization of these images in both folders follows the sequence and order as depicted in the corresponding sections of Figure 6, ensuring easy crossreferencing and verification.

4. Results

The Figure 7 illustrates the effectiveness of the proposed data mining pipeline in the context of astronomical image processing. On the left side of the image, we see an example of a blurred astronomical image, where stars appear as elongated streaks due to motion blur or atmospheric disturbances during the capture. This blurring effect can obscure important details and hinder the analysis of celestial objects [37].

On the right side of the image, the same scene has been processed using the implemented pipeline. The result is a significantly clearer image where the stars are now sharp points of light, revealing more detailed and accurate representations of the astronomical scene including reference stars [38]. On the closer view of the provided frames this difference is even more observable (see Figure 8).

5. Conclusions

This paper presents a sophisticated data mining pipeline designed for the automated restoration and analysis of the single astronomical objects from the blurred CCD frames. The research was conducted in scope of the CoLiTec (Collection Light Technology) project [39].

The pipeline incorporates advanced methodologies, specifically the median filter and the LucyRichardson algorithm, to effectively mitigate noise and deblur images, thereby enhancing the quality of astronomical observations, which is very important for the photometry tasks [40].

The median filter is crucial for noise reduction, particularly in mitigating impulsive noise while preserving essential image details and edges. This pre-processing step is fundamental in preparing images for subsequent deblurring. The Lucy-Richardson algorithm then iteratively refines the deblurred images by compensating for the blurring effects characterized by the PSF. This combination ensures that the images are not only clearer but also retain critical astronomical details necessary for accurate data analysis

Our pipeline is embedded within a robust information system designed for high performance and scalability, leveraging contemporary technologies such as Python, Redis, FastAPI, React, Docker, and Caddy.

This architectural design ensures the system's ability to handle large volumes of data efficiently, addressing the common requirements in astronomical research. The effectiveness of the implemented pipeline is demonstrated through significant improvements in image clarity, as illustrated in our results section. The developed pipeline can be also used for the different automated monitoring and visualization systems [41] to track the astronomical objects in real-time.

In conclusion, the developed pipeline offers a powerful solution to the challenges posed by blurred astronomical images. By integrating efficient noise reduction and deblurring techniques within a scalable system architecture and data stream clustering [42], this work significantly advances the capabilities of automated data mining [43] in astronomy. Also, the results of our implementation with a higher quality and statistical precision [44] will be useful in application of the machine learning methods [45].

The results underscore the potential of this approach to improve the accuracy and reliability of astronomical observations, thereby supporting more detailed and precise astronomical research of the Solar System objects and even of the high-speed aircraft [46] and low-altitude mobile robots [47].

Acknowledgements

The research was supported by the Ukrainian project of fundamental scientific research Development of computational methods for detecting objects with near-zero and locally constant motion by optical-electronic 0124U000259 in 2024-2026 years. 33695-0_57. [39] S. Khlamov, et al., Machine Vision for Astronomical Images using The Modern Image Processing Algorithms Implemented in the CoLiTec Software, Measurements and Instrumentation for Machine Vision, (2024) 269 310. doi: 10.1201/9781003343783-12. [40] A new tool for an automated reduction of photometric observations, Contributions of the Astronomical Observatory Skalnate Pleso, 49 2 (2019) 151153. doi:2019CoSka..49..151P. [41] V. Lyashenko, A. T. Abu-Jassar, V. Yevsieiev, and S. Maksymova, Automated Monitoring and Visualization System in Production, Int. Res. J. Multidiscip. Technovation, 5 6, (2023) 9-18. doi: 10.54392/irjmt2362. [42] P. Zhernova, et al., Data stream clustering in conditions of an unknown amount of classes, Advances in Intelligent Systems and Computing 754 (2019) 410 418. doi: 10.1007/978-3-31991008-6_41. [43] I. Perova, Y. Brazhnykova, N. Miroshnychenko, and Y. Bodyanskiy, Information Technology for Medical Data Stream Mining, in: 15th International Conference on Advanced Trends in Radioelectronics, Telecommunications and Computer Engineering, 2020, 93 97 pp. doi: 10.1109/TCSET49122.2020.235399. [44] V. Shvedun, et al., Statistical modelling for determination of perspective number of advertising legislation violations, Actual Problems of Economics 184 10 (2016) 389-396. [45] L. Kirichenko, O. Pichugina, T. Radivilova, and K. Pavlenko, Application of Wavelet Transform for Machine Learning Classification of Time Series, Lecture Notes on Data Engineering and Communications Technologies 149 (2023) 547 563. doi: 10.1007/978-3-031-16203-9_31. [46] A. Tantsiura, et al., Evaluation of the potential accuracy of correlation extreme navigation systems of low-altitude mobile robots, International Journal of Advanced Trends in Computer Science and Engineering 8 5 (2019) 2161 2166. doi:10.30534/ijatcse/2019/47852019. [47] N. Yeromina, V. Tarshyn, S. Petrov, et al., Method of reference image selection to provide highspeed aircraft navigation under conditions of rapid change of flight trajectory, International Journal of Advanced Technology and Engineering Exploration 8 85 (2021) 1621 1638. doi:10.19101/IJATEE.2021.874814.

References

[1] J. Bennett , S. Shostak , N. Schneider , and M. MacGregor, Life in the Universe. Princeton University Press, 2022 .

[2] V. Troianskyi , V. Kashuba , O. Bazyey , et al., First reported observation of asteroids 2017 AB8 , 2017 QX33, and 2017 RV12, Contributions of the Astronomical Observatory Skalnaté Pleso 53 ( 2023 ) 5 - 15 . doi: 10 .31577/caosp. 2023 . 53 . 2 .5. devices, International Journal of Circuit Theory and Applications , vol. 48 , issue 7 , ( 2020 ) 1001 - 1016 . doi: 10 .1002/cta.2784.

[4] D. Oszkiewicz , et al., Spins and shapes of basaltic asteroids and the missing mantle problem , Icarus , vol. 397 , ( 2023 ) 115520 . doi: 10 .1016/j.icarus. 2023 . 115520 .

[5] Y. Luxin , et al., Atmospheric-Turbulence-Degraded Astronomical Image Restoration by Minimizing Second-Order Central Moment , IEEE Geoscience and Remote Sensing Letters 9 4 ( 2012 ) 672 - 676 . doi: 10 .1109/LGRS. 2011 . 2178016 .

[6] V. Troianskyi , V. Godunova , A. Serebryanskiy , G. Aimanova , L. Franco , A. Marchini , P. Bacci , et al., Optical observations of the potentially hazardous asteroid (4660) Nereus at opposition 2021 , Icarus 420 ( 2024 ) 116146 . doi: 10 .1016/j.icarus. 2024 . 116146 .

[7] V. Savanevych , et al., Mathematical methods for an accurate navigation of the robotic telescopes , Mathematics 11 10 ( 2023 ) 2246 . doi: 10 .3390/math11102246.

[8] D. Oszkiewicz , et al., Spin rates of V-type asteroids , Astronomy and Astrophysics 643 ( 2020 ) A117 . doi: 10 .1051/ 0004 -6361/202038062.

[9] S. Cavuoti , M. Brescia , and G. Longo , Data mining and knowledge discovery resources for astronomy in the Web 2.0 age, SPIE Astronomical Telescopes and Instrumentation, Software and Cyberinfrastructure for Astronomy II 8451 2012 . doi: 10 .1117/12.925321.

[10] S. Khetkeeree , Optimization of Lucy-Richardson Algorithm Using Modified Tikhonov Regularization for Image Deblurring , J. Phys.: Conf. Ser . 1438 ( 2020 ) 012014 . doi: 10 .1088/ 1742 - 6596/1438/1/012014.

[11] V. Akhmetov , et al., Astrometric reduction of the wide-field images , Advances in Intelligent Systems and Computing 1080 ( 2020 ) 896 909. doi: 10 .1007/978-3- 030 -33695-0_ 58 .

[12] R. Mor , et al., Expanding Big Data mining for Astronomy, XIV Scientific Meeting of the Spanish Astronomical Society ( 2020 ) 235 . doi: 2020sea.. confE.235M.

[13] Y. Bodyanskiy , S. Popov , F. Brodetskyi , and O. Chala , Adaptive Least-Squares Support Vector Machine and its Combined Learning-Selflearning in Image Recognition Task , International Scientific and Technical Conference on Computer Sciences and Information Technologies ( 2022 ) 48 51. doi: 10 .1109/CSIT56902. 2022 . 10000518 .

[14] Peng , L. Xiyu , L. Zhengyang , et al., Point spread function modelling for wide-field smallaperture telescopes with a denoising autoencoder , MNRAS 493 ( 2020 ) 651 - 660 . doi: 10 .1093/mnras/staa319.

[15] M. Dadkhah , et al., Methodology of wavelet analysis in research of dynamics of phishing attacks . International Journal of Advanced Intelligence Paradigms 12 3 - 4 ( 2019 ) 220 - 238 . doi: 10 .1504/IJAIP. 2019 . 098561 .

[16] S. Khlamov , et al., Development of computational method for matched filtration with analytic profile of the blurred digital image , Eastern-European Journal of Enterprise Technologies 5 119 , ( 2022 ) 24 32. doi: 10 .15587/ 1729 - 4061 . 2022 . 265309 .