The very fast technological progress provokes the creation of a big volume of the information that can be fed in different forms. There are various science directions that use high dimensional data sets for the analysis. In this paper we presented the few aspects of the "knowledge discovery in databases" (KDD) process related to the Data Mining stage in astronomy, analyzed and reviewed Data Mining approaches. We presented the examples of astronomical sources of Big Data, instruments, information types, processing algorithms that can be used for the Data Mining process in astronomy. The paper deals with applying the CoLiTec (Collection Light Technology) software for the online processing of the different types of astronomical information using the Data Mining approach. This is achieved by using of the developed OnLine Data Analysis System (OLDAS), which helps with solving of the Data Mining tasks, like clustering, classification, and identification. Data mining, big data, knowledge discovery in databases, recognition patterns, image COLINS-2022: 6th International Conference on Computational Linguistics and Intelligent Systems, May 12-13, 2022, Gliwice, Poland ORCID: 0000-0001-9434-1081 (S. Khlamov); 0000-0001-8840-8278 (V. Savanevych); 0000-0002-4550-5606 (O. Briukhovetskyi); 00000001-6629-4927 (I. Tabakova); 0000-0003-2689-2679 (T. Trunova)
processing, classification, datasets, series of images
The huge engineering revolution is closely connected with the 21st century and characterized by
the terrific technological progress. Such progress causes the creating of a large number of the various
data that fed in an online or offline modes in the form of data streams, predefined sets, series, video,
etc. [
This ability of data to grow is ahead of all computing abilities of the already existed
computers/machines/servers. In this case the processing optimization of the data streams, sets, data is
very important by using only required input information to help computers/machines/servers to work
more productively.
exception [
The data mining and knowledge discovery approaches become more and more popular and actual in the different research and experiments to improve productivity and efficiency of the processing algorithms in the different fields of interest. Astronomy as a research field of interest is not an
So, what is the data mining approach? It is a process of the information receiving from the large data sets by using the extracting or discovering patterns and involving the methods at the intersection of disciplines like computer science, statistics, machine learning and database systems. EMAIL: (S.
Khlamov); izumsasha@gmail.com
2022 Copyright for this paper by its authors.
The data mining carries out about the information extracting using the intelligent methods from a
data set to transform it into the obvious structure for the further use. The data mining is an analysis
step of the "knowledge discovery in databases" (KDD) process [
The main goal of the data mining is extraction of the potentially useful information for the
knowledge from the given large input data sets/streams/video using the appropriate associations,
relationships, or recognition patterns [
In this paper we presented the few aspects of the "knowledge discovery in databases" process
related to the data mining stage in astronomy, analyzed and reviewed data mining approaches. The
examples of astronomical sources of big data, instruments, kind of information, processing algorithms
are provided. The goal of current research is to apply the developed CoLiTec (Collection Light
Technology) software [
The data mining and knowledge discovery become areas of growing significance. Such growing
was caused by the increasing needs for KDD techniques for the different directions, like databases,
knowledge gathering, machine learning, statistics, data visualization, and high performance
computing. The data mining and knowledge discovery also is highly useful for the artificial
intelligence techniques in many areas, like industry, commerce, government, education, astronomy
and so on [
The data mining in astronomy is a very powerful approach, which has a big potential for the fully exploitation the exponentially increasing amount of data and promises an excellent scientific progress. But with the wrong using it can be little more than the black box application of complex computing algorithms that can provide very questionable results. So, the data mining can be much more powerful tool, which is pretty good adapted for the astronomical tasks, instead of accurate selection or continually modification an appropriate processing algorithm.
Nowadays, in the big data era, there are different fields in astronomy that are vital for the dealing
with big data and data mining issues. They are astroinformatics, astrostatistics, astrochemistry, etc.
All progressive astronomers, researchers, scientists are ready to face the technological challenges and
opportunities provided by the massive data volume and open exciting perspectives for the new
astronomical discoveries by applying of the advanced data mining approach. The diversity of
scientific tasks and complexity of the astronomical big data provoke the development of innovative
processing algorithms and methods as well as a highly usage of the Information and Communications
Technologies (ICT) [
There are a lot of different scientific programs, projects, databases, virtual observatories [
The data mining problems of data analysis and visualization [
Especial attention can be paid to the SDSS (Sloan Digital Sky Survey) [19], as to the most successful sky survey in the astronomy history. The SDSS project has formed the most detailed threedimensional maps of the Universe with deep multi-color images of 1/3 of the sky, and spectra for more than three million astronomical objects [20].
One more interesting and more huge wide-field survey with reflecting telescope is under construction and called Large Synoptic Survey Telescope (LSST) [21]. It has a primary mirror with diameter 8.4 meters and includes three mirrors with a very wide field of view (FOV) of 3.5-degree, which is presented in the Figure 2.
The LSST science database is focused on the following goals: • scalability (at petabytes scales) of existing machine learning and data mining algorithms; • development of grid with enabling the parallel data mining algorithms; • designing a robust system for brokering classifications from the event pipeline; • indexing of multi-attribute and multi-dimensional petascale astronomical databases for the rapid querying; • multi-resolution methods (object classification, outlier identification, anomaly detection).
The data mining purposes regarding the astronomical image processing are focused on but not
limited to the following tasks: brightness equalization [22], background alignment [23], object’s
images detection [
The data mining of astronomical images includes the different major areas of application of image and signal processing like the following [27].
• Filtering. The clear raw signals in astronomy are very rarely existed without noise, so the
removal of noise is necessary for the future useful data interpretation. In common, the data
cleaning is required to bypass the artifacts of instrumental measurements without changing of
the complexity of data.
• Deconvolution. The signal "deblurring" is also used for reasons that are very similar to
filtering, as a preliminary to the data interpretation. Deblurring of the objects motion in images
is very important in astronomy, as well as the removing effects of atmospheric blurring, to
improve the quality of seeing.
• Compression. There are several facts that show the importance of effective and efficient
compression technology: long-term storage of astronomical data, developing of detectors for
the ever-larger image sizes, research in astronomy is a geographically distributed activity.
• Mathematical morphology. The combinations of erosion and dilation operators often
provoke the opening and closing operations. So, in the greyscale/boolean images they allow
creation of immediately practical framework for the processing. The median function plays
such role for the order and rank functions. In this case, the multiple scale mathematical
morphology is an immediate generalization of the astronomical images processing [29].
• Edge detection. The gradient information is not very popular information for the astronomical
image analysis because of their boolean nature. So, in this case, the objects edges identification
is used more often like curves in the image by the brightness changes or discontinuities [30].
• Corner detection. A group of algorithms that are used within computer vision systems to
extract certain kinds of the features and infer contents of an image [31]. It is often used for the
object’s recognition, object’s image registration, object’s motion detection [
Such regions have a difference in brightness and color that are compared to the neighboring regions. The blob is a region with points in which properties are constant or approximately constant, so all points in the blob are like each other [32]. • Ridge detection. The mathematical methods for the ridge’s localization in the image that defined as curves whose points are the function’s local maximum, like the geographical ridges [33]. • Segmentation and pattern recognition. In astronomy, the segmentation and pattern recognition is used for the object detection while the term feature selection [26] is more popular in areas outside astronomy. In common, they are used for the assignment of the object’s images to a proper class by the highlighting of significant features that characterize this class [34]. • Hough and Radon transforms. The detection of curves is required for the many segmentation classes and feature analysis. It does not matter if the signal is faint or strong, the noise is usually the most critical one. The Ridgelet and Curvelet transforms provide the powerful generalizations for resolving such problems [27].
The described above mathematical image processing methods are different but all of them can be used as pre-processing stage of the data mining of astronomical images in the processing pipeline before the main image processing algorithm (object’s image recognition, object detection, objects parameters estimation, trajectory detection, trajectory parameters estimation) is applied. 3.2.
The object classification is an important initial step in the scientific data mining process because it provides the algorithms and methods for organizing the scientific information in a way that can be used to make the appropriate hypotheses and to compare with the existing models.
Because of the small physical size of stars compared to their distance from the observing point, almost all stars are unresolved in the photometric datasets, and thus appear as the point objects in the CCD-image [35]. The galaxies in common case subtend a larger angle, even when they are further away, so appear as the extended objects in the CCD-image. But the other astrophysical objects such as quasars and supernovae also appear as point objects. So, the separation of photometric catalogs into stars, galaxies, and other objects, is an important and difficult task.
The huge number of galaxies and stars in typical surveys requires the morphology separation as a process, which is automated or semi-automated. This task is a well-studied and the several automated approaches for big data analysis were implemented, like for the digitization of the scanned photographic plates by machines such as the Automatic Plate Measuring (APM) [36] and Palomar Digital Sky Survey (DPOSS) [37].
Also, the several data mining methods have already been developed and implemented using the Artificial Neural Network (ANN) [38], mixture modeling [39], where the most methods achieving over 95% efficiency.
In general, such methods are based on the astrophysical object’s classification using a set of the measured morphological parameters that are received from the survey photometry, with shape, structure, texture, inclination, arm pitch, color, resolution, exposure, colors, spectra, and other astrophysical information.
The main advantage of these data mining methods is that all such information about each astrophysical object is easily extended and incapsulated into the massive datasets [40].
There is a various morphology of galaxies based on the wide range of different sizes and shapes of them. The most popular system of the morphological classification of galaxies is the Hubble Sequence of spiral, barred spiral, elliptical, and irregular, and galaxies from the different subclasses [41]. This system correlates to the many important physical properties in the formation and evolution of galaxies [42].
The galaxy morphology is a very complex phenomenon, which is correlated to the underlying physics, but it is not unique to any one given process. But, anyway, the Hubble sequence is still actual, even if it being rather subjective and based on the visible-light morphology, which was originally received from the blue-biased photographic plates.
The Hubble sequence was extended in different ways using the data mining approach and ANN applying [38] to predict the galaxies’ type at low redshift and finding the equal accuracy to human experts. ANNs were also applied to the higher redshift data to distinguish between normal and peculiar galaxies. Also, the fundamentally topological and unsupervised SOM ANN was used for the galaxy’s classification based on the CCD-images, received from the Hubble Space Telescope [43], where the initial distribution of classes is not known. The approach of using the ANNs also was used to determine the morphological types from galaxy spectra [44].
For the galaxy morphology research even the Fourier decomposition was used on the galaxy images implemented with ANNs for the bars detection and types assigning [45].
The different data mining approaches for the astronomical images processing is provided by the
CoLiTec software [
The high level processing pipeline with developed modules and implemented methods of the CoLiTec software is presented in the Figure 3.
The processing steps of the CoLiTec software in the pipeline according to the data mining approaches are described below. 4.1.
The pre-processing step of the CoLiTec software in OnLine Data Analysis System (OLDAS) mode includes the input data set processing as soon as they successfully received from different sources. Such raw data is moderated before the computational process starts. The unsupported and corrupted frames are rejected at this step. The useful information from the input data set is only used during the computational process. 4.2.
The selected useful information from the input data set is categorized into clusters using the specified attributes. The CoLiTec software uses the different attributes, such as equatorial coordinates, filter type, telescope, investigated object and others. Based on these attributes the necessary information from the input data set is separated into subsets with similar data and stored at the different distributed servers, clusters or even networks.
After clustering process, the created subsets of data are classified by the applying of a known structure of the raw astronomical data that specified in Flexible Image Transport System (FITS) standard by NASA [46]. FITS standard is the most used digital file format in astronomy. Such format is designed especial in form of the image metadata, which includes different scientific data, like astrometric, photometric, calibration information and others. After classification the FITS files are sent to the processing pipeline. 4.4.
During processing pipeline all received classified FITS files pass through the identification step. At this step all FITS files related to the service master-frame are used for the frame's calibration (e.g., bias, dark, darkflat, flat). Otherwise, if this is a raw light frame the processing pipeline starts computing process. 4.5.
The computing process in the processing pipeline is managed by the OLDAS and includes two
stages: intraframe and interframe processing. The intraframe stage includes the various processes for
the image filtration and objects detection. The major goal of the object’s detection in the series of
images is to recognize the object, its borders and determine the parameters of its image [
There are a different recognition patterns or types of the astronomical objects in the image that can
be detected: point objects, long objects, blurred objects, objects with flare or intersection with another
objects. Such types of objects can be belonged to the galaxy, star, robot [47], drone [48], rocket,
satellite [49], and even comet [50] or asteroid [
The features of CoLiTec software related to the intraframe stage are described below: • processing of the very wide field of view (FOW) – up to 10 square degrees; • automated calibration process; • cosmetic correction process; • FrameSmooth software for background alignment and brightness equalization [22]; • automated rejection of the worst observations; • fully automated robust algorithm of astrometric reduction; • semi-automated algorithm of photometric reduction; • automated rejection of objects with bad or unclear measurements.
The object’s image detection process during the intraframe processing is presented in the Figure 4.
The interframe stage includes the various processes for detection of the objects motion. The major
goal of the moving objects detection in the series of images is to recognize the object’s trajectory and
determine its parameters [
The features of CoLiTec software related to the interframe stage are the following: • automated detection of the faint moving objects with signal-to-noise ratio (SNR) more than 2.5; • automated detection of very slow objects with near-zero apparent motion from 0.7 pix./frame; • automated detection of very fast objects with apparent motion up to 40.0 pix./frame.
The object’s motion detection process during the interframe processing is presented in the Figure 5.
The data mining analysis by the CoLiTec software is performed using the following technological features: • multi-threaded processing; • multi-cores systems using with managing the individual treatment processes; • deciding system, which allows adapting the user settings for the processing; • notification system, which informs user about the correct results at each processing stage; • data control managing during processing using the subject mediator.
After pipeline processing and data mining analysis, the CoLiTec software produces the various forms of results representation, including visualization results and reports generation for the different services. To summarize results and for the visual analysis of them, the LookSky viewer with userfriendly GUI is used (see Figure 6).
As a result, about 700,000 observations were made using the CoLiTec software with approach for the data mining of astronomical images. According to these observations the following discoveries were also done: • more than 1,600 asteroids; • 5 Near Earth Objects (NEOs); • 21 Trojan asteroids of Jupiter; • 1 Centaur (2013 UL10); • 5 comets (C/2010 X1 (Elenin), P/2011 NO1 (Elenin), C/2012 S1 (ISON) [50], P/2013 V3 (Nevski), C/2017 T3 (ATLAS) [51]).
All mentioned above observations and discoveries are approved and confirmed by the Minor Planet Center (MPC) as an official organization for the observing and reporting on minor planets under the auspices of the International Astronomical Union (IAU).
Below are presented the several examples of the series of images processing after the data mining
stage in the processing pipeline of the CoLiTec software [
The few processing results of the automated calibration, cosmetic correction, background alignment and brightness equalization processes are presented in the Figure 7.
The few processing results of the object’s measurements mining, object’s image detection, and object’s apparent motion detection processes are presented in the Figure 8.
For all detected by the CoLiTec software objects a lot of apparent motion parameters are determined. Some of them are the object’s velocity V in the equatorial system (right ascension RA and declination DE) and in the cartesian system according to two axis x and y, and the distance S, which was passed by the object from the first positional measurement till the last one in the observing series of images.
The few processing results with the determined apparent motion parameters of the real Solar System objects (SSOs) are presented in the Table 1.
The values of S distance, which was passed by the different objects from the first positional measurement till the last one in the observing series of images, shows that such objects have near-zero apparent motion, and the CoLiTec software successfully detected these objects and estimates the positional parameters and the motion parameters.
The very fast technological progress, networks of automated ground- and space-based observation systems, new scientific programs, surveys, projects lead to the fast growing of astronomical data.
We presented the developed CoLiTec software [
The CoLiTec software realizes different data mining principles and stages of processing, such as anomaly detection, pre-processing, clustering, classification, identification, processing (edge detection, segmentation, recognition patterns, object [52] and motion detection) and summarization.
Such data mining principles in the CoLiTec software are implemented by the especial developed
mathematical methods and components for the intraframe and interframe processing, astronomical
data mining from the different on-line services, archives [
The scientific novelty of the current research is that the CoLiTec software is the first astronomical software, which fully implements all steps according to the data mining approach and pipeline to process the different astronomical data.
Using the described in the paper data mining approaches the CoLiTec software helps with countless observation and discoveries of SSOs, which are confirmed by the MPC and the appropriate Minor Planet Electronic Circulars (MPECs).
The authors thank all observatories, online services and tools that provided data to conduct the current research for testing the developed CoLiTec software with realization of the data mining approaches.
[13] E. Ríos-López, et al., 2D surface brightness modelling of large 2MASS galaxies–I: photometry and structural parameters, MNRAS, vol. 507, issue 4, pp. 5952-5973, 2021. doi: 10.1093/mnras/stab2321. [14] B. Lyu, et al., WISE View of Changing-look Active Galactic Nuclei: Evidence for a Transitional
Stage of AGNs, ApJ, vol. 927, issue 2, 227, 2022. doi:10.3847/1538-4357/ac5256. [15] C. Baccigalupi, et al., Euclid preparation, A&A, vol. 657, A90, 2022. doi:10.1051/00046361/202141393. [16] K. Xiao, and H. Yuan, Validation and Improvement of the Pan-STARRS Photometric Calibration with the Stellar Color Regression Method, The Astronomical Journal, vol. 163, issue 4, p. 185, 2022. doi:10.3847/1538-3881/ac540a. [17] J. Guiraud, and W. Roux, New Questions Opened by the Big Data in the World of the Science Data Processing Centre for Gaia Mission in CNES, Space Operations. Springer, Cham, pp. 309324, 2022. doi:10.1007/978-3-030-94628-9_14. [18] P. Prabahar, and N. Radhakrishnan, Thirty Meter Telescope (TMT) Research as Reflected in Web of Science: A Study, Science & Technology Libraries, pp. 1-13, 2022. doi:10.1080/0194262X.2021.2018634. [19] Yi, Zhenping, et al., Automatic detection of low surface brightness galaxies from SDSS images,
MNRAS, stac775, 2022. doi:10.1093/mnras/stac775. [20] Y. Zhang and Y. Zhao, Astronomy in the big data era, Data Science Journal, vol. 14, 2015. [21] D. J. Hoover, D. Z. Seligman, and M. J. Payne, The Population of Interstellar Objects Detectable with the LSST and Accessible for In Situ Rendezvous with Various Mission Designs, The Planetary Science Journal, vol. 3, issue 3, p. 71, 2022. doi:10.3847/psj/ac58fe. [22] P. A. Dubovský, et al., FrameSmooth software-new tool for the calibration of astronomical images, 48th Conference on Variable Stars Research, 16 p., 2017. [23] Š. Parimucha, et al., CoLiTecVS – A new tool for an automated reduction of photometric observations, Contributions of the Astronomical Observatory Skalnate Pleso, vol. 49, issue 2, pp. 151-153, 2019. doi:2019CoSka..49..151P. [24] V. Akhmetov, S. Khlamov, V. Khramtsov, and A. Dmytrenko, Astrometric reduction of the wide-field images, Advances in Intelligent Systems and Computing, vol. 1080, pp. 896–909, 2020. doi: 10.1007/978-3-030-33695-0_58. [25] I. Kudzej, et al., CoLiTecVS – A new tool for the automated reduction of photometric observations, Astronomische Nachrichten, vol. 340, pp. 68–70, 2019. [26] V. Savanevych, V. Akhmetov, et al., Selection of the reference stars for astrometric reduction of CCD-frames, Advances in Intelligent Systems and Computing, vol. 1080, pp. 881–895, 2020. doi: 10.1007/978-3-030-33695-0_57. [27] R. Gonzalez, and R. Woods, Digital image processing, Fourth edition, New York, NY: Pearson, 2018. [28] M. Dadkhah, V. V. Lyashenko, Z. V. Deineko, et al., Methodology of wavelet analysis in research of dynamics of phishing attacks, International Journal of Advanced Intelligence Paradigms, vol. 12(3-4), pp. 220-238, 2019. doi:10.1504/IJAIP.2019.098561. [29] W. Burger, and M. Burge, Principles of digital image processing: fundamental techniques, New
York, NY: Springer, 2009. [30] D.-W., Lee, and S.-H. Lee, Edge Detection using Cost Minimization Method, Journal of Internet of Things and Convergence, vol. 8, issue 1, pp. 59-64, 2022. doi:10.20465/KIOTS.2022.8.1.059. [31] Y. Wang, et al., A Target Corner Detection Algorithm Based on the Fusion of FAST and Harris,
Mathematical Problems in Engineering, vol. 2022, 4611508, 2022. doi:10.1155/2022/4611508. [32] T. Lindeberg, Scale selection properties of generalized scale-space interest point detectors, Journal of Mathematical Imaging and vision, vol. 46(2), pp. 177-210, 2013. doi:10.1007/s10851012-0378-3. [33] G. S. Shokouh, et al., Ridge detection by image filtering techniques: a review and an objective analysis, Pattern Recognition and Image Analysis, vol. 31, issue 3, pp. 551-570, 2021. doi:10.1134/S1054661821030226. [34] J-L. Starck, and F. Murtagh, Astronomical image and data analysis, Astronomy and Astrophysics
Library, Second edition, Springer, 2007.