The huge engineering revolution is closely connected with the 21st century and characterized by the terrific technological progress. Such progress causes the creating of a large number of the various data that fed in an online or offline modes in the form of data streams, predefined sets, series, video, etc. [1]. Such all data as a huge number of files, streams, memory grow and grow. It requires a lot of storage space like data centers, servers, archives [2], Virtual Observatories [3,4], etc. This ability of data to grow is ahead of all computing abilities of the already existed computers/machines/servers. In this case the processing optimization of the data streams, sets, data is very important by using only required input information to help computers/machines/servers to work more productively.
The data mining and knowledge discovery approaches become more and more popular and actual in the different research and experiments to improve productivity and efficiency of the processing algorithms in the different fields of interest. Astronomy as a research field of interest is not an exception [5].
So, what is the data mining approach? It is a process of the information receiving from the large data sets by using the extracting or discovering patterns and involving the methods at the intersection of disciplines like computer science, statistics, machine learning and database systems.
The data mining carries out about the information extracting using the intelligent methods from a data set to transform it into the obvious structure for the further use. The data mining is an analysis step of the "knowledge discovery in databases" (KDD) process [6]. The full flow with all intermediate stages of the "knowledge discovery in databases" process is presented in the Figure 1.
The main goal of the data mining is extraction of the potentially useful information for the knowledge from the given large input data sets/streams/video using the appropriate associations, relationships, or recognition patterns [7]. Then the received data is transformed into the subsets with known and required structure. These formed subsets are used for the future effective analysis and usage.
In this paper we presented the few aspects of the "knowledge discovery in databases" process related to the data mining stage in astronomy, analyzed and reviewed data mining approaches. The examples of astronomical sources of big data, instruments, kind of information, processing algorithms are provided. The goal of current research is to apply the developed CoLiTec (Collection Light Technology) software [8] to the data mining purposes with the astronomical images.
The data mining and knowledge discovery become areas of growing significance. Such growing was caused by the increasing needs for KDD techniques for the different directions, like databases, knowledge gathering, machine learning, statistics, data visualization, and high performance computing. The data mining and knowledge discovery also is highly useful for the artificial intelligence techniques in many areas, like industry, commerce, government, education, astronomy and so on [9].
The data mining in astronomy is a very powerful approach, which has a big potential for the fully exploitation the exponentially increasing amount of data and promises an excellent scientific progress. But with the wrong using it can be little more than the black box application of complex computing algorithms that can provide very questionable results. So, the data mining can be much more powerful tool, which is pretty good adapted for the astronomical tasks, instead of accurate selection or continually modification an appropriate processing algorithm.
Nowadays, in the big data era, there are different fields in astronomy that are vital for the dealing with big data and data mining issues. They are astroinformatics, astrostatistics, astrochemistry, etc. All progressive astronomers, researchers, scientists are ready to face the technological challenges and opportunities provided by the massive data volume and open exciting perspectives for the new astronomical discoveries by applying of the advanced data mining approach. The diversity of scientific tasks and complexity of the astronomical big data provoke the development of innovative processing algorithms and methods as well as a highly usage of the Information and Communications Technologies (ICT) [10].
There are a lot of different scientific programs, projects, databases, virtual observatories [4], services that solve different research tasks by using the data mining and "knowledge discovery in databases" approaches. The DAME (DAta Mining & Exploration) program includes a set of webbased services that perform scientific investigation and analysis of the astronomical big data sets [11]. The engineering design and requirements are constructed on the new paradigm of web-based resources that realize the efficient data mining framework in the data-centric era [2].
The data mining problems of data analysis and visualization [12] from the huge stellar catalogues that contain billions of objects are more difficult because of appearing the massive data sets, like 2MASS (Two Micron All Sky Survey) [13], WISE (Wide-field Infrared Survey Explorer) [14], ESA Euclid space mission [15]. Such astronomical big data is received by the modern robotic telescopes, like Pan-STARRS (Panoramic Survey Telescope and Rapid Response System) [16], ESA GAIA (Global Astrometric Interferometer for Astrophysics) space mission [17], Thirty Meter Telescope (TMT) [18].
Especial attention can be paid to the SDSS (Sloan Digital Sky Survey) [19], as to the most successful sky survey in the astronomy history. The SDSS project has formed the most detailed threedimensional maps of the Universe with deep multi-color images of 1/3 of the sky, and spectra for more than three million astronomical objects [20].
One more interesting and more huge wide-field survey with reflecting telescope is under construction and called Large Synoptic Survey Telescope (LSST) [21]. It has a primary mirror with diameter 8.4 meters and includes three mirrors with a very wide field of view (FOV) of 3.5-degree, which is presented in the Figure 2. The LSST science database is focused on the following goals:
• scalability (at petabytes scales) of existing machine learning and data mining algorithms; • development of grid with enabling the parallel data mining algorithms; • designing a robust system for brokering classifications from the event pipeline; • indexing of multi-attribute and multi-dimensional petascale astronomical databases for the rapid querying; • multi-resolution methods (object classification, outlier identification, anomaly detection).
The data mining purposes regarding the astronomical image processing are focused on but not limited to the following tasks: brightness equalization [22], background alignment [23], object's images detection [8], moving objects detection [8], astrometry of objects (positional object coordinates estimation in the image that are re-calculated into the sky position) [24], photometry of objects (object's brightness estimation in the magnitude) [25], the parameters determination of the object's image and apparent motion [8], reference objects cataloging [26], objects recognition [27], Wavelet coherence analysis [28] and others.
The data mining of astronomical images includes the different major areas of application of image and signal processing like the following [27].
• Filtering. The clear raw signals in astronomy are very rarely existed without noise, so the removal of noise is necessary for the future useful data interpretation. In common, the data cleaning is required to bypass the artifacts of instrumental measurements without changing of the complexity of data. • Deconvolution. The signal "deblurring" is also used for reasons that are very similar to filtering, as a preliminary to the data interpretation. Deblurring of the objects motion in images is very important in astronomy, as well as the removing effects of atmospheric blurring, to improve the quality of seeing. • Compression. There are several facts that show the importance of effective and efficient compression technology: long-term storage of astronomical data, developing of detectors for the ever-larger image sizes, research in astronomy is a geographically distributed activity. • Mathematical morphology. The combinations of erosion and dilation operators often provoke the opening and closing operations. So, in the greyscale/boolean images they allow creation of immediately practical framework for the processing. The median function plays such role for the order and rank functions. In this case, the multiple scale mathematical morphology is an immediate generalization of the astronomical images processing [29]. • Edge detection. The gradient information is not very popular information for the astronomical image analysis because of their boolean nature. So, in this case, the objects edges identification is used more often like curves in the image by the brightness changes or discontinuities [30]. • Corner detection. A group of algorithms that are used within computer vision systems to extract certain kinds of the features and infer contents of an image [31]. It is often used for the object's recognition, object's image registration, object's motion detection [8], video tracking, and 3D reconstruction. • Blob (point) detection. The mathematical methods for the region's detection in the image.
Such regions have a difference in brightness and color that are compared to the neighboring regions. The blob is a region with points in which properties are constant or approximately constant, so all points in the blob are like each other [32]. • Ridge detection. The mathematical methods for the ridge's localization in the image that defined as curves whose points are the function's local maximum, like the geographical ridges [33]. • Segmentation and pattern recognition. In astronomy, the segmentation and pattern recognition is used for the object detection while the term feature selection [26] is more popular in areas outside astronomy. In common, they are used for the assignment of the object's images to a proper class by the highlighting of significant features that characterize this class [34]. • Hough and Radon transforms. The detection of curves is required for the many segmentation classes and feature analysis. It does not matter if the signal is faint or strong, the noise is usually the most critical one. The Ridgelet and Curvelet transforms provide the powerful generalizations for resolving such problems [27]. The described above mathematical image processing methods are different but all of them can be used as pre-processing stage of the data mining of astronomical images in the processing pipeline before the main image processing algorithm (object's image recognition, object detection, objects parameters estimation, trajectory detection, trajectory parameters estimation) is applied.
The object classification is an important initial step in the scientific data mining process because it provides the algorithms and methods for organizing the scientific information in a way that can be used to make the appropriate hypotheses and to compare with the existing models.
Because of the small physical size of stars compared to their distance from the observing point, almost all stars are unresolved in the photometric datasets, and thus appear as the point objects in the CCD-image [35]. The galaxies in common case subtend a larger angle, even when they are further away, so appear as the extended objects in the CCD-image. But the other astrophysical objects such as quasars and supernovae also appear as point objects. So, the separation of photometric catalogs into stars, galaxies, and other objects, is an important and difficult task.
The huge number of galaxies and stars in typical surveys requires the morphology separation as a process, which is automated or semi-automated. This task is a well-studied and the several automated approaches for big data analysis were implemented, like for the digitization of the scanned photographic plates by machines such as the Automatic Plate Measuring (APM) [36] and Palomar Digital Sky Survey (DPOSS) [37].
Also, the several data mining methods have already been developed and implemented using the Artificial Neural Network (ANN) [38], mixture modeling [39], where the most methods achieving over 95% efficiency.
In general, such methods are based on the astrophysical object's classification using a set of the measured morphological parameters that are received from the survey photometry, with shape, structure, texture, inclination, arm pitch, color, resolution, exposure, colors, spectra, and other astrophysical information.
The main advantage of these data mining methods is that all such information about each astrophysical object is easily extended and incapsulated into the massive datasets [40].
There is a various morphology of galaxies based on the wide range of different sizes and shapes of them. The most popular system of the morphological classification of galaxies is the Hubble Sequence of spiral, barred spiral, elliptical, and irregular, and galaxies from the different subclasses [41]. This system correlates to the many important physical properties in the formation and evolution of galaxies [42].
The galaxy morphology is a very complex phenomenon, which is correlated to the underlying physics, but it is not unique to any one given process. But, anyway, the Hubble sequence is still actual, even if it being rather subjective and based on the visible-light morphology, which was originally received from the blue-biased photographic plates.
The Hubble sequence was extended in different ways using the data mining approach and ANN applying [38] to predict the galaxies' type at low redshift and finding the equal accuracy to human experts. ANNs were also applied to the higher redshift data to distinguish between normal and peculiar galaxies. Also, the fundamentally topological and unsupervised SOM ANN was used for the galaxy's classification based on the CCD-images, received from the Hubble Space Telescope [43], where the initial distribution of classes is not known. The approach of using the ANNs also was used to determine the morphological types from galaxy spectra [44].
For the galaxy morphology research even the Fourier decomposition was used on the galaxy images implemented with ANNs for the bars detection and types assigning [45].
The different data mining approaches for the astronomical images processing is provided by the CoLiTec software [8], which allows the input data processing in near real time/online mode. This is a very complicated system for the astronomical data sets processing, which includes the different features, user-friendly tools for the processing management, results reviewing [12], integration with online catalogs and a lot of various computational components that are based on the developed methods [8,24,26]. The processing results are also available and can be visualized.
The high level processing pipeline with developed modules and implemented methods of the CoLiTec software is presented in the Figure 3. The processing steps of the CoLiTec software in the pipeline according to the data mining approaches are described below.
The pre-processing step of the CoLiTec software in OnLine Data Analysis System (OLDAS) mode includes the input data set processing as soon as they successfully received from different sources. Such raw data is moderated before the computational process starts. The unsupported and corrupted frames are rejected at this step. The useful information from the input data set is only used during the computational process.
The selected useful information from the input data set is categorized into clusters using the specified attributes. The CoLiTec software uses the different attributes, such as equatorial coordinates, filter type, telescope, investigated object and others. Based on these attributes the necessary information from the input data set is separated into subsets with similar data and stored at the different distributed servers, clusters or even networks.
After clustering process, the created subsets of data are classified by the applying of a known structure of the raw astronomical data that specified in Flexible Image Transport System (FITS) standard by NASA [46]. FITS standard is the most used digital file format in astronomy. Such format is designed especial in form of the image metadata, which includes different scientific data, like astrometric, photometric, calibration information and others. After classification the FITS files are sent to the processing pipeline.
During processing pipeline all received classified FITS files pass through the identification step. At this step all FITS files related to the service master-frame are used for the frame's calibration (e.g., bias, dark, darkflat, flat). Otherwise, if this is a raw light frame the processing pipeline starts computing process.
The computing process in the processing pipeline is managed by the OLDAS and includes two stages: intraframe and interframe processing. The intraframe stage includes the various processes for the image filtration and objects detection. The major goal of the object's detection in the series of images is to recognize the object, its borders and determine the parameters of its image [8].
There are a different recognition patterns or types of the astronomical objects in the image that can be detected: point objects, long objects, blurred objects, objects with flare or intersection with another objects. Such types of objects can be belonged to the galaxy, star, robot [47], drone [48], rocket, satellite [49], and even comet [50] or asteroid [8].
The features of CoLiTec software related to the intraframe stage are described below:
• processing of the very wide field of view (FOW)up to 10 square degrees;
• automated calibration process;
• cosmetic correction process;
• FrameSmooth software for background alignment and brightness equalization [22];
• automated rejection of the worst observations;
• fully automated robust algorithm of astrometric reduction;
• semi-automated algorithm of photometric reduction;
• automated rejection of objects with bad or unclear measurements. The object's image detection process during the intraframe processing is presented in the Figure 4. The interframe stage includes the various processes for detection of the objects motion. The major goal of the moving objects detection in the series of images is to recognize the object's trajectory and determine its parameters [8].
The features of CoLiTec software related to the interframe stage are the following:
• automated detection of the faint moving objects with signal-to-noise ratio (SNR) more than 2.5;
• automated detection of very slow objects with near-zero apparent motion from 0.7 pix./frame; • automated detection of very fast objects with apparent motion up to 40.0 pix./frame. The object's motion detection process during the interframe processing is presented in the Figure 5.
The data mining analysis by the CoLiTec software is performed using the following technological features:
• multi-threaded processing;
• multi-cores systems using with managing the individual treatment processes;
• deciding system, which allows adapting the user settings for the processing;
• notification system, which informs user about the correct results at each processing stage;
• data control managing during processing using the subject mediator.
After pipeline processing and data mining analysis, the CoLiTec software produces the various forms of results representation, including visualization results and reports generation for the different services. To summarize results and for the visual analysis of them, the LookSky viewer with userfriendly GUI is used (see Figure 6).
As a result, about 700,000 observations were made using the CoLiTec software with approach for the data mining of astronomical images. According to these observations the following discoveries were also done:
• more than 1,600 asteroids;
• 5 Near Earth Objects (NEOs);
• 21 Trojan asteroids of Jupiter;
• 1 Centaur (2013 UL10);
• 5 comets (C/2010 X1 (Elenin), P/2011 NO1 (Elenin), C/2012 S1 (ISON) [50], P/2013 V3 (Nevski), C/2017 T3 (ATLAS) [51]). All mentioned above observations and discoveries are approved and confirmed by the Minor Planet Center (MPC) as an official organization for the observing and reporting on minor planets under the auspices of the International Astronomical Union (IAU). Below are presented the several examples of the series of images processing after the data mining stage in the processing pipeline of the CoLiTec software [8]. Such data mining stage was performed using the astronomical information from the different astronomical archives [4] and at the real-time receiving of CCD-images right from the telescopes during observations.
The few processing results of the automated calibration, cosmetic correction, background alignment and brightness equalization processes are presented in the Figure 7. The few processing results of the object's measurements mining, object's image detection, and object's apparent motion detection processes are presented in the Figure 8. For all detected by the CoLiTec software objects a lot of apparent motion parameters are determined. Some of them are the object's velocity V in the equatorial system (right ascension RA and declination DE) and in the cartesian system according to two axis x and y, and the distance S, which was passed by the object from the first positional measurement till the last one in the observing series of images.
The few processing results with the determined apparent motion parameters of the real Solar System objects (SSOs) are presented in the Table 1. The values of S distance, which was passed by the different objects from the first positional measurement till the last one in the observing series of images, shows that such objects have near-zero apparent motion, and the CoLiTec software successfully detected these objects and estimates the positional parameters and the motion parameters.
The very fast technological progress, networks of automated ground-and space-based observation systems, new scientific programs, surveys, projects lead to the fast growing of astronomical data.
We presented the developed CoLiTec software [8], which is used for the online processing of the different types of astronomical information using the data mining approaches. As described in the paper the knowledge discovery in databases with the data mining analysis step is applicable and very practical for the optimization of data stream processing and receiving only useful information. It allows applying only necessary input data to improve the computing abilities of machines.
The CoLiTec software realizes different data mining principles and stages of processing, such as anomaly detection, pre-processing, clustering, classification, identification, processing (edge detection, segmentation, recognition patterns, object [52] and motion detection) and summarization.
Such data mining principles in the CoLiTec software are implemented by the especial developed mathematical methods and components for the intraframe and interframe processing, astronomical data mining from the different on-line services, archives [2], Virtual Observatories [4], visualization [12] under the CoLiTec project [8].
The scientific novelty of the current research is that the CoLiTec software is the first astronomical software, which fully implements all steps according to the data mining approach and pipeline to process the different astronomical data.
Using the described in the paper data mining approaches the CoLiTec software helps with countless observation and discoveries of SSOs, which are confirmed by the MPC and the appropriate Minor Planet Electronic Circulars (MPECs).
The authors thank all observatories, online services and tools that provided data to conduct the current research for testing the developed CoLiTec software with realization of the data mining approaches.