In this paper we presented the realization of the data mining approach related to the metadata of astronomical files from the big archives. Each astronomical file has the commonly defined structure, which contains the especial format of the metadata. Such necessary astronomical information, which is required for the proper storing, data mining, processing, analyzing under research. This realization was implemented as software called “Telescope” using the C# programming language, .NET platform, Windows Forms technology and equipped with the MDB database file for Microsoft Access DBMS. The software has two modes: console mode for the automated integration with the processing pipelines and mode with a graphical user interface (GUI) for the visualization of processing and the additional useful features. The Telescope software was designed for mining the big astronomical data from the different archives, parsing the metadata from each astronomical file, and collecting it with the further insertion into the database. Such parsed data were used for the different purposes of the astronomical image processing and machine vision. The Telescope software was developed during research under the CoLiTec project and was tested with the astronomical files from several archives on the different observatories. Also, the implemented and installed on the astronomical image processing pipelines in such observatories. Data mining, big data, metadata, database, image processing, machine vision, C#, .NET, permissions, etc.).
• • •
Almost all astronomical frames are made by the CCD-camera [1] and can be received from the different sources: archives, servers, predefined series of frames, Virtual Observatories [2], clusters, etc. Each software for preparing the astronomical frames creates them as the digital files in the FITS (Flexible Image Transport System) format [3].
This format is a digital files format for storing and transferring of their image and metadata (spreadsheets). Metadata is a kind of data, which provides the information about other data, except the original data content. There are a lot of different types of metadata, as following [4]:
Descriptive metadata is an information about the resource, which is used for identification and includes the elements, like author, title, abstract, and keywords.
Structural metadata is an information about the data containers and how the objects are collected. It includes the elements, like relationships, versions, types, etc. publishing the statistical data.
Administrative metadata is an information for managing resources (creation date, edition date, Reference metadata is an information about the static data, references to them, and contents.
Statistical metadata is an information about the processes for collection, producing, and
2023 Copyright for this paper by its authors. • Legal metadata is a legal information about the copyright, creator, and licensing.
The main purpose of metadata is to provide an information about the different aspects of original data and to summarize a basic information about it, which make tracking and processing it easier. The examples of metadata are as following: time and date of data creation, its meaning and purpose, creator or author, location, file size, used standards, sources, quality, etc.
For example, the digital image includes the metadata, which describes image size, its color depth, resolution, creation time and date, exposure time, etc. A metadata of the text document includes an information about author, processing time of document, short summary, etc. The web pages includes metadata, which describes a description of page content, and keywords linked to it.
In astronomy metadata is used for the different image processing and machine vision purposes [5], like analyzing, acquiring, pre-processing, processing, and extraction of high-dimensional astronomical information [6].
Such purposes [7] are focused on but not limited to the following tasks: brightness equalization and background alignment [8], object’s images detection [9], moving objects detection [10], astrometry of object’s image [11], photometry of object’s image [12], the estimation of the object’s image and motion parameters [13], reference objects cataloging [14], objects recognition [15], time series analysis [16], Wavelet coherence analysis [17] and others.
There are different types of astrophysical objects that can be detected, like galaxies, stars, robots [18, 19], drones [20], rockets, satellites [21], and even comets or asteroids [22].
In this paper we presented a description of the astronomical metadata from the real examples of CCD-images [23], usage in the data mining approach from the big archives, and its implementation as a developed Telescope software, which is designed for mining the big astronomical data from the different archives, parsing the metadata from each astronomical file, and collecting it with the further insertion into the database.
Almost all astronomical frames have a FITS format with standardized file structure and extension. In the common case the astronomical file extensions are: *.fits, *.FITS, *.fts, *.FTS, *.fit, *.FIT. Such FITS format is commonly used for the transformation, transferring, and archiving of astronomical data.
The FITS format was developed by National Aeronautics and Space Administration (NASA) and is accepted as an international astronomical standard and is used by many astronomical and scientific organizations, like International Astronomical Union (IAU) [24], and other national and international organizations that deal with the astronomy or related scientific fields.
The FITS format is commonly used for the storing the data without the image, like spectrums, photons list, data cubes or even structured data, such as databases with multiple tables. The FITS format includes many provisions to describe the photometric and spatial calibration, as well as image metadata.
The structure of the FITS file consists of a header with metadata and a binary image. The header size is 2880 bytes and contain the list of human readable metadata in fixed string form of 80 symbols. Each string is an ASCII [25] stroke, which contains the pair with key and value, and have the common form: “KEYNAME = value / comment string”. Each header block should be ended with the especial key “END” with the empty value. The example of a header with metadata of the real astronomical FITS file is presented in the Figure 1.
There are the minimum list of the required keywords to make the header and the whole FITS file valid. They are: • “SIMPLE” (file conforms to FITS standard); • “BITPIX” (bitrade of FITS file, bits per pixel); • “NAXIS” (number of axes); • “NAXIS1” (number of points along axe 1); • “NAXIS2” (number of points along axe 2); • “END”.
Under the research in scope of the CoLiTec project [26] we have developed the Telescope software for mining the big astronomical data from the different storages and archives, parsing the metadata from each astronomical file, and collecting it with the further insertion into the database. Such parsed data were used for the different purposes of the astronomical image processing and machine vision. 3.1.
The Telescope software realized the different data mining tasks, like receiving, storing, selecting, preprocessing, transforming, useful data extraction, classification, and knowledge discovery in databases (KDD) [27]. The following stack of technologies were used for the software development: C# programming language, .NET platform, Windows Forms technology and MDB database file for Microsoft Access database management system (DBMS). The Windows Forms (WinForms) is an opensource and free graphical library, which is in a scope of the Microsoft .NET Framework [28] and play the role as a platform for developing the client applications for desktop, laptop, and tablet PCs.
As a database the developers have selected a MDB database file as a native format for the Microsoft Access DBMS with “.mdb” extension [29]. There are different fields located in several tables of the MDB database file in the Telescope software that represent the necessary for research metadata of the astronomical file. The list of such fields with their description and types including the system fields are presented in the Table 1. • searching for the metadata in a range by the especial criteria in the database (“find [RAfrom] [RAto] [DEfrom] [DEto] [PathToFile.txt]”); • exporting the MDB database file to the user’s local folder (“export [PathToFolder]”); • importing the MDB database file from the user’s local folder (“import [PathToDBFile.mdb]”). 3.3.
The mode with GUI of the Telescope software is designed for the independent big astronomical
metadata preparation. The processing pipeline includes the following steps:
• selecting the work folder including subfolders with the different astronomical FITS files (see
Figure 2);
• recurrency searching for the astronomical FITS files according to the extensions in the work
folder and forming the results list (see Figure 3 (left));
• parsing the astronomical FITS files from the results list, extracting the metadata and insertion
it into the MDB database file (see Figure 3 (right));
• finding the different astronomical metadata in the MDB database file and forming the results list
(see Figure 4 (left));
• converting from arcseconds (angular hours / minutes / seconds) to decimal (
In general, positional coordinates of objects can be of two types: decimal Cartesian (x and y) in the image plane and stellar (right ascension RA and declination DE) as angular coordinates in the sky [14]. So, to use the appropriate type of positional coordinates, the following mutual recalculation can be used: = ((( 60 + )/60) + ) ∗ 15 ; = (( + )/60) + ;
60 :
: = { ( : : = { ( ( − − )∗ 60 ( − − )∗ 60
( ) )( − )∗ 60 }, ( )
15
)( − )∗ 60 } ;
(
H, M, S are the angular hours, minutes, seconds accordingly. 3.4.
The relationships between tables and fields in the Microsoft Access solution of the MDB database file integrated into the Telescope software is presented in the Figure 6.
The example of the filled “Fits” table in the MDB database file integrated into the Telescope software with the real astronomical data is presented in the Figure 7.
Each value from metadata in header of the FITS file found by the Telescope software was successfully parsed and filled into the appropriate field in the “Fits” table in the MDB database file for the further using and processing.
All interactions between such a database file and the Telescope software are realized by using the structured query language (SQL) as a programming language for storing and processing information in a relational database. 3.5.
The Telescope software implements the following algorithm for astronomical metadata mining. 1. Selecting the work folder including subfolders with the different astronomical FITS or TXT files. 2. Database import from the pre-filled MDB database file, which was previously exported from the last session. 3. Recurrency searching for the astronomical FITS or TXT files according to the extensions in the work folder. 4. Getting access to the astronomical FITS or TXT files (in case when location of big astronomical data is in the different remote/web archives) and download them. 5. Reading the astronomical FITS or TXT files (for FITS files splitting for two parts: header with astronomical information and body with image bytes). 6. Parsing the astronomical FITS or TXT files. 7. Astronomical data receiving from the astronomical FITS or TXT files (for FITS files from header). 8. Astronomical data converting using the different mathematical methods [30] (e.g., positional coordinates conversion). 9. Astronomical data structurization according to the patterns/classes/clusters based on the statistical modeling [31]. 10. Metadata mining from the astronomical data structure according to the appropriate parameters/fields/properties. 11. Filling in the appropriate fields and tables in database using the metadata accordingly. 12. Database export to the MDB database file (if needed).
The metadata mining algorithm implemented in the Telescope software is presented as UMLdiagram in the Figure 8.
The Telescope software was installed in the different observatories (ISON-NM and ISONKislovodsk observatories, Vihorlat Observatory [8], Mayaki Astronomical Observatory [32, 33]), astronomical archives [2], and Ukrainian Virtual Observatory (UkrVO) [34].
The observatory "ISON-NM observatory" has the 0.4 m SANTEL-400AN telescope with CCD-camera FLI ML09000-65 (3056×3056 pixels, 12 microns).
The observatory "ISON-Kislovodsk" has the 19.2 cm wide-field GENON (VT-78) telescope with CCD-camera FLI ML09000-65 (4008×2672 pixels, 9 microns).
The observatory "Vihorlat Observatory in Humenne" has the Vihorlat National Telescope (VNT) – Kassegren telescope with 1 m main mirror with focal length 8925 mm and CCD-camera FLI PL1001E (512×512 pixels).
The Vihorlat Observatory also has the Celestron C11 telescope – Schmidt-Cassegrain telescope with 28 cm main mirror with focal length of 3060 mm and CCD-camera G2-1600 (resolution 768×512 pixels).
The Mayaki observing station of "Astronomical Observatory" Research Institute of I. I. Mechnikov Odessa National University has the 0.48 m AZT-3 telescope – reflector with focal length 2025 mm and CCD-camera Sony ICX429ALL (resolution 795×596).
Totally was processed up to 1 million astronomical files both archived and original formed from the telescopes with a lot of metadata in their headers. The received information was inserted to the database as a big data and processed by the UkrVO, which also processed a lot of different big astronomical archives [35] both digital and even plates.
The Telescope software with the realization of the data mining approach related to the metadata of astronomical files from the big archives was developed. The Telescope software is implemented using the C# programming language, .NET platform, Windows Forms technology and equipped with the MDB database file for the Microsoft Access DBMS. The SQL was used as a programming language for storing and processing information in a relational database.
The software has two modes: console mode for the automated integration with the processing pipelines and mode with a graphical user interface (GUI) for the visualization of processing and the additional useful features. The Telescope software was designed for mining the big astronomical data from the different archives, parsing the metadata from each astronomical file, and collecting it with the further insertion into the database. Such parsed data were used for the different purposes of the astronomical image processing and even for the Wavelet coherence analysis purposes [36].
The Telescope software was developed during research under the CoLiTec project [37]. It was tested with up to 1 million astronomical files from several archives on the different observatories. Such archives included astronomical files of different formats and types of metadata. All such metadata was parsed and structured, which given us an opportunity to perform the proper metadata mining. Such proper metadata was used in the further research and calculations, where the measurements of each known objects are used to clarify the typical form [38] of image, its orbits, motion parameters and other important astronomical properties on the long historical period.
Also, the Telescope software was successfully implemented and installed on the astronomical image processing pipelines in such observatories.
The authors thank all observatories that provided access to their archives with astronomical data to conduct the current research and test the developed Telescope software.
[3] W. D. Pence, L. Chiappetti, C. G. Page, R. A. Shaw, E. Stobie, Definition of the flexible image transport system (fits), version 3.0, Astronomy & Astrophysics, 524 42 (2010). doi: 10.1051/0004-6361/201015362. [4] J. Furner, Definitions of “metadata”: A brief survey of international standards, Journal of the
Association for Information Science and Technology, 71 6 (2020). doi: 10.1002/asi.24295. [5] R. Klette, Concise computer vision, Springer, London, 2014. [6] V. Akhmetov, et al., New approach for pixelization of big astronomical data for machine vision purpose, in: IEEE International Symposium on Industrial Electronics, 2019, pp. 1706–1710. doi: 10.1109/ISIE.2019.8781270. [7] C. Steger, M. Ulrich, C. Wiedemann, Machine vision algorithms and applications, John Wiley &
Sons, 2018. [8] Š. Parimucha, et al., CoLiTecVS – A new tool for an automated reduction of photometric observations, Contributions of the Astronomical Observatory Skalnate Pleso, 49 2 (2019) 151153. [9] L. Mykhailova, et al., Method of maximum likelihood estimation of compact group objects location on CCD-frame, Eastern-European Journal of Enterprise Technologies, 5 4 (2014) 16-22. doi:10.15587/1729-4061.2014.28028. [10] S. Khlamov, et al., Development of computational method for detection of the object's near-zero apparent motion on the series of CCD–frames, Eastern-European Journal of Enterprise Technologies, 2 80 (2016) 41-48. doi: 10.15587/1729-4061.2016.65999. [11] V. Akhmetov, et al., Astrometric reduction of the wide-field images, Advances in Intelligent
Systems and Computing, 1080 (2020) 896–909. doi: 10.1007/978-3-030-33695-0_58. [12] V. Savanevych, et al., CoLiTecVS software for the automated reduction of photometric observations in CCD-frames, Astronomy and Computing, 40 15 (2022). doi: 10.1016/j.ascom.2022.100605. [13] S. Khlamov, V. Vlasenko, V. Savanevych, O. Briukhovetskyi, T. Trunova, V. Chelombitko, I.
Tabakova, Development of computational method for matched filtration with analytic profile of the blurred digital image, Eastern-European Journal of Enterprise Technologies, 5 119 (2022) 24–32. doi: 10.15587/1729-4061.2022.265309. [14] V. Savanevych, et al., Selection of the reference stars for astrometric reduction of CCD-frames, Advances in Intelligent Systems and Computing, 1080 (2020) 881–895. doi: 10.1007/978-3-03033695-0_57. [15] R. Gonzalez, and R. Woods, Digital image processing, 4th edition, NY: Pearson, 2018. [16] L. Kirichenko, A.S.A. Alghawli, T. Radivilova, Generalized approach to analysis of multifractal properties from short time series, International Journal of Advanced Computer Science and Applications, 11 5 (2020) 183–198. doi: 10.14569/IJACSA.2020.0110527. [17] M. Dadkhah, et al., Methodology of wavelet analysis in research of dynamics of phishing attacks, International Journal of Advanced Intelligence Paradigms, 12 (2019) 220-238. doi: 10.1504/IJAIP.2019.098561. [18] A. Tantsiura, et al., Evaluation of the potential accuracy of correlation extreme navigation systems of low-altitude mobile robots, International Journal of Advanced Trends in Computer Science and Engineering, 8 5 (2019) 2161–2166. doi: 10.30534/ijatcse/2019/47852019. [19] N. Yeromina, V. Tarshyn, S. Petrov, et al., Method of reference image selection to provide highspeed aircraft navigation under conditions of rapid change of flight trajectory, International Journal of Advanced Technology and Engineering Exploration, 8 85 (2021) 1621–1638. doi: 10.19101/IJATEE.2021.874814. [20] M. Ivanov, et al., Effective informational entropy reduction in multi-robot systems based on realtime TVS, in: IEEE International Symposium on Industrial Electronics, 2019, pp. 1162–1167. doi: 10.1109/ISIE.2019.8781209. [21] V. Akhmetov, et al., Cloud computing analysis of Indian ASAT test on March 27, 2019, in: Proceedings of the 2019 IEEE International Scientific-Practical Conference: Problems of Infocommunications Science and Technology, PIC S and T 2019, 2019, pp. 315–318. doi: 10.1109/PICST47496.2019.9061243. [22] S. Khlamov, V. Savanevych, Big astronomical datasets and discovery of new celestial bodies in the Solar System in automated mode by the CoLiTec software, Knowledge Discovery in Big Data from Astronomy and Earth Observation, Astrogeoinformatics: Elsevier, 2020, pp. 331–345. doi: 10.1016/B978-0-12-819154-5.00030-8. [23] G. K. Adam, et al., Embedded microcontroller with a CCD camera as a digital lighting control system, Electronics, 8 1 (2019) 33. doi: 10.3390/electronics8010033. [24] J. Andersen, D. Baneke, C. Madsen, The International Astronomical Union: Uniting the
Community for 100 Years, Springer, 2019. [25] H. T. Elshoush, M. M. Mahmoud, A. Abdelrahman, A new high capacity and secure image realization steganography based on ASCII code matching, Multimedia Tools and Applications, 81 4 (2022) 5191-5237. doi: 10.1007/s11042-021-11741-y. [26] S. Khlamov, et al., Data Mining of the Astronomical Images by the CoLiTec Software, CEUR
Workshop Proceedings, vol. 3171, 2022, pp. 1043–1055,. [27] Y. Zhang, Y. Zhao, C. Cui, Data mining and knowledge discovery in database of astronomy,
Progress in Astronomy, 20 4 (2002) 312-323. [28] S. Bisser, Introduction to the Microsoft Bot Framework, Microsoft Conversational AI Platform for Developers, 2021, pp. 25-66. doi: 10.1007/978-1-4842-6837-7_2. [29] J. Eckstein, B. R. Schultz, Introductory relational database design for business, with Microsoft
Access, John Wiley & Sons, 2018. [30] V. Savanevych, et al., Mathematical methods for an accurate navigation of the robotic telescopes,
Mathematics, 11 10 (2023) 2246. doi: 10.3390/math11102246. [31] V. Shvedun, S. Khlamov, Statistical modelling for determination of perspective number of advertising legislation violations, Actual Problems of Economics, 184 10 (2016) 389-396. [32] S. Kashuba, et al., The Simeiz plate collection of the ODESSA astronomical observatory, in:
Proceedings of the 11th Bulgarian-Serbian Astronomical Conference, 2018, pp. 207-216. [33] V. Troianskyi, P. Kankiewicz, D. Oszkiewicz, Dynamical evolution of basaltic asteroids outside the Vesta family in the inner main belt, Astronomy and Astrophysics, 672 (2023). doi: 10.1051/0004-6361/202245678. [34] I. B. Vavilova, et al., Astrometry and photometry of asteroids from the UkrVO database of astroplates, in: Proceedings of the IAU, 15 364 (2021) 239-245. doi: 10.1017/S1743921322000047. [35] L. Kirichenko, P. Zinchenko, T. Radivilova, Classification of time realizations using machine learning recognition of recurrence plots, Advances in Intelligent Systems and Computing, 1246 (2021) 687–696. doi: 10.1007/978-3-030-54215-3_44. [36] V. Baranova, et al., Wavelet Coherence as a Tool for Studying of Economic Dynamics in Infocommunication Systems, in: IEEE International Scientific-Practical Conference Problems of Infocommunications, Science and Technology (PIC S&T), 2019, pp. 336-340. doi: 10.1109/PICST47496.2019.9061301. [37] I. Kudzej, et al., CoLiTecVS – A new tool for the automated reduction of photometric observations, Astronomische Nachrichten, 340 3 (2019) 68-70. doi: 10.1002/asna.201913562. [38] V. Savanevych, et al., Formation of a typical form of an object image in a series of digital frames, Eastern-European Journal of Enterprise Technologies, 6 120 (2022) 51–59. doi: 10.15587/1729-4061.2022.266988.