Astronomical data mining plays a crucial role in modern astrophysics, enabling the discovery and analysis of celestial phenomena through the processing of vast observational datasets. As the volume of astronomical data continues to grow exponentially, the need for efficient, automated decision-making within data processing pipelines becomes increasingly critical. This paper presents an artificial intelligence (AI) driven decision-making module designed to optimize workflow management and anomaly detection in large-scale astronomical data processing. Integrated with a PostgreSQL-based logging system, it enhances real-time monitoring, streamlines error identification, and improves overall data processing efficiency. The proposed approach leverages advanced computational techniques to automate key decision points, reducing manual intervention and mitigating the risk of processing failures. We outline the methodology and architectural framework, detailing its implementation and integration into existing data processing pipelines in the Lemur software of the CoLiTec (Collection Light Technology) project. A comparative analysis with conventional techniques highlights the advantages of the proposed system in terms of accuracy, computational efficiency, and robustness. Experimental results demonstrated significant improvements in identifying anomalies, optimizing resource allocation, and enhancing the reliability of automated decision-making process. The proposed hybrid rule-based + AI approach demonstrated a 65% improvement in decision-making speed and a 50% reduction in failure recovery time compared to traditional rule-based monitoring. The findings underscore the potential of AI-driven decision modules in advancing astronomical research by enabling more efficient and accurate data analysis within large-scale observational studies.
The increasing volume of astronomical big data from space and ground-based observatories [
Traditional approaches rely on rule-based monitoring, which is often inefficient, inflexible, and
prone to errors in large-scale operations. Such rule-based monitoring systems struggle with the
growing complexity and volume of observational logs, making real-time decision-making [
Artificial intelligence (AI) [
Artificial intelligence has revolutionized multiple fields by enabling automation, predictive modeling, and intelligent decision-making. The literature analysis highlights AI's transformative impact across domains, including informational technology (IT), economics, robotics, healthcare, finance, autonomous systems, environmental science and even astronomy.
In the IT sector an AI strengthens cybersecurity, optimizes data management, automates software
development, and enhances cloud computing efficiency [
Financial institutions leverage AI for risk management [
AI plays a crucial role in climate modeling, weather prediction, and disaster management. Neural
networks enhance climate simulations by refining atmospheric models and predicting extreme
weather events with greater accuracy [
AI transforms learning by enabling personalized education [
Several AI-driven approaches have been applied to astronomical data mining. Machine vision
techniques [
In paper authors proposed the framework, which bridges the gap by integrating real-time log evaluation, AI-driven anomaly detection, and automated decision execution to optimize astronomical data processing (Table 1).
No Yes • • • • • • The proposed framework is an advanced, AI-powered decision-making system for astronomical data mining, designed with a modular and scalable architecture to efficiently handle vast and continuously growing datasets of astronomical big data. By integrating intelligent automation, realtime data processing, and adaptive learning capabilities, it optimizes workflow efficiency, enhances anomaly detection, and ensures robust decision-making across diverse astronomical research applications.
The architecture of proposed framework consists of three main components:
PostgreSQL database for structured log storage; decision-making module leveraging AI models to assess message logs and update process states; automated workflow control for handling failures, timeouts, and success conditions.
These components work together to ensure efficient log processing, anomaly detection, and automated decision execution (Figure 1).
The core of proposed framework is a decision-making module responsible for analyzing log messages, identifying process states, and determining the appropriate actions. The module operates using a combination of rule-based logic, machine learning models, and anomaly detection techniques.
Key AI techniques employed include: • • • •
Rule-based decision processing. This approach utilizes predefined decision trees and state transitions to analyze log messages. For instance, if an inputFail log is detected, the system Supervised learning for failure prediction. Historical log data is utilized to train machine learning models, such as random forests and support vector machines. These models predict the probability of processing failures, timeouts, or anomalies based on past events.
Reinforcement learning for dynamic thresholds. This technique adapts timeouts, error tolerance, and retry strategies in realNLP for log classification. NLP enables the extraction of patterns from textual log messages, facilitating the categorization of errors, warnings, and successes. This approach aids in detecting rare failure cases that might elude traditional rule-based logic.
The decision-making module processes incoming logs according to predefined AI-driven rules [
The proposed framework processes logs in five sequential stages: 1. Process initiation and logging: each data processing zone generates log entries stored in the PostgreSQL database, log messages such as startModule, inputPass, and endProcessFail are tracked; 2. AI-based anomaly detection: a supervised random forest model predicts failure probabilities based on historical log data: failure = ( 1, 2, … , ), (1) where represents extracted log features; 3. Timeout-based failure detection: if no messages arrive within a predefined threshold, the system computes the following: stalled = current − lastMessage , (2) where current and lastMessage is time of the current and last message log; if > , a terminated log entry is generated; 4. Reinforcement learning for adaptive decision-making dynamically adjusts retry intervals and error handling mechanisms based on system performance; 5. User alerting and decision execution: upon anomaly detection, the monitoring dashboard alerts users and logs the final state transition in the database.
The proposed framework provides a real-time monitoring interface that allows users to track processing states, inspect logs, and manually override decisions if necessary.
The monitoring dashboard incorporates several key features that augment its functionality and usability. It provides real-time log visualization, displaying incoming messages, processing states, and anomaly alerts. Users possess the capability to filter these visualizations by zone, module, and message type, thereby enabling tailored insights into system activities.
Furthermore, the dashboard incorporates an automated alerting system that promptly informs users about the different anomalies, failures, and prolonged processes via various channels, including email, web notifications, and external application programming interface (API) integrations. Additionally, users have the option to manually intervene in processes, with options to restart terminated modules, modify thresholds, or approve AI-generated decisions. All manual interventions are meticulously documented, facilitating system optimization and future decision-making.
The dashboard also permits user-configurable settings, allowing customization of decision rules, timeouts, and message filtering. This adaptability is further enhanced by the capability to dynamically adjust AI thresholds based on current system load, ensuring optimal performance.
The proposed framework utilizes a PostgreSQL database for structured logging, process tracking, and decision-making. The schema consists of several interrelated tables that store information about zones, messages, processing states, and decision rules.
The database structure includes the following core tables: • • • • control.zonestates table represents the state of zone processing, tracking different stages from creation to completion; control.zones table tracks individual data processing zones, including timestamps for start and end processing (Table 2); control.keywords table contains predefined keywords (Table 3) used for log classification and anomaly detection; control.messagetypes table defines the types of messages, which are allowed for processing; control.messages table logs all system messages, including timestamps, associated modules, and keywords ( • • The following entity-relationship diagram (ERD) illustrates the database schema (Figure 2).
Several database functions assist in retrieving zone IDs, updating states, and processing logs dynamically: • • • •
BigInt GetZoneId(varchar(255) zonePath) function creates a new entry in control.zones if the
ID, otherwise if the zone already exists, it returns the existing ID, which should be used during inserting new record in the control.messages table (field zoneId); void StartZoneProcessing(bigint zoneID, int fitscount) function updates startProcessingDate, resets endProcessingDate, sets trackscount to zero, and changes zonestateid to Processing (if no zones with ID found, updating is not performed); void EndZoneProcessing(bigint zoneID, int state, int trackscount) function updates endProcessingDate, zonestateid, and trackscount based on completion results (if no zones with ID found, updating is not performed); void UpdateZoneState(bigint zoneID, int state) function modifies the zonestateid to reflect the current processing status (if no zones with ID found, updating is not performed).
Our decision-making process leverages a hybrid approach that combines rule-based systems for structured logic and interpretability with machine learning models for adaptive pattern recognition and predictive analytics, ensuring both accuracy and flexibility in handling complex astronomical data: • • • natural language processing extracts patterns from log messages to identify anomalies; decision trees and random forests predict the likelihood of process completion and flag failures; reinforcement learning optimizes decision-making by dynamically adjusting thresholds based on historical performance.
A web-based monitoring dashboard offers visualizations of process states, failure alerts, and log analytics. It supports manual intervention, allowing researchers to override AI decisions when necessary.
This comprehensive dashboard is an indispensable tool for organizations relying on AI-driven processes. It provides real-time visualizations of process states, enabling instant comprehension of complex data streams.
Advanced failure alerts immediately notify users of any anomalies or disruptions in system performance.
The dynamic and user-friendly interface allows users to delve into log analytics, offering insights into historical data and trends for predictive maintenance and informed decision-making. One key
While AI systems autonomously manage operations, the dashboard empowers researchers and operators to override AI decisions whenever required.
This ensures human oversight remains a crucial part of the operational workflow, allowing experts to apply their judgment in situations where AI may fail, such as interpreting nuanced data or responding to unexpected scenarios.
Furthermore, the dashboard can be customized to meet diverse industrial requirements, It is equipped with robust security protocols to protect sensitive data, maintaining confidentiality and integrity.
As a result, this tool enhances operational efficiency and fosters trust in AI technologies by ensuring transparency and accountability in automated processes.
The proposed framework automates workflow using an AI-based decision engine that monitors logs in real-time. The decision-making module evaluates input, processing, and output logs using predefined rules and machine learning models to optimize the workflow.
This structured approach ensures the following advantages: • • • efficient error handling ensures immediate detection and logging of processing failures; automated decision-making dynamically determines whether to retry, escalate, or terminate processing; scalability support of the high-volume astronomical data processing with minimal manual intervention.
The following diagram illustrates the decision-making workflow visually (Figure 3).
The proposed framework, developed with the utmost care, incorporates an extensive suite of features designed to enhance the efficiency and accuracy of data processing within astronomical pipelines. Beyond its core decision-making capabilities, the system provides advanced functionalities for managing and analyzing log messages.
The system empowers users to sort and cleanse database messages, ensuring that only pertinent and structured data is retained for processing. This eliminates redundant or outdated entries, thereby reducing database clutter and optimizing query performance.
Furthermore, proposed framework incorporates filtering mechanisms that enable users to refine messages based on specific criteria, such as zones, message types, or processing states. This selective retrieval process optimizes workflow efficiency by allowing operators to focus on critical information.
Another noteworthy feature is the dynamic manual update functionality, which enables users to refresh displayed messages in real-time. This ensures that the most recent log entries are always readily available for review.
Additionally, the system supports zone-based message retrieval, allowing users to inspect logs associated with specific data processing zones. This facilitates debugging and troubleshooting by providing insights into localized processing events.
To enhance user experience and system adaptability, proposed framework incorporates a customizable settings module, enabling users to configure thresholds, logging levels, timeout rules, and processing preferences. These settings ensure that the system can be tailored to accommodate diverse operational requirements and research needs.
By integrating these advanced sorting, filtering, manual updating, zone-based retrieval, and userconfigurable settings, proposed framework offers a robust and scalable solution for automating and optimizing decision-making in astronomical data mining workflows.
The proposed framework enhances traditional log-based monitoring systems by integrating artificial intelligence-driven decision-making capabilities. This integration merges the strengths of rule-based logic and AI learning, enabling enhanced flexibility, improved failure detection, and diminished human intervention.
Below is a comparison with other commonly used techniques (Table 5).
The proposed framework is designed to efficiently process large-scale astronomical datasets while maintaining low latency, high accuracy, and fault tolerance. As the volume of observational data grows, the system must handle increasing log entries, maintain real-time decision-making, and ensure reliable execution of data processing tasks.
Scalability is achieved through database indexing, partitioning, and caching techniques. The system utilizes a B-tree index in PostgreSQL to accelerate log retrieval times, reducing the complexity of queries from ( ) to (log ).
For instance, a dataset containing 10 million log entries is indexed in under 2 seconds, compared to an unindexed query that takes over 30 seconds.
To further enhance scalability, horizontal partitioning is implemented by segmenting logs based on processing zones and timestamps. When processing 100 zones simultaneously, each generating 1,000 log entries per minute, partitioning ensures that queries remain efficient, preventing database slowdowns.
In real-world scenarios, proposed framework has been tested in scope of the Collection Light Technology (CoLiTec) project [33] with an astronomical observation dataset consisting of 50,000 logs per hour, demonstrating a 45% reduction in query execution time through caching and optimized indexing strategies.
Performance benchmarks indicate that proposed framework outperforms traditional rule-based logging systems [34] by a significant margin. The AI-based decision-making module improves error detection accuracy using a combination of supervised learning and anomaly detection techniques. making time is modeled as: (3)
(4) total = query + analysis + decision, where
is the time required to retrieve log data from the database; is the time needed to process logs using AI models; is the execution time for generating a final system response.
For an astronomical dataset with 500,000 logs, traditional rule-based systems [35] require an average processing time of 520 ms, while proposed framework reduces this to 180 ms, demonstrating a 65% improvement.
The system enhances automated failure recovery by leveraging reinforcement learning techniques to adjust error-handling strategies dynamically. If a processing module fails, proposed framework assesses historical failure patterns and determines whether to retry execution, escalate the issue, or terminate the module.
For example, in a dataset where 5% of processing modules encounter unexpected failures, proposed framework reduces failure resolution time from 30 minutes to under 10 minutes by automating recovery processes. The effectiveness of this approach is measured using the Mean Time to Recovery (MTTR) equation: = ∑ repair failures , where
is the time taken to resolve each failure; is the total number of failures observed.
With reinforcement learning, MTTR is reduced by 50%, significantly improving system uptime.
Consider an application in which proposed framework is deployed to monitor a radio telescope array collecting signals from deep-space objects. The system processes 300 GB of raw data per day, generating 2 million log messages related to signal calibration, noise filtering, and anomaly detection.
Without optimization, traditional log processing systems require approximately 5 seconds per query, making real-time decision-making impractical. With AI-based approach in the proposed framework, query times are reduced to 1.2 seconds, allowing for near-instantaneous responses to anomalies such as unexpected signal drops or sensor malfunctions.
By combining high-performance AI decision-making, scalable database structures, and adaptive learning techniques, proposed framework ensures that the system remains reliable, efficient, and capable of handling the increasing demands of modern astronomical research.
To summarize the experiment results, the following criteria were analyzed (Figure 4): • • • • processing time reduction (lower is better): proposed framework reduces processing time significantly, achieving a 65% improvement. automated failure recovery (higher is better): proposed framework improves failure recovery efficiency by 50%. error detection accuracy (higher is better): traditional rule-based systems have 72% accuracy, while proposed framework enhances it to 92% using AI-based anomaly detection. system downtime reduction (lower is better): proposed framework reduces downtime by 50%, ensuring greater system availability.
The paper introduced an AI-powered decision-making system for automated log evaluation, anomaly detection, and process monitoring during the astronomical data mining process.
The research demonstrated that integrating AI-driven decision-making with a structured logging system significantly enhances the efficiency and reliability of different types of astronomical data processing, like knowledge discovery in databases, data mining, image recognition [36], machine vision [37], image filtration [38, 39], object detection, etc. The hybrid rule-based and AI approach proved to be highly effective, achieving substantial improvements in decision-making speed and failure recovery time. The proposed decision-making process is a substantial advancement in astronomical data mining pipelines.
By automating anomaly detection and optimizing workflow management, the proposed system minimizes manual intervention and enhances the robustness of large-scale observational data analysis. By addressing critical challenges and leveraging machine learning [40], the framework improves both efficiency, accuracy and data transmission speed [41]. These advancements pave the way for more efficient data handling in astronomical research, supporting the discovery and classification of celestial phenomena with greater accuracy. The proposed hybrid rule-based + AI approach demonstrated a 65% improvement in decision-making speed and a 50% reduction in failure recovery time compared to traditional rule-based monitoring [42].
Future work will focus on refining AI models, expanding adaptability to diverse datasets, and further integrating decision-making automation into broader astronomical data processing frameworks. Also, the authors plan to focus on scaling the pipeline for even larger datasets and incorporating additional data sources to further validate its effectiveness. The enhancing decentralized [43] multi-node distributed processing for real-time log analysis also will be useful.
The research was supported by the Ukrainian project of fundamental scientific research -zero and locally constant motion by optical
The authors have not employed any Generative AI tools. -2026 years. [15] S. Khlamov, et al., Statistical modeling for the near-zero apparent motion detection of objects in series of images from data stream, in: Proceedings of the 12th IEEE ACIT 2022, Ruzomberok, Slovakia, September 26th 28th, 2022, pp.126-129. doi: 10.1109/ACIT54803.2022.9913151. [16] M. Abdel-Basset, R. Mohamed, S. A. A. Azeem, et al., Kepler optimization algorithm: A new wledge-Based [33] S. Khlamov, et al., Machine Vision for Astronomical Images using The Modern Image Processing Algorithms Implemented in the CoLiTec Software, Measurements and Instrumentation for Machine Vision (2024) 269-310. doi: 10.1201/9781003343783-12. [34] O. K. Aimiyekagbon, et al., Rule-based Diagnostics of a Production Line, PHM Society European
Conference 6 1 (2021) 10. doi: 10.36001/phme.2021.v6i1.3042. [35] S. A. Murad, et al., A review on job scheduling technique in cloud computing and priority rule based intelligent framework. Journal of King Saud University-Computer and Information Sciences 34 6 (2022) 2309-2331. doi: 10.1016/j.jksuci.2022.03.027. [36] S. Khlamov, et al., The astronomical object recognition and its near-zero motion detection in series of images by in situ modeling, in: Proceedings of the 29th IEEE IWSSIP 2022. doi: 10.1109/IWSSIP55020.2022.9854475. [37] R. Gonzalez, and R. Woods, Digital image processing, Fourth edition, NY: Pearson, 2018. [38] V. Vlasenko, et al., Devising a procedure for the brightness alignment of astronomical frames background by a high frequency filtration to improve accuracy of the brightness estimation of objects, Eastern-European Journal of Enterprise Technologies 2 2 128 (2024) 31 38. doi: 10.15587/1729-4061.2024.301327. [39] S. Khlamov, et al., Development of the matched filtration of a blurred digital image using its typical form, Eastern-European Journal of Enterprise Technologies 1 121 (2023) 62-71. doi: 10.15587/1729-4061.2023. 273674. [40] Y. Bodyanskiy, S. Popov, F. Brodetskyi, and O. Chala, Adaptive Least-Squares Support Vector Machine and its Combined Learning-Selflearning in Image Recognition Task, in: International Scientific and Technical Conference on Computer Sciences and Information Technologies, 2022. pp. 48-51. doi: 10.1109/CSIT56902.2022.10000518. [41] O. Suprun, et al., Development of a modified steganographic model of data transmission using
IPv6 protocol, CEUR Workshop Proceedings, vol. 3925, (2024) 35-46. [42] V. Lyashenko, A. T. Abu-Jassar, V. Yevsieiev, and S. Maksymova, Automated Monitoring and Visualization System in Production, Int. Res. J. Multidiscip. Technovation 5 6 (2023) 9-18. doi: 10.54392/irjmt2362. [43] N. Savorona, et al., Development of a decentralized voting system based on blockchain technology, CEUR Workshop Proceedings, vol. 3925, (2024) 239-248.