The laser-based synchronization system of the European X-Ray Free-Electron Laser is responsible for precisely synchronizing various components within the large scale facility. It comprises several embedded components that are directly connected to the accelerator control system. In this study, we introduce a data acquisition system, which is integrated into the control system and builds the base for data-driven root cause analysis and predictive maintenance. To optimize the data retrieval process, we extended the existing data acquisition system by a lightweight database system based on Apache Parquet. This extension significantly enhances the data readout speed by a factor of 2000 , enabling eficient processing of operation-critical data. Additionally, we present a user-friendly dashboard that visualizes the operation-critical data, allowing for intuitive monitoring and analysis. Moreover, an unsupervised fault detection pipeline is created, capable of identifying faults retrospectively. Further validation through extensive real-world testing and deployment in daily operations is essential to ascertain the reliability and efectiveness of the integrated solution. This study serves as a foundation for future research and development eforts in optimizing and automating data acquisition and fault diagnosis methodologies for large-scale complex facilities, enhancing their robust performance and ensuring operational reliability.
CEUR ceur-ws.org A
The 3.4 km long linear accelerator-driven European X-ray Free-Electron Laser (EuXFEL) [
CEUR Workshop Proceedings (CEUR-WS.org)
system [
In Section 2, we present relevant literature concerning database systems for similar largescale facilities. Section 3 gives an overview of the architecture of the LbSync system and its DAQ system, including the challenges faced. In Section 4, we discuss the main contribution of this work which is the extension to the existing DAQ system that efectively addresses these challenges. Finally, Section 5 concludes this work, summarizing the key ideas and underlining the importance of the proposed solutions to increase the LbSync system availability.
Several researchers have made contributions to the field of predictive maintenance. The authors
of [
The authors of [
The research of [
Based on the review of related work, we present the experience, challenges, and the current state of the LbSync DAQ system. Utilizing the findings of the existing work, we developed an extension to the existing LbSync DAQ system, improving the data retrieval process.
The following three research papers present dashboards to depict data from big data systems.
The work of [
A schematic overview of the EuXFEL and the LbSync system is given in Figure 1. The main
oscillator (MO) [
The pulse train from the MLO is split and transmitted to various fiber link stabilizing units (LSU) which actively stabilize their length. These fibers are employed to establish connections between the LSUs and the respective end-stations in the accelerator, such as laser synchronization setups, the RF re-synchronization units, and Bunch Arrival Time Monitors (BAM). Furthermore, a sub-distribution system is set up in the experimental hall 3.4 km away, in which another two laser oscillators are phase-locked to the LbSync system. This arrangement allows the sub-distribution laser oscillators (SLO) to replicate the MLO signal and to distribute the optical synchronization signals to additional end stations. This large setup ensures synchronization on a femtosecond level between the accelerator components and the various experimental lasers, facilitating advanced research at the EuXFEL.
All LbSync devices are driven by interconnected hardware running control algorithms. The devices generate data through diferent means: sensors attached to the devices, diagnostic data derived as a result of the control algorithms, and configuration parameters. The LbSync system is a dynamic system subject to ongoing maintenance, upgrades, and configuration changes. To obtain a comprehensive overview of the entire system, it is essential to capture monitoring, diagnostic, and configuration data from all LbSync components.
In total, we collect data from the following components as depicted in Table 1.
A pulse-synchronized data acquisition system has been seamlessly integrated into the accelerator
Distributed Object-Oriented Control System (DOOCS) [
The LbSync DAQ system aims to capture all available monitoring, diagnostics, and
configuration data from the LbSync system over extended periods, distinguishing itself from existing
DAQ systems that only store selected data channels for a limited duration. Table 1 gives an
overview of how many data channels are planned for long-term storage and their respective
data rate.
3.2.1. Data Flow
Figure 2 illustrates the data flow from the the physical systems to the final long term storage. In
the LbSync DAQ system, multiple physical devices such as laser oscillators and link stabilizing
units are connected to respective device hosts, which enables the exchange of monitoring,
configuration, and diagnostics data specific to each device. Each device host executes at least
one DAQ sender instance, responsible for transmitting its data to a centralized DAQ server
through middle-layer services. The DAQ server collects data packages from all the DAQ senders,
where each package comprises approximately 60 s of data. These packages are merged and stored
in a single file using a proprietary raw file format, which is optimized for synchronized and
compressed storage of data. This means that all data sharing the same bunch ID or timestamp
respectively is grouped together in the storage, making it easier and more eficient to access
and retrieve data for specific bunch IDs or timestamps. To ensure data integrity, an automated
process transfers the raw files to a long-term storage system utilizing dCache [
semi automatic updates
server configurations device host (MLO1) snapshot collector
DAQ sender
MLO 1 device host (LSU27)
DAQ sender snapshot collector
LSU 27
DAQ Server raw files dCache The DAQ is designed to accommodate varying update rates for diferent data channels. These update rates range from infrequent updates up to 300 MHz. Slow channels primarily involve configuration properties, such as hostname or software versions, while fast channels encompass controller I/O or ADC/DAC data. Most of the DAQ channels have an update rate of 10 Hz, aligning with the accelerator’s electron bunch rate. However, due to network limitations, the current infrastructure cannot handle the data throughput of the fast DAQ channels. As a result, the system currently skips most of the fast channels.
Fast data channels however can provide valuable insights into the overall performance of a system. To capture the most critical fast data channels, we utilize snapshot collectors running on the respective device hosts. Due to limited storage space on the device hosts, it is not feasible to record the fast channels continuously. In order to process all of the fast data channels, we implemented a strategy to collect operation-critical data for a fixed duration of 10 s every day at 5 am. These periodic snapshots complement the DAQ system by allowing us to obtain a daily overview of the system’s health under the same conditions.
The configuration of DAQ senders on the device hosts plays a crucial role in determining which
data channels are sent to the central DAQ server. As the device server’s features continuously
evolve, the available data channels undergo regular modifications, expansions, or removals.
Consequently, the configuration of the DAQ senders must be consistently updated to reflect
these changes. To ensure eficiency, the process of updating DAQ sender and DAQ server
configurations is performed semi-automatically using a version-controlled central
configuration database, enabling comprehensive tracking of all changes. A detailed description of the
configuration update process is given in [
Figure 3 provides a detailed representation of the specific time periods during which the DAQ system was actively recording data. It is important to note that the DAQ is intentionally not operational during the EuXFEL shutdown maintenance phase, which is dedicated to rolling out major changes and updates to the device servers. Following the completion of device server updates, the DAQ undergoes its own maintenance phase. Due to the timing of server rollouts at the end of the shutdown maintenance phase, the DAQ maintenance naturally coincides with the initial period of EuXFEL operation. Overall, the DAQ system operated for approximately 54.70 % of the total EuXFEL operation time, resulting in the collection and storage of approximately 160 TB of data. The downtimes of the DAQ can be explained by the fact that the DAQ system itself was being commissioned.
In contrast, Figure 4 provides an overview of the time periods during which the snapshot servers were operational. The snapshot servers were collecting data at 97.29 % of the EuXFEL operation days, which is significantly higher compared to the DAQ servers. This is primarily due to the minimal maintenance requirements of the snapshot servers, as the underlying DOOCS properties typically remain unchanged for extended periods. As a result, the snapshot servers reliably and consistently capture snapshots throughout the EuXFEL operation, resulting in an accumulation of 1.3 TB of data.
The current data acquisition (DAQ) system stores data as raw files, which poses limitations for eficient data retrieval. The primary objective of the LbSync DAQ is to quickly display and analyze operation critical data within specific time ranges and to support fault diagnosis and predictive maintenance applications. Both applications require a fast and reliable data readout process. The existing DAQ solution is not optimized for use cases where raw files have a large number of data channels. It has been primarily optimized for feedback applications and direct communication with device servers.
Visualizing a week’s worth of operation critical data (90 diferent data channels) amounts to approximately 2.5 GB, taking more than 5 days to retrieve. Typically, the data readout speed is around 5 kB/s. This extended retrieval time is not acceptable for daily operations. We have extended the existing architecture (depicted in Figure 5) to overcome these challenges and incorporate improved data retrieval capabilities.
Figure 5 illustrates the data flow of the extension to the existing DAQ system. The DAQ system utilizes raw files which contain data from all data channels. To eficiently extract operation critical data, an automatic transformation script is executed as soon as a new raw file is created by the DAQ Server. This script extracts the operation relevant data, including the corresponding timestamps and bunch IDs. The step of extracting data from the raw files is the bottleneck to make the data accessible. We store this extracted data in Apache Parquet files, creating a separate Parquet file for each data channel and raw file. The organization of Parquet files is based on data channel names, resulting in a well-structured database. This solution satisfies the requirements of the dCache technology, which emphasizes write-once read many (WORM) ifles. The raw files are retained on the dCache system to preserve data from all channels for root cause analysis. Consequently, the extracted data is duplicated on the dCache. Additionally, the raw files are retained to allow potential future enhancements or modifications aimed at improving raw file management and handling.
Figure 6 illustrates the superior readout speeds achieved by this extension compared to the raw file solution. This results in a readout duration of 9.85 ⋅ 10−3 ⋅ ⋅ sconds for the raw file solution and 5.448 ⋅ 10−6 ⋅ seconds for the Apache Parquet extension, where corresponds to the number of extracted channels and corresponds to the window length of requested data in seconds. Furthermore, the Apache Parquet extension not only enhances data readout speed but also supports access from multiple programming languages. Subsequently, we present two applications built on the Apache Parquet database: a dashboard and an unsupervised anomaly detection pipeline.
This subsection presents the eforts in developing an automatic fault detection system [ 25] that examines operation critical data obtained from the DAQ system and its subsequent integration into the daily operation of the LbSync system. To identify faulty or unusual system behavior, an unsupervised detection pipeline was designed for analyzing the time series data, as illustrated in Figure 7. The first step involves dividing the signals into 30 s segments. Subsequently, a compact set of features, similar to the proposed EficientFCParameters features by tsfresh [26], is calculated for each signal. The features are then transformed using z-normalization. Next, an IsolationForest is fitted to the normalized features to generate an anomaly score for each 30 s window. If the anomaly score surpasses a pre-defined threshold, an automatic email will be sent to the system operators.
30 s windows anomaly detection feature extraction
The visualization of data is facilitated through a Plotly [27] dashboard that is connected to the dCache long-term storage. The dashboard serves as a user interface (see Figure 8), allowing the selection of a specific LbSync device and day for analysis. It is designed to visualize both snapshot data and DAQ data in a uniquely combined view.
The snapshot data is presented in both the time and frequency domains, as both domains provide distinct information crucial for analyzing the health status of the systems. The dashboard application ofers configurations for the frequency domain calculations such as window functions and sizes. Furthermore, operation critical LbSync data channels of the selected device are plotted for the selected day.
The operation critical signals are segmented into windows, and for each window, an anomaly score was calculated. These anomaly scores are further visualized on the dashboard, allowing for an intuitive representation of the detected anomalies in the time series data.
In our study, we presented the successful development of a comprehensive database system which collects signals from multiple sending servers running on device hosts interconnected with various physical devices. The semi-automatic configuration of these sending servers enabled the accumulation of 40 221 distinct data channels with diferent update rates. By implementing an extension based on Apache Parquet to the existing DAQ system, we significantly improved data retrieval speed for applications like history visualizations and fault analysis. Moving forward, several improvements and next steps are identified. Firstly, incorporating root cause analysis and fault localization into the anomaly detection process would enhance efectiveness of the fault diagnosis. Functionalities should be added to the dashboard, allowing users to select any data channels and not only the predefined set of operation critical data. Furthermore, it should be noted that the dashboard and fault detection pipeline are not yet in daily use, so the design and layout of the dashboard components is not yet finalized. The reliability of the fault detection pipeline still requires validation through daily operation, ensuring its efectiveness and robustness.
We acknowledge the support by DASHH (Data Science in Hamburg - HELMHOLTZ Graduate School for the Structure of Matter) with the Grant-No. HIDSS-0002.
We acknowledge the support from DESY, a member of the Helmholtz Association HGF. dcache-whitepaper-light.pdf. [25] A. Grünhagen, J. Branlard, A. Eichler, G. Martino, G. Fey, M. Tropmann-Frick, Fault Analysis of the Beam Acceleration Control System at the European XFEL using Data Mining, in: 2021 IEEE 30th Asian Test Symposium (ATS), 2021, pp. 61–66. doi:10.1109/ ATS52891.2021.00023. [26] M. Christ, N. Braun, J. Neufer, A. W. Kempa-Liehr, Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh – A Python package), Neurocomputing 307 (2018) 72–77. URL: https://www.sciencedirect.com/science/article/pii/S0925231218304843. doi:https://doi.org/10.1016/j.neucom.2018.03.067. [27] P. T. Inc., Collaborative Data Science, 2015. URL: https://plot.ly.