SLURMminer is a tool designed to analyze SLURM systems in High-Performance Computing (HPC) clusters. It utilizes process mining techniques to generate event logs, extract process models, and visualize critical business intelligence metrics. The tool's unique log extraction approach for SLURM clusters allows for a detailed analysis of jobs and workflows. By providing insights into workflow execution, workload patterns, system utilization, workload balancing, and anomaly detection, SLURMminer equips researchers and system administrators with essential data for performance assessment and user behavior analysis. This paper details installation procedures and describes the tool's functionalities, facilitating eficient management and optimization of HPC clusters with SLURMminer.
∗Corresponding author. http://mpegoraro.net/ (M. Pegoraro); https://vdaalst.com/ (W. M.P. v. d. Aalst) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). allocation and workload management. The use of process mining techniques uncovers patterns, bottlenecks, and anomalies that were previously overlooked by conventional monitoring tools, leading to improved system comprehension and resource utilization. On the other side, SLURMminer can demonstrate how its capabilities can optimize processes, increase eficiency, and uncover hidden opportunities for improvement within organizations that rely on SLURM-based HPC clusters for their computational needs.
SLURMminer was developed on the foundations laid by the author’s previous research [
SLURMminer, a Python-based tool built using the PM4Py library, facilitates advanced process mining operations on SLURM event data.
Constructed on the Flask framework1, it allows HTTP/HTTPS deployment and is compatible with enterprise-grade servers. With Flask’s Cross-Origin Resource Sharing (CORS2) support, SLURMminer ofers a smooth user experience by facilitating multi-origin requests.
The user interface of SLURMminer, crafted using the Bootstrap framework 3, ensures responsive design, accelerated development time, and browser compatibility. Bootstrap’s customizability and integration with JavaScript libraries permit dynamic and interactive UI component development.
SLURMminer stands out for its ability to observe live event streams from the SLURM system, a feature enabled by Flask and PM4Py. It processes and visualizes real-time SLURM events, providing users with instantaneous insights into system behavior and performance.
SLURMminer, an evolutionary tool built upon preceding research, enables thorough SLURM system analysis by integrating system logging, process mining, and business intelligence visualization. The following list proposes some use cases of the tool: – System Logging: Users first engage with the system logging component, logging into the HPC system via a dedicated form on the user interface (UI) (Fig.1, right). The mechanism monitors the SLURM system’s ongoing jobs and records them using the “squeue” command. Logging initiation and reset are controlled by “Start Logging” and “Restart Logging” buttons, respectively, with the latter resetting the log located in the “slurm-log” directory (Fig.1, left). – Process mining: The SLURM log generated can be leveraged by process mining technologies. An event log, structured for process mining, is generated by the “Generate
and Download Normal Log” button. This log underpins the creation of process models visualizing SLURM system operations, extending knowledge about job executions and possible ineficiencies (Fig. 2). The normal event log also elucidates the job execution lifecycle, with stages like PENDING, CONFIGURING, RUNNING, and COMPLETING. – Business Intelligence visualization: SLURMminer encapsulates performance analysis and visualization, providing crucial metrics and graphical illustrations. These visuals cover details such as job counts, job distribution, and the status of active or pending jobs (Fig.3, right). An additional feature identifies batch activities, where numerous similar computations are launched simultaneously. The tool flags potential batching instances by monitoring job initiations within a defined time window and applies a threshold to iflter out high-frequency accounts, resulting in a plot representing potential batch activity periods (Fig.3,left).
SLURMminer amalgamates system logging, process mining, and business intelligence visualization, providing a robust analytical tool for efective HPC cluster management.
SLURMminer’s codebase is available on GitHub at https://github.com/zasab/SLURMminer. The README.md file outlines the setup procedure.
Prerequisites for installation include Node.js4 and NPM for frontend, and Python with p i p 5 for backend dependencies.
To start, clone the repository via g i t c l o n e , then install the required Python packages using p i p i n s t a l l . Launch the Flask server by executing the “app.py” script, making it available in your web browser. For frontend setup, navigate to the “DEV-front” directory and run n p m i n s t a l l followed by n p m r u n d e v . Once the Node.js server is up, access the application via the provided “Local” URL (e.g., http://localhost:5173/).
Please check out my demo video of the tool paper at https://youtu.be/y0ujrvHY2kE for a detailed demonstration.
SLURMminer, bridging process mining with SLURM-based HPC cluster management, provides a novel look at job behaviors and anomaly detection, forging a path toward improved performance and refined optimization strategies. Its intuitive interface provides sophisticated analytical abilities to administrators and users, improving resource management and job scheduling in HPC contexts.
Future iterations of SLURMminer envision a more feature-rich platform, with an expanded array of metrics and a broader spectrum of process mining techniques. These enhancements promise to augment SLURMminer’s impact, continuously steering the evolution of SLURMbased HPC cluster management toward greater eficiency and productivity.
The authors gratefully acknowledge the German Federal Ministry of Education and Research (BMBF) and the Ministry of Education and Research of North-Rhine Westphalia for supporting this work/project as part of the NHR funding. Also, we thank Alexander von Humboldt (AvH) Stiftung for supporting our research.