The Compact Muon Solenoid (CMS) is one of the experiments at the CERN Large Hadron Collider (LHC). CMS is a generalpurpose detector able to record particle collisions up to 40 million times each second. 40 MHz rate results in approx 40 Tb/s, however journey of recorded CMS data is long and only small fraction makes to the nal step - physics analysis. Chain of automated and semi-automated processes lter, reconstruct, calibrate and verify data before it can be used for physics analysis. Data certi cation process starts with data aggregation in histograms, plots and various statistical quantities, and nishes with manual data quality assessment and decision. During the last step multiple experts review and evaluate (certify) data as being \good" or \bad". This results in high manpower demand and occasional involuntary human errors. Data certi cation process consists of online (to determine possible problems during the data taking, only the fraction of data is used) and o ine (not constrained in time full data set analysis including physics data reconstruction). Main goal of this research is to investigate the applicability and provide the comparison of various supervised machine learning techniques for o ine data certi cation process automation. Removal or signi cant reduction of manual labor and exclusion of human errors from the CMS data certi cation process are the main motivations of this research.
Data Quality Monitoring (DQM) is one of the central and the critically
important projects of the Compact Muon Solenoid (CMS) detector at the CERN
Large Hadron Collider (LHC). Main goal of the DQM is to provide and support
a single end-to-end process for reliable certi cation of the recorded data. The
certi cation process comprises of multiple parts, starting from lling and
handling multiple histograms and scalar monitor elements and nishing with the
list of \good" list of runs and lumisections. The goal of the DQM is to discover
and pinpoint errors, problems occurring in detector hardware or reconstruction
software, early, with su cient accuracy and clarity to maintain good detector
and operation e ciency [
While online DQM process only the small fraction of sampled data for
immediate response and provides just technical Detector Performance Group (DPG)
histograms, the o ine DQM examines the full dataset and generates both DPG
and physics POG (Physics Objects Group) histograms. Output is being
accumulated into the ROOT [
Since the start of the experiment in 2008 the CMS DQM processes and tools have been worked out and are stable. A decade of data taking have accumulated the su cient amount of labeled DQM data. This allows the scientists to examine the possibility to automate the nal step of the data certi cation process by using the recent advances in Machine Learning algorithms for binary classi cation. In the current system, hundreds of di erent histograms come as an input and the output is one of \good" or \bad" for each lumisection. 2
Currently, there are several initiatives trying di erent approaches for the data
certi cation and anomaly detection. One of the most advanced research is done
by Yandex School of Data Analysis [
Data used in this research was collected by CERN CMS [
Data preprocessing consists of two steps: merging dataset with GoldenJSON and feature normalization. We used standard score normalization to center and scale data so that distribution of each feature has a mean value 0 and standard deviation of 1. 3.1
Dataset contains very small amount (1.8%) of so-called \bad" lumisections, hence class distribution ratio is 49:1. Therefore, multiple tactics are used to deal class imbalance: class weights penalty, strati ed fold cross validation and rank based performance metrics (see Section 3.3).
First of all, we introduced class weights during training of a model. Additional penalty for classi cation mistakes can a ect model to pay more attention to minority class.
Secondly, we use strati ed cross validation to preserve percentage of samples
for each class during cross validation. Natural distribution of classes in train and
test splits helps model to ght class imbalance during training. However, classical
strati ed cross validation is limited to a certain number of splits. We decided to
use Strati edShu eSplit [
Five methods were selected for a comparison: Support Vector Machine, Random Forest, Naive Bayes, Gradient Boosted Trees and Arti cial Neural Network. We chose popular methods from these categories: probabilistic, ensemble and hierarchical.
Support Vector Machine (SVM) We use scikit-learn [
Random Forest (RF) We use scikit-learn implementation of Random Forest
classi er. The forest has 64 trees with depth of 7 layers, these are hand-picked
parameters.
Gradient Boosted trees (XGB) We use XGBoost [
Gaussian Naive Bayes (NB) We use scikit-learn implementation of Gaussian Naive Bayes algorithm for classi cation. The method trains very fast, but its predictive power is too poor for this task.
Arti cial Neural Network (ANN) We use Keras [
Threshold based. Accuracy (ACC) is a very simple and an intuitive method as it measures actual predicted values. However, that makes ACC poor metric for imbalanced data.
F1 score is a weighted average of precision and recall. Both false positives and false negatives are taken into account which makes F1 score usually more useful than accuracy.
Rank based. Receiver Operating Characteristic (ROC) with Area Under Curve
(AUC) measures how well positive cases are ordered before negative cases [
Experimental Setup. Software used: Python (v3.6), Keras (v2.1.5), Tensorow (v1.7), XGBoost (v0.71), scikit-learn (v0.19.1). Hardware used: PC with NVIDIA GPU (GeForce GTX 1080 Ti) and virtual machine (8 cores 2.2 GHz, 16 GB RAM) Primary task of this research is to compare supervised machine learning models and nd the most applicable for data certi cation. Each model was crossvalidated for 10 times. Support Vector Classi er was eliminated at early stage due to excessive training time. Naive Bayes classi er was eliminated second, because of poor performance for this task. Other 3 models show good performance with ROC AUC score over 0.95. The best model which works almost out-ofthe-box is Gradient Boosted Trees (XGB) with average ROC AUC score 0.987. Average ROC AUC scores are shown in Fig. 2. Full details about each model and each metric is in Table 1. Best scores are in bold.
1.0 0.8 e taR0.6 e iitsvPo eu0.4 r T 0.2 0.0
Receiver operating characteristic 0.0 0.2
ANN (AUC = 0.954) XGB (AUC = 0.987) Random Forest (AUC = 0.971)
Naive Bayes (AUC = 0.706)
Fa0ls.4e Positive0R.a6te 0.8 1.0 Secondary task follows the hypothetical automatic classi cation process in a way that model is trained on historical data and only then it is used to classify the new incoming data. In order to mimic this use-case we sorted dataset timewise and split into 80:20. All models were trained using same split. Fig. 3 shows ROC curves with AUC value. All models perform quite well with 94+% AUC score. 1.0 0.8 e taR0.6 e v iits o P eu0.4 r T 0.2 0.0 0.0 0.2
ANN (AUC = 0.937) Random Forest (AUC = 0.968) XGB (AUC = 0.983)
Naive Bayes (AUC = 0.942) 0.4 0.6 0.8 1.0
False Positive Rate
However due to lucky train-test set distribution Naive Bayes model performs so well, nearly 20% better than average (see Fig. 2). This behavior once again proves necessity of cross validation for a proper model validation. 5
In this paper we ran experiments with 5 classi cation methods and compared their performance on CERN CMS JetHT2016 dataset. Best performing model is Gradient Boosted Trees which ROC AUC score equals 0.987. It seems to be a su cient score already, but due to imbalanced class distribution (98.2% to 1.8%) it is barely better than "most popular class" classi er. Since only manual search of hyper-parameters was performed, therefore we believe that full potential of deep neural network is not yet discovered. Further grid-search of hyper-parameters (learning rate, dropout rate, batch size, number of neurons, hidden layers, epochs, etc) should be done to improve performance and predictive power.