EOS is a CERN-developed storage system that serves several hundred petabytes of data to the scienti c community of the Large Hadron Collider (LHC). In particular, it provides services to the four largest LHC particle detectors: LHCb, CMS, ATLAS, and ALICE. Each of these collaborations uses di erent work ows to process and analyse its data. EOS has a monitoring system that collects detailed information on the le accesses and can give important insights about the speci cs of the physics experiments' work ows. In our study, we analyse the monitoring information accumulated over a six months period and amounting to over 1.3 terabytes and have the goal to help the IT department and the experiments' operations teams to better understand the EOS data ows. In this contribution, we describe a pipeline, mainly developed in R, for processing large volumes of access logs and perform a comparative analysis of the storage usage in scienti c work ows. In particular, we calculate aggregated statistics over a six months period and provide a high-level overview of the experiments' data ows. Additionally, we study how the frequency of data accesses changes over time and estimate to what extent di erent experiments may bene t from an additional caching layer.
The Large Hadron Collider (LHC) is the world's largest and most powerful particle accelerator. It is a massive and long-lasting project and therefore requires state-of-the-art approaches to the tasks of data storing and processing. The four largest particle detectors (ALICE, ATLAS, CMS, and LHCb) are located along the LHC ring. Each of them is a large international collaboration that brings together scientists with di erent backgrounds.
Copyright ' 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The European Organization for Nuclear Research (CERN) provides an infrastructure for the high-energy physics scientists working at the LHC and its particle detectors. In particular, EOS is a multi-purpose storage system used for the experiment measurements and user analysis data and has been developed at CERN since 2010. As of today, EOS operates over 320 PB of raw disk space and provides multi-protocol, secure access and multi-user management.
EOS aims to provide a reliable service that satis es the needs of CERN experiments and users. Nevertheless, the scale of the experiments and the diversity of the physics community make it di cult for the operations teams to monitor the system and to adapt it to the ever-increasing user needs. Since the main target area for the EOS service is physics data analysis, it is characterised by many concurrent users, a signi cant fraction of random data accesses and a large le-open rate.
In this work, we perform a study of EOS as a large distributed storage system in order to help the IT development and operation teams to better understand the needs of the LHC physics experiments. We implement and describe a pipeline for processing EOS access log les and perform a comparative analysis of the four EOS instances serving the needs of LHCb, CMS, ATLAS, and ALICE experiments. We explore the di erences between the instances' data work ows. In particular, we give a high-level overview of data life cycles and describe how data popularity changes over time.
The paper is organized as follows. After outlining the background for our research in Section 2, we formalize the problem and describe the analysis pipeline in Section 3. In Section 4, we present the results and interpretations and Section 5 concludes the paper. 2
CERN operates multiple EOS instances, including one for each LHC large particle detector. Each EOS instance consists of metadata servers (also called management (MGM) nodes) and up to several hundred disk servers ( le storage (FST) nodes). The MGM servers store lists of lenames together with metadata such as replica numbers, owner rights, timestamps of the main operations ( le creation, last update, last access, etc.) and so on. Each disk node contains 24-72 disks and keeps the content of the les. In total, today EOS has 4:8 104 disks and stores 6:1 109 les.
In order to reduce the data access latency and to optimize the data placement
and replication policies, the rst version of an XRootD-based proxy cache
system was introduced [
The CERN Data Centre has been monitored, for more than a decade, with in-house central solutions gathering and storing at the CERN storage facilities a large number of metrics and logging information.
As part of the EOS monitoring system [
This work is performed using the analysis utilities of the CERN Data Centre including some large memory machines available for the research needs. As the development tools, we used the R programming language together with the interactive development environment RStudio. 3
The life cycle of les stored in EOS usually consists of three main steps: 1. Creation (a new le coming directly from an experiment, a new le generated by a user as a part of his/her analysis, or a copy of an already existing le generated for the service reliability); 2. Access (the number of accesses can vary from zero to a very large number); 3. Deletion (most of the les coming directly from the physics experiments are not meant to be deleted and are stored in EOS forever).
The process of deletion is not atomic and usually consists of two main steps: { a version of the le is deleted from a local disk (FST deletion). In this case, other replicas of this le can still exist in the system. Regardless of the number of replicas left, this type of deletion does not erase information about the le on the management node; { deletion from the management node (MGM deletion). This happens when the le is deleted from the system altogether. Usually, this deletion is propagated to the FST where the local versions of the les are deleted.
The EOS access log les are formatted as text les and each line represents
a separate log record [
EOS produces three types of access logs [
Another type is deletion records and it has two subtypes that correspond to FST and MGM deletions. These records contain fewer metrics and store the following information: log record, le and lesystem identi ers; le size before the deletion; timestamps of the last change in metadata, the last modi cation in the le content, the last access and the deletion itself.
The MGM deletion records are more representative of the user work ows comparing to the FST ones since the former is only produced when a user explicitly deletes a le. FST deletions, on the other hand, could be triggered by internal system processes that are not user-related, like balancing. 3.1
Analysis Pipeline The EOS access logs analysis starts with a daemon running every night to collect log records from the management nodes, where they are generated, to the machinery for further processing. This data is parsed and saved in a more convenient tabular format. At this stage, the log records are separated based on their type (events or deletions). In this study, we use log records only of the rst type.
The data ltering step is vital since the raw volume of the analysed log les reaches more than 1.3 terabytes. In order to separate user activity from the system events, we exclude log records that have `daemon' username as part of the trace identi er1 and/or have 0, 1 or 2 as the real-user identi cation number. After that, we reduce the number of metrics and leave only the ones of interest for our research: le identi er, trace identi er, le size on opening and closing, amount of bytes read and written, opening and closing timestamps.
Since there can be more than one opening of a le during one session, we merge log records that have the same le identi ers and happen within one session. For that, as a global session identi er, we use a substring of trace identi er (username, local process identi er and origin host parameters). After aggregation, we update the rest of the metrics within one session as follows: { opening le size is the opening le size from the record with the earliest opening time; { closing le size is the closing le size from the record with the latest closing time; { opening and closing timestamps - the earliest and the latest timestamps accordingly; { amount of bytes read and written - the sums of all the read and written bytes accordingly.
The operation type (create, read, update, etc.) is not present in the log records explicitly, but can be derived from the existing metrics. We use the following heuristic based on the four metrics (osize, csize, rb, wb - opening le size, closing le size, read bytes, written bytes) to classify records into the following ve categories: 1. Creation (osize == 0 and csize > 0 and wb > 0 and rb == 0) 2. Read (osize > 0 and csize == osize and wb == 0 and rb > 0) 3. Update (osize > 0 and csize > 0 and wb > 0) 4. Noop (wb == 0 and rb == 0) 1 The structure of the trace identi er eld is
<user name>.<process id>:fd@<origin host>[.<domain>] 5. Abnormal (csize == 0 and (wb > 0 or rb > 0))
According to the speci cs of the typical ows of physics data in EOS, most of the data is coming directly from the experiments, stored in EOS forever and is not updated afterward. To prove this assumption, we check the number of `Update', `Noop' and `Abnormal' operations to verify that they do not have a signi cant in uence on our research. Furthermore, some les have a phenomenon of multiple `Creation' operations. This can happen when a le was `Created' with csize > 0, then the content was removed with an 'Abnormal' operation with csize == 0. The subsequent update operations will be misclassi ed as 'Create'.
In Table 1, we present the total fraction of les with `Update', `Noop', `Abnormal' operations and/or multiple creations. Both the number and the volume of such les constitute less than one percent for every experiment. Therefore, we assume that most of the tra c on EOS instances for the LHC detectors is `Create' and `Read' operations and that the majority of data les are immutable.
Metric LHCb CMS ATLAS Other Operations, % of related les 0.06 0.26 0.61
Other Operations, % of Instance Volume 0.89 0.05 0.14
After processing daily data, we aggregate it over the full six months period that we considered in our study (from 01/01/2019 to 30/06/2019). This allows us to obtain robust statistical results and avoid insigni cant uctuations that happen over shorter periods of time.
In the following section, results are shown for ATLAS, CMS, and LHCb. Since the ALICE experiment generates the highest volume of log les and given the available computational resources, we obtained aggregated statistics for this instance only for a three months period. We do not present the ALICE results here since the purpose of this paper is to present the comparative analysis of the experiments' work ows on the same timescale. 4
Most of the calculated metrics signi cantly vary in absolute numbers from experiment to experiment. Therefore, we decided to additionally compare the obtained metrics with the total instance volume. The result is presented in Table 2. To obtain the total instance volume, we extracted the respective quantities (total space, used space) from the EOS operation web console. This measure is slightly changing over time, but an average approximation over a one year period is precise enough to serve as a reference for our analysis.
Table 2 reveals that LHCb is the smallest experiment and ATLAS is the largest in terms of total operated volume. `Total Accesses' shows the experiments' turnover (the total amount of read and written bytes). For LHCb and ATLAS, this number reaches more than 300% of the total instance volume, which is signi cantly higher compared to CMS, where it is less than 200%.
The row `Writes' shows the total amount of written bytes and the row `Reads' { the amount of bytes read only during reading operations. Over a period of six months, the LHCb experiment produces a data volume larger than the total instance size, which implies that there is a high number of deletions and that the data is constantly updated. In contrast, ATLAS has a relatively low rate of write operations, but the most intense read tra c; CMS has the lowest read rate amongst experiments.
The last row `Repeated Reads' shows the fraction of repeated read tra c. For ATLAS and CMS, these fractions are high with respect to the total read workload and account for approximately 80% of it. For LHCb, only 30% of the read workload is repeated and hence we estimate a reduced potential pro t from caching.
Additionally, the derivation of le-speci c quantities has enabled us to compare the total volume of the created les versus the read les (see Table 3). `Created Volume' is de ned as the total volume of les with `Create' operations and `Read Volume' is the total volume of les with `Read' operations. `Repeated Read Volume' indicates the total volume of les with more than one `Read' operation during the monitored period.
The LHCb experiment has a well-organized work ow. It produces and reads a big amount of data, most of it goes through `Create' ! `Read' ! `Delete' cycles and is not re-read very often. Also, it has a high rate of deletions, which indicates that it has a limited in storage space.
On the other hand, CMS and ATLAS produce fewer data with respect to their instance size. The read rates show that they mostly use only (50-60%) of their space. Nevertheless, the chances of data re-uses are higher than those for LHCb.
Moreover, we discovered that the amount of read bytes is often signi cantly smaller than the total le size. Therefore, we included the statistic that shows which fraction of a le is read on average. For ATLAS and LHCb, this number is approximately 80-90%; for CMS, only 55% of a le is read on average. The lower numbers indicate that the caching systems will work less e ciently if they support only a full- le prefetching mode. For example, an average read fraction of 50% means that potentially only half of the cache space will serve its purpose.
To compare the le sizes for the experiments, we plotted the le count density distribution over the le size (see Figure 1). The density plots are normalized and the total area under the curve equals one.
The peaks on the plot show that experiments have di erent preferences for le sizes. Most of the les at the LHCb instance are of size 1 kB approximately. For CMS, the most popular le size is around 100 MB. The ATLAS plot has two peaks: around 10 MB and 1 GB. The ALICE instance has the biggest relative number of large les; it has a distinct peak around 1 GB.
Furthermore, we compared the usage rates of the newly created les with the ones that were stored in EOS before the monitored period (Table 4). We de ned as `New' the les that were created during the period of consideration, `Old' are the ones that were created before 01/01/2019.
The LHCb experiment reads almost all of the data that it produces ( 95%); for ATLAS and CMS, these numbers are lower. On the opposite, most of the data that was stored since the beginning of the monitored period was never used afterward. Metric LHCb CMS ATLAS Created and Read, PB 21.6 18.2 23.6 Created and Not Read, PB 1.2 13.2 5.7 Created and Read, % of Created Volume 94.4 58.0 80.4 Created and Not Read, % of Created Volume 5.62 42.0 19.6 Old and Read, PB 3.9 6.5 6.3 Old and Not Read, PB 12.9 35.1 45.5 Old and Read, % of Old Volume 23.5 16.0 11.2
Old and Not Read, % of Old Volume 76.6 84.0 88.8
We explored in more detail how the le popularity changes over time (see Figure 2). On this plot, the dots show the dependency between the accessed volume density over the time elapsed since the les' creation. As expected, we observe a decreasing likelihood of data accesses with the increasing time since the data creation.
We tted this historical data to an exponential decay function: y(t) = yf + (y0 yf )e t where y is the accessed volume density, y0 is the total created volume, yf is a constant, t is the time elapsed since the creation, is the decay rate.
The lines on the plot in Figure 2 represent the tted exponential decay functions. The legend shows the decay rate and the residual sum-of-squares (RSS) error estimate. Additionally, we nd a point in time t after which half of the volume will not be accessed again: 1 y(t ) = 2 y0
CMS data stays popular the longest: the rate is 0.02 day 1 and t is almost 60 days. This means that the probability of a le revisit after 60 or more days since its creation is approximately 50%. ATLAS has a higher decay rate: is 0.04 day 1 and t is only 20 days.
When the LHCb data is tted to an exponential decay function, the rate is the biggest, amounting to 0.16 day 1. The accessed volume never goes as low as 21 y0, therefore the t point does not exist. Even though most of the les quickly become unpopular after the creation, there is some fraction of the data that always remains accessed. 5
We have performed a rst-time analysis of the EOS access logs and developed a set of analysis tools to parse, clean and aggregate them. Using the re ned data obtained in this way, we derived aggregated statistics over a six months period. In particular, we compared the total turnover and the read/write rates of the EOS instances dedicated to ATLAS, CMS and LHCb experiments. We also demonstrated how the probability of data re-usage decreases depending on the time elapsed since its creation.
In the future, we plan to expand and improve our analysis by using new tools for data processing and by obtaining updated statistics for the year 2020. Speci cally, we plan to migrate our pipeline from R to Spark. This will remove some of the memory constraints of our R implementation and enable us to extend the monitored period as well as include ALICE into the comparative analysis.
With the expansion of the main LHC experiment and the construction of
the new High-Luminosity LHC (HL-LHC), we expect a signi cant increase in
data to be processed. This could lead to scaling problems in storage systems,
networks, and data distribution. There has been some work towards better data
locality, which seems to be a promising approach to mitigate this problem [