-MIRF is an open-source library for a convenient creation of applications which process medical data. Architectural style of that library is Pipes and Filters which means that components that process and transform data are connected together in the pipeline. In this paper we describe a new version of the library based on microservice architecture where services are deployed and maintained independently. This could solve a problem with a lack of computational resources needed for image processing. Some new applications of MIRF library are also presented, namely ECG arrhythmias diagnostics and intracranial hemorrhage detection. Implementation of the ECG processing in MIRF was especially interesting due to the fact that ECG is a signal, not an image. Detailed description of ECG processing tool and results of our experiments are also presented. Index Terms-medical images, microservices, electrocardiogram, convolutional neural network 4CVPR 16.02.2020) 5ICLR home page, URL: https://iclr.cc/ (accessed: 16.02.2020)
Medical Images Research Framework (MIRF, [
MIRF project was started in 2018 as an informal
collaboration between students of St. Petersburg State University
and radiologists of one of the private clinics in St. Petersburg.
It was designed as a framework written in Kotlin for a
development of desktop and mobile applications for processing
DICOM and NifTI scans. It is integrated with TensorFlow,
thus allowing to use neural networks, has its own utilities for
DICOM and NifTI processing, can generate PDF reports [
This article presents our experience in creating a new distributed architecture of Medical Images Research Framework home page, URL: http://cvpr2020.thecvf.com/ (accessed: and a several new applications including automated diagnostic of cardiovascular diseases by electrocardiogram and intracranial hemorrhage detection. Electrocardiogram processing is especially interesting because electrocardiogram is not actually an image, so a successful electrocardiogram diagnostic application shows that MIRF is able to work effectively with more general medical data (despite “Images” in its name). As most of the medical tasks, it is an extremely challenging problem, solving which c help save many lives.
MIRF is divided into 2 global packages — core and features.
• core contains the basic types of libraries: abstract classes and interfaces representing blocks, filters and the base classes for medical data — image series, ECG series. • features contains the different MIRF functionality — API to access the repositories, reporting tools, various data formats parsers, image series segmentation and visualization tools
Data in MIRF is represented as a child classes of the abstract class Data. The main objective of the class is to provide convenient access to the metadata stored as list of attributes (the AttributeCollection class in MIRF), as well as to ensure pipeline integrity — only Data-derived classes are to transfer via pipelines.
Computational elements are presented as the implementations of the Algorithm functional interface. It is expected that Algorithm would be implemented only with the pure functions. Such a reduction of possible implementations along with the fact that the algorithm can be easily created from one method (using the SimpleAlg class) provides opportunities for flexible algorithm and hierarchies creation. A typical example is static classes containing handler methods for data of one types, each of which can be converted into an algorithm instead class institutions for each method.
The main purpose of the MIRF library is to create pipelines — series of data handlers. In the terminology of Pipes & Filters architecture, the filters are instances of Algorithm encapsulated in PipelineBlock — a class designed to connect Algorithms among themselves. Data transfer between the blocks is based on event-oriented model, and PipelineBlocks implement Observer pattern. Some blocks are used only for linking algorithms (such as the AlgorithmHostBlock class), whereas others may also be used for the data aggregation or have a specific purpose. So, for example, the AccumulatorBlock class can subscribe to several blocks and signal the result only if the results of all input blocks are ready, but ConsumerBlock does not allow subscribe to itself and serves as a pipeline terminator.
To improve performance and to ensure the possibility of using our system by an unlimited number of people, as well as to save users from the need to install the library on a local computer, we have developed and implemented a microservice architecture. Each Block of MIRF is implemented as a microservice. This allows to flexibly add, remove blocks, change the network configuration, track performance issues. The whole system consists of a set of Blocks, Repository and
Block is a microservice that supports one or more MIRF blocks (a microservice wrapper for PipelineBlock class). Blocks do not have an internal state and continuously process incoming data using contained Algorithm objects.
Repository is a shared (possibly distributed) database that provides a convenient access to data. This entity is necessary because some blocks require more than one data set (for example, the result of processing images in two projections) and the microservices that support such blocks do not have to store them locally and wait until the data is ready. They just store data in the repository and start processing other data sets if there is at least one such ready-made set, or wait until the complete data set appears in the repository.
Orchestrator is a microservice that distributes the load between other microservices. It receives data and specifications from users and sends them to the repository. Specification is a connected acyclic graph that contains information about data processing. A graph always has one input vertex and one output vertex. Graph nodes are the types of blocks that should process data at the current stage. As soon as the repository receives data, the Orchestrator asks the blocks for status, looks which block is the least loaded and sends it a message that the data is loaded and ready.
Online storage is used when there is a large amount of data which cannot be stored on the local machine. For such purpose server for storing medical files and supporting Kotlin library was created.
Server: there are several realizations which could be used
as a server. The first one is to create your own server which
stores files. The reason why it is not suitable is because
there are medicine oriented PACS6 which has its own server
realizations. On the other hand, you can use NAS7. It has
possibilities to use compression while transferring files (for
example, JPEG 20008) and more advanced network protocol
therefore, data transfer speed may increase significantly [
6Picture Archiving and Communication System, URL: https://en.wikipedia.org/wiki/Picture archiving and communication system (accessed: 11.02.2020)
7Network Attached Storage, URL: https://en.wikipedia.org/wiki/Networkattached storage (accessed: 11.02.2020)
8JPEG 2000, URL: https://en.wikipedia.org/wiki/JPEG 2000 (accessed: 11.02.2020)
Library: as MIRF is written in the Kotlin programming in a separate MIRF block. Pipes and Filters architecture gives
language9, the goal was to use capabilities of the language us an opportunity to combine these blocks, which serve for
and the libraries written in the language to achieve maximum different purposes, in a one ECG-processing pipeline.
usability. For network communication, choice of libraries was
between those written in Java (Kotlin is free to use with such A. Database
libraries) or in Kotlin. The second option was picked, since Two biggest public datasets with annotated ECG records are
it uses the ”coroutine” paradigm 10, which is not supported in PTB Diagnostic ECG Database [
At the time of creation, MIRF worked only with im- As an electrical signal, ECG is distorted by different kinds
ages. Nevertheless we thought that its functionality could of noises such as the Gaussian white noise and baseline wander
be expanded to solve problems related to electrocardiogram noise.
signal processing. The electrocardiogram itself is a record White noise is a sequence of independent numbers with
of electrical currents generated by heart. These currents are constant spectral density over all real values. ECG signals are
measured with electrodes placed on the surface of the different usually contaminated with Gaussian white noise which has
parts of body. Potential difference between electrodes is called a normal distribution with zero mean. This kind of noise is
a Lead. In a heartbeat of a healthy person QRS-complex caused by muscle contractions.
could be defined as well as many different waves, segments ECG is a non-stationary signal, therefore standard filtering
and intervals. The morphology of ECG signal is illustrated in methods could not be used (e.g., Fourier Transform).
ComFig. 1. monly used methods for removing white noise from ECG are
adaptive filtering, discrete wavelet transform (DWT),
SavitzkyGolay filtering [
Wavelets are functions that are localized in time and frequency domain. Discrete wavelet transform is based on a signal representation as a linear combination of dilated and shifted versions of mother wavelet.
Denoising with the help of DWT is performed in several
steps. First step is decomposition of a signal which gives
us approximate and detail coefficients. Approximation
coefficients represent low frequency parts, i.e., main features of a
given signal. Some of the coefficients are being removed or
Fig. 1. ECG signal morphology [
ECG data preprocessing and classification could be done in Daubechies mother wavelets are usually used because their
several consistent steps. Each of them then would be wrapped waveforms resemble the form of a QRS-complex which means
that ECG-signal could be well represented by them. Symlet
9Kotlin official documentation, URL: https://kotlinlang.org (accessed: was chosen as a mother function considering good results in
11.02.2020) papers [
Another kind of noise is baseline wander. It is caused by
respiration or body movements. The method using wavelets
proposed in [
Many different algorithms have been proposed to classify abnormalities in ECG records. They could be divided in two classes.
Algorithms from the first class are based on extraction
of morphological features such as amplitudes, segment and
interval lengths. These features are then used as an input to
the classification algorithm. As a classification method we can
use k-nearest members method [
Algorithms from the second class work with images
obtained from raw digital data. One of the most commonly used
technique for ECG image classification are convolutional
neural networks (CNN) as they are able to learn spatial hierarchies
of patterns. Researches made in [
MIT-BIH database consists of records with 14 different types of abnormalities. Healthy heartbeats and heartbeats with 7 clinically significant types of arrhythmias were extracted from MIT-BIH database. Each chosen heartbeat belongs to one of 8 classes: normal beat, right bundle branch block beat, left bundle branch block beat, premature ventricular contraction beat, paced beat, atrial premature contraction beat, ventricular flutter wave beat, and ventricular escape beat. In database annotations they are shortened to NOR, RBB, LBB, PVC, PAB, APC, VFW and VEB respectively. We obtained 100852 distinct heartbeat images from MIT-BIH database.
Due to the imbalanced distribution across classes when more than 50% of data belongs to NOR class, images that belong to arrhythmias need to be augmented. Nine cropping methods are applied and all images are resized to 128×128 before feeding them to a model.
The model has 11 Conv2D, MaxPooling and Dense layers.
The model summary could be seen in Table I. To avoid overfitting batch normalization layers and dropout regularization layers are added. Xavier initializer is used to set initial random weights to layers. According to the original publication, ELU activation function showed better results compared to ReLU and LReLU. Cross-entropy function serves as loss function and optimized with Adam.
To evaluate model after training special cases could be defined: true positive (TP)–model correctly detects arrhythmia, false positive (FP)–model detects arrhythmia in a healthy ECG, true negative(TN)–model does not detect arrhythmia in a healthy ECG, false negative(FN)–model incorrectly classifies ECG with arrhythmia.
Certain characteristics based on these cases exist to check if the diagnosis tool produces good results. Such metrics are accuracy, sensitivity, specificity and positive predictive value.
They are defined below:
T P +T N Accuracy = T P +T N+F P +F N ∗ 100%
T N Specificity = F P +T N ∗ 100%
T P Sensitivity = F N+T P ∗ 100%
T P
Positive Predictive Value = T P +F P ∗ 100%
Using these metrics the model could be evaluated and compared to the other classification solutions. The comparison 0 1 2 3 4 5 6 7 8 9 10 11 could be seen in Table II.
The model does not reach the classification performance of other solutions. Nevertheless, the results are considered to be satisfying since the aim of this paper is to show that MIRF is a convenient tool for a quick medical application development.
The model was implemented in Python language using open-source library Keras12. For training the network NVIDIA Tesla P100 GPU with CUDA 10.1 was used.
The pipeline that processes ECG in MIRF is illustrated with Fig. 4. Custom functions are wrapped in AlgorithmHostBlocks which are later connected to each other.
Medical image analysis competitions have gained great popularity among researchers in the field of machine learning. A large number of people take part in competitions and discuss their results. Our team have decided to integrate the best solutions from the competition into MIRF. With this approach, the library would be expanded with new blocks. What is more, 12Keras, URL: https://github.com/fchollet/keras (accessed: 10.11.2019) it could draw attention of other researchers and users interested in this topic.
There is a large number of venues where data analysis contests are held. The most popular of them are: Kaggle, Codalab, CrowdAI. Different venues differ from each other in the way they present data and the prize pool which greatly affects the number of participants and the level of competition.
It was decided to add CNN based solution from RSNA
Intracranial Hemorrhage Detection because this competition
became one of the most popular competitions on Kaggle in
2019. The purpose of this competition is to classify
intracranial hemorrhage into 6 categories: epidural, intraparenchymal,
intraventricular, subarachnoid, subdural and any other
hemorrhage, including its absence. Labeled data for training and
testing was provided by the organization comitee. The success
of the solution was evaluated by the LogLoss metric [
At the moment, our team is developing a block at MIRF which implements the solution from the competition.
There are some other systems that allow to create medical imaging processing applications and build research and clinical software prototypes. The main criteria for the comparisons of these systems is a support for different medical imaging types, tool functionality, languages and type of platform to run in. Also some of the medical imaging processing tools are open source or have free version to be used noncommercially, while others have only commercial versions. For instance, the Medical Imaging Interaction Toolkit (MITK) is a software system for development of an interactive medical image processing software. MITK framework is a software tool that combines the Insight Toolkit (ITK) and the Visualization Toolkit (VTK) 13. One of the main disadvantages of MITK is that tools are built separately for each platform. NiftyRec could also be mentioned. It is one of the projects developed at University College London. NiftyRec is a software for tomographic reconstruction, providing the fastest GPU-accelerated reconstruction tools for emission and transmission computed tomography14. Besides that NiftyRec works with only brain tomography, it has only Matlab and Python interfaces. Another existing tool is Slicer 15 which is a software modular platform for medical image processing, and three-dimensional visualization. Slicer can be expanded for necessary task with specifically written plugins for these platforms, but this approach does not give the developers enough flexibility to create and adjust their own systems and functionality.
Our future research directions will be focused on improvement of existing functionality and adding new features for a quick and efficient development. By means of new blocks creation, we will provide an ability to work with text data and histology images. These additions are related to the clinical practice guidelines that consist of recommendations for optimizing patient care, and according to that doctor should take into account all patient data at diagnosis, including laboratory tests and histological techniques. Likewise, one of the most important goals of our future research is to make machine learning algorithms accessible to the doctors and medical researchers. It would save their time in routine operations such as compiling a report and would give an opportunity to compare doctor’s diagnosis with a machine learning tool prediction. At the beginning of our work with MIRF framework, our main advisors among the medical community were radiologists. A radiologist is a physician who obtains and interprets medical images directly. Thereby, radiology was the first medical specialization to computerize and, historically, radiologists use wide range of software tools. For these reasons, most radiologists differ from the other medical specialists, so far as having technical skills such as programming. Working on the tasks of the ECG analysis, we collaborated with cardiologists and had completely realized that most doctors do not have any previous programming experience and do not have time to get it. The actual challenge for us is to make MIRF framework available for an average doctor who does not have any coding experience. With that in mind, we directed our attention to visual programming technology, that allows non-developers to build applications. One of advantages of visual languages is that skills in writing code are not needed. Syntax errors are most common types of 13The Medical Imaging Interaction Toolkit (MITK), URL: http://mitk.org/wiki/The Medical Imaging Interaction Toolkit (MITK) (accessed: 09.02.2020)
14NiftyRec 2.0, URL: http://niftyrec.scienceontheweb.net/wordpress/ (accessed: 09.02.2020) 153D Slicer, URL: https://www.slicer.org/ (accessed: 09.02.2020) errors made by programmers. These types of errors are avoided with block-based approach. Doctors will be able to create tools based on machine learning algorithms by dragging icons and fitting them together.
The block-based approach of visual programming gained
popularity while introducing programming to kids. But
recently, the largest technology company and research groups
have been releasing graphical and drag and drop development
tools for different tasks, including machine learning (e.g.,
Watson Studio by IBM, Microsoft drag and drop machine
learning tool, MIT project Northstar [
MIRF framework is evolving in the ecosystem of Software
Engineering Department, we are well acquainted with the
REAL.NET research project. REAL.NET framework uses
multilevel metamodeling approach for fast creation of
graphical programming tool which includes new language, editor
and code generator for wide range of tasks and areas [
In this paper we described additions and improvements made to the MIRF library.
Firstly, we moved MIRF to microservices. A system of Blocks, Repository and Orchestrator, where each Block is implemented as a microservice, was developed. Thus, we aimed to improve performance and made it easier to add, remove blocks, change the network configuration and track performance issues. However, performance experiments have not yet been conducted. This is an important area of further work.
Secondly, we examined storage implementation variants and selected the best one — PACS. Basic repository access library functionality was implemented. PACS server gave us opportunity to conveniently process, upload and download medical data to our server. It would also allow us to process files remotely in the future.
We also added new medical functionality to MIRF and showed that this library is capable of processing medical data apart from images. We investigated various techniques which are used for ECG processing and chose the ones that showed the best results. Algorithms for denoising and decoding were implemented and a convolutional neural network for arrhythmia classification was trained.
Finally, our plans for future MIRF development were described such as adding blocks for processing histology images. We also want our tools to be used by doctors in real clinical practice. For that purpose we are planning to develop visual language which will allow to create tools for medical data processing by dragging icons and fitting them together.