Advances in deep learning enable the automation of a multitude of image analysis tasks. Yet, many solutions still rely on less automated, less advanced processes. To transition from an existing solution to a deep learning based one, an appropriate dataset needs to be created, preprocessed, as well as a model needs to be developed, and trained on these data. We successfully employ this process for bronchoconstriction analysis in Precision Cut Lung Slices for pre-clinical drug research. Our automated approach uses a variant of U-net for the core task of airway segmentation and reaches (mean) Intersection over Union of 0.9. It performs comparably to the semi-manual previous approach, but is approximately 80 times faster.
Inflammatory and allergic lung diseases, such as lung injury, pneumonia, asthma, chronic obstructive pulmonary disease, or pulmonary hypertension are still dificult to treat. A common symptom of these diseases is the obstruction of the airways or Airway Hyperresponsiveness (AHR). Bronchodilators aim to avoid or reduce these symptoms.
With advances in the field of deep learning, computers
reach and sometimes surpass human level performance in
specicfi tasks like image analysis [
There is a continuous need of innovative or advanced drugs
to treat the symptoms of obstructive lung diseases. PCLS
is one of the developed models to evaluate
bronchodilatory compounds in the non clinical phase of drug
development. Bronchoconstriction analysis with this model is
applied in house since 2012. PCLS allows to target small
airways of the lung in diefrent animals, (mouse, rat, or
non-human primates) or human lung explant tissue. They
can be analyzed to assess potency of bronchodilatory drugs
through concentration-response curves. Mean inhibitory
concentrations (IC50 values) can be compared to reference
drugs within one human donor or animal to compare e-fi
cacy of the drugs [
We use images that were acquired in previous studies
testing eficacy of bronchodilatory drugs. In those
studies, PCLSs were prepared from tumor-free parts of lung
explants or from freshly isolated lungs after necropsy of
animals, as described in [
In our previous studies, we used a semi-manual approach to
analyze the microscopy images. The Image J plugin “PCLS
Area Tool ” was used for the airway segmentation. It is based
on the IsoData approach [
In recent years, NNs have caused big leaps forward in
semantic segmentation tasks [
In the context of lungs, U-net is frequently used. It
can be extended to tfi specific tasks, e. g., through adding
residual and recurrent connectivity [
To automate airway segmentation with NNs, we need labelled data for supervised training. To the best of our knowledge, no public dataset for airway segmentation exists. Thus, we create our own (Section 4.1) and propose a data pipeline for preprocessing (Section 4.2). In Section 4.3, we describe the development of an ML model for semantic segmentation and analyse it in Sections 4.4 and 4.5. 4.1
Because we are working towards the goal of optimal automatic segmentation, the quality of data labels is essential. The upper limit of model performance depends on the label quality. Deviation from the ground truth means a loss in model potential. Because the existing tools yield unsatisfactory labels, the image labelling is performed manually: Each label is created by a trained specialist. It is then reviewed and possibly adjusted by another one. (a) 3D representation. The dark plane represents the camera’s focus plane. (b) 2D representation from above, similar to the camera perspective.
The main target of bronchoconstriction analysis is the relative change of airway volume. Our annotations resemble an approximation of the ground truth: We consider airways simpliefid as prisms (see Figure 2a). The volume of a prism is denfied as = ∗ ℎ, where is the base area and ℎ is the height of the prism. The airway cross-section surface is proportional to the airway volume; thus, we can use it as an indicator for the volume. If an airway lies perfectly orthogonal to the camera, all cross-section surfaces overlap. If not, cross-sections of diefrent depths are shifted on the image plane (see Figure 2b). Some cross-sections might be out of focus and choosing an arbitrary cross-section might lead to a less stable model. To overcome these issues, we label the cross-section in the focus plane, i. e., the plane that lies parallel to the camera and is the sharpest. This includes more detail and allows more unambiguous labels. We create labels as RGB images representing class membership on pixel level, where the target class is one of background, border, or airway. Each target class is assigned a unique RGB color with maximal distance to other classes.
The labelled dataset should resemble real life data. In order to select a diverse dataset from as few images as possible, we choose one image per experiment setup as described in Section 2.1, yielding 420 samples. 4.2
The unprocessed sample data is preprocessed as depicted in the pipeline in Figure 3 before being fed to a model. First, one sample is denfied as a raw image, its corresponding label created as described in Section 4.1, as well as its meta data (e. g., quality, see Section 5). The samples are collected in what we call the original dataset. From this original set, we can create a subset (e. g., by quality). This step is optional and depends on the goals of the test run.
Since working with image data is resource intensive, we
aim to optimize computation time and cost. To that end,
we store the dataset variants in the tfrecords format2. We
read those lfies into tensorflow.data.Dataset 3 data
structures for further processing. Both are part of the tensorflow
framework [
We want to use class based metrics (see Section 4.4) and hence one hot encode the label images. To that end, we transform them from RGB space to HSV space. We apply the same transformation to the target class representations described in Section 4.1. We empirically chose thresholds for hue (10), saturation (100) and value (100) to determine whether or not a pixel belongs to a target class. Each pixel is assigned at most to one target class that way. For each label, we use this to create binary masks for the classes. Finally, we concatenate the binary masks as channels, forming the one hot encoded labels (see last step in Figure 3). 4.3
Within the scope of the model development process, we
work towards fast exploration and experimentation.
Maintaining high code quality and avoiding bugs are further
meta goals. Hence, we employ the following mechanisms.
Our model development process can be described as two
intertwined cycles of research on one hand and development
2For details on tfrecords, see https://www.tensoroflw.org/tutorials/
load data/tfrecord#setup (last accessed on 2021–03–18).
3For details on tf.data.Dataset, see https://www.tensoroflw.org/api
docs/python/tf/data/Dataset (last accessed on 2021–03–18).
on the other (see Figure 4). We start by phrasing a question
like “How does image size influence model performance? ”.
If that question can be answered with the current code
basis, we continue with the research cycle. We denfie the
necessary test run parameters and congfiuration, before
executing it in our research environment. This also serves
the purpose of documentation and increases reproducibility
as the same congfiuration can be run again. We use jupyter
notebooks [
If a question cannot be answered with the current code
basis, we continue with the software development cycle.
Driven by reusability and modularization, we formulate
generalized requirements from the question (e. g., resize
images). We then implement and test the missing features
using test driven development [
In order to compare performance of diferent models or
settings, we use numeric metrics. In our semantic segmentation
4For details on pytest, see https://docs.pytest.org/en/stable/ (last
accessed on 2021–03–18).
task, we use per-pixel labelling. Due to strong class
imbalance (usually there is more background than foreground),
simply using accuracy does not sufice. By choosing metrics
that consider both, the number of correct predictions, as
well as the ratio of classes, they better represent the actual
prediction quality. Due to its wide spread use [
+ 1 =0 + + where is the number of target classes, is the number of true positive, the number of false positive, and the number of false negative pixels for the target class .
We also use (mean) Dice Similarity Coeficient (mDSC),
also called quotient of similarity, F1-score, or harmonic
average between precision and recall (cf. [
2 + 1 =0 2 + +
We use Categorical Cross Entropy (CCE) as both, loss and metric (lower is better), denfied as:
= − Õ ∗ log =0 (1) (2) (3) where is the total number of values (pixels through all channels), is the model prediction for index , and is the corresponding target value.
In the upcoming experiments, we consider diefrent variables, like resolution, and their efect on prediction performance. We use the following default setting, unless speciefid otherwise: (1) U-net with two encoder and decoder blocks (2) image dimensions of 128 by 128 (3) 64 filters in rfist convolutional block (4) spatial dropout of 0.5 in encoder (5) ReLU as activation function in inner layers (6) SoftMax as activation function in output layer (7) CCE as loss function (8) batch normalization between convolutional layers (9) training for 50 epochs with a batch size of 8. Using batch normalization and dropout reduces overtfiting.
Further, we use a variant of ten-fold cross validation: First, we split the data into train, test, and validation sets. For training, we then further split the train set into ten separate folds, each missing a diferent tenth of the original set. We use the validation set to observe training performance. After training, we use the test set to obtain results (i. e., the ten diefrent folds are evaluated on the same test data). We choose this variant to be able to better compare evaluation of diferent subsets (see Section 5). The reported metrics refer to either the average (CCE, mDSC, mIoU) over all folds, or sum (duration). All test runs are performed on an Intel(R) Xeon(R) Gold 6252 CPU and two NVIDIA Tesla T4 GPUs. 4.5
The approach described in Sections 4.2 and 4.3 enables us to eficiently perform experiments. We can evaluate hypotheses in fast cycles and robustly implement missing functionality. We start with small images. Then, we identify and address issues that arise as we scale up. In a rfist experiment, the input images and labels are bilinearly resized to diefrently sized squares (for images of size 1024, batch size is reduced to 4). The results are depicted in Table 1. The best results are achieved when using the smallest images, increasing image size leading to a decreasing performance.
We conclude that the task of airway segmentation is easier, if the image resolution is smaller, i. e., there are fewer details available. Also, models with smaller image sizes converge faster, which we show in another experiment: We increase the maximum number of epochs, but stop training early, if the model converges. To evaluate convergence, we consider validation loss and stop learning when there is no improvement for twelve epochs. This further reduces overtfiting. The results are depicted in Table 2. When comparing the average duration in both experiments, we see: While models trained on images smaller than 128 pixels stop training earlier than 50 epochs, those trained on bigger images need more epochs before stopping. This also improves the results for bigger image size.
Apart from the model, we also reconsider our dataset in order to improve results. Our dataset is heterogeneous. It contains images from various specimens and was collected by vfie researchers. Due to diferent experiment setups, they also vary in visual properties like brightness (standard deviation of per image average normalized brightness of 0.11), hue (0.22), saturation (0.12), and value (0.11) in HSV space. This afects the overall quality of each image. We empirically identify the following crucial factors: (1) contrast 68 = 3 68 = 3 140 = 4 68 = 3 140 = 4 284 = 5 mDSC mIoU between the airway and the surrounding tissue (2) airway alignment in relation to the camera (3) sharpness of the airway edge (4) amount of airway occlusion (by tissue, particles or image distortions like aflres). Because we found our perception inuflenced by these factors, we hypothesize this might be the case for our model. Based on these criteria, we manually categorize the samples into 22 “bad ”, 345 “good ”, and 53 “great ” ones and perform further experiments.
First, we inspect data subsets by quality. A test set of 30 samples is randomly selected from the original dataset. It contains two “bad ”, 24 “good ”, and four “great ” samples and exhibits similar visual properties: it difers only slightly in average brightness (−0.01), contrast (+0.001), hue (+0.03), saturation (+0.02), and value (−0.01). The remaining data is split by quality label. Each subset is used as training set and evaluated on the test set. The results are depicted in Table 4. We can explain the main diference in performance by diferent subset sizes: The best outcomes result from training on 321 “good ” samples, while using 49 “great ” samples yielded inferior results, followed by 20 “bad ” samples.
Next, we use random under sampling to balance the three subsets. When we repeat the experiment with only 20 samples per subset, the performance diference decreases (see us in Table 4). The prediction performance strongly correlates with our assigned quality label. We hypothesize that the order in which the samples are presented to the model might inuflence its performance. We test this hypothesis in an experiment, in which we use up to three distinct subsets to iteratively train a model. We stop training early and maintain the trained state of the model between sets. Because the order matters, we average the results of ten independent iterations instead of using cross validation as before. The results are depicted in Table 4. Overall, diferences between settings are small, but there seems to be a slight advantage in starting with the worst and nfiishing with the best samples (bad→good→great). 6
When we consider the lessons learned in Sections 4 and 5, we create a model that difers from our defaults in the following way to achieve the best trade-of between image resolution and prediction quality. (1) U-net with four encoder and decoder blocks (2) image dimensions of 512 by 512 pixels (3) training for 500 epochs with early stopping and a batch size of 8 (4) training by quality meta label (bad→good→great).
To compare our new approach to the semi-manual one, we use the test set denfied in Section 5. The above U-net
In this work, we demonstrated how we transition from a semi-manual analysis of bronchodilators to an automated workoflw using deep learning. To that end, we created a heterogeneous dataset. We presented how to use Tensorlfow
to create an optimized data pipeline; this pipeline can be used in other projects using Tensorflow with minor adjustments. Further, we propose an iterative model development process applicable to any data science project that requires custom code. We showed the capabilities of our approach in a set of experiments for the semantic segmentation of airways. In these experiments, we trained a variant of U-net to segment airways in PCLS microscopy images with mIoU of 0.881 and mDSC of 0.935. We also demonstrated that image quality and training order can improve predictions. In comparison to our previous semimanual approach, the proposed automated method yields comparable results, but does so in a signicfiantly faster way. Thus, our automated approach constitutes a valid alternative. These segmentation results assign individual pixels to the airway target class. We can count those pixels to derive the airway area. From this, we can determine a relative change of the area (and hence the volume as described in Section 4.1) in a sequence of images and thus determine the bronchoconstriction.
Parts of our implementation are afected by randomness.
As such, small diefrences might be due to chance. We
expect additional improvements in semantic segmentation
performance, once we add traditional or NN based data
augmentation as in [
This work was supported by Fraunhofer Internal Programs under Grant No. Attract 042-601000 and Central Innovation Programme for small and medium-sized enterprises (ZIM) on behalf of Federal Ministry for Economic Aafirs and Energy (BMWi) under Grant No. ZF4196502 CS7.