Many scientific domains gather suficient labels to train machine algorithms through human-in-the-loop techniques provided by the Zooniverse.org citizen science platform. As the range of projects, task types and data rates increase, acceleration of model training is of paramount concern to focus volunteer efort where most needed. The application of Transfer Learning (TL) between Zooniverse projects holds promise as a solution. However, understanding the efectiveness of TL approaches that pretrain on large-scale generic image sets vs. images with similar characteristics possibly from similar tasks is an open challenge. We apply a generative segmentation model on two Zooniverse project-based data sets: (1) to identify fat droplets in liver cells (FatChecker; FC) and (2) the identification of kelp beds in satellite images (Floating Forests; FF) through transfer learning from the first project. We compare and contrast its performance with a TL model based on the COCO image set, and subsequently with baseline counterparts. We find that both the FC and COCO TL models perform better than the baseline cases when using > 75% of the original training sample size. The COCO-based TL model generally performs better than the FC-based one, likely due to its generalized features. Our investigations provide important insights into usage of TL approaches on multi-domain data hosted across diferent Zooniverse projects, enabling future projects to accelerate task completion.
ing labels to train machine learning algorithms typically
training models from scratch e.g., [
Transfer learning (TL) is an established approach, where the feature space from a pretrained model can be transferred to another framework and fine tuned to perform analogous or diferent tasks. Feature extraction is typically performed using Deep Convolutional Neural Networks (CNNs) such as [13, 14]. Transfer learning generally uses models trained on data that is either “out
Citizen Science has established itself as a valuable method
for distributed data analysis enabling research teams from
diverse domains to solve problems involving large
quantities of data with complexity levels requiring human
pattern recognition capabilities [
In order to leverage the Zooniverse’s large library of image-label pairs across multiple domains, there is thus a clear need to better understand the efectiveness of cross-domain transfer learning. In particular, we are interested in the application of transfer learning specifically to projects that share task similarity across a wide range of domains. For example, image segmentation tasks vary across vastly diferent disciplines, from cell biology to satellite imagery. Frameworks such as the U-Net [20], Recurrent Convolutional Networks such as Mask-RCNNs [21], and Generative Adversarial Networks (GANs; e.g., [22, 23]) have been used to perform such object segmentation across multiple domains and data sets. However, robust learning of such segmentation models from scratch often requires large annotated training samples that may not be available (e.g., medical imaging), which can lead to poor generalizability of the learnt features to newer data, even in related domains. While Zooniverse can provide these large annotation sets per project, this comes at the cost of volunteer efort which we seek to optimize.
In an efort to increase project completion rates, this study investigates potential machine performance gains through transfer learning across domains by leveraging the shared task similarity between Zooniverse projects. We use a PatchGAN-based [23] segmentation model1 to investigate the efectiveness of segmenting kelp beds from satellite images. Particularly, we test transfer learning from the COCO dataset (i.e., out-of-domain) and microscopy imaging of lipid droplets in liver cells (pseudoin-domain) and compare them to their corresponding “trained from scratch” counterparts.
the Pix2Pix GAN architecture in [23], which is a conditional GAN for realizing paired image-to-image translation. The PatchGAN architecture consists of a Generator () and Discriminator ():
The generator is composed of a U-Net [20], a U-shaped encoder-decoder neural network, with skip connections across the bottleneck layer (Figure 1). The encoder (decoder) comprises of 6 downsampling (upsampling) blocks, each consisting of 4 × 4 convolution (transposed convolution), Leaky ReLU activation, and a batch normalization layer. All the blocks in the inner layers of the network also include a dropout layer which omits 50% of the extracted features during training. The outputs of the transposed convolutions are also concatenated with the corresponding skip connection feature map from the encoder block.
The discriminator is a patch-wise binary classifier that takes a concatenation of the input image and its corresponding ground truth or generated mask and outputs a 30 × 30 probability matrix. Each unit of this matrix represents a 70 × 70 patch of the input image, and provides the probability that the patch is real. 2.2. Data
In this section, we detail our PatchGAN architecture [23], pairs: the Floating Forests dataset, Etch-a-Cell dataset the training and testing data and its preparation, and the and the COCO-stuf. The former two are Zooniverse description of the five models analyzed in our work. projects focusing on image segmentation, while the latter represents a generic image dataset that is used in computer vision, representing an out-of-domain dataset compared to the former two. These three data sources 1https://github.com/ramanakumars/patchGAN/ represent a diverse feature set on which to perform our transfer learning experiment. Figure 2 shows an example of an image-mask pair from each dataset. 2.2.1. Floating Forests ( )
project hosted on Zooniverse.org2 to identify kelp beds in Landsat imagery. The project presents segments of Landsat data to Zooniverse volunteers, who draw outlines around the kelp beds. These annotations are aggregated using a pixel-by-pixel consensus to create masks of the kelp beds in the corresponding Landsat segments. We use 4 channels from the Landsat data (Blue, Green, Red and near Infrared) to train the patchGAN on the imagemask pairs. This FF data comprises 6,967 (350 × 350 pix) image-mask pairs. We pre-process these data such that each pair is cropped into four 256 × 256 overlapping cutouts, and augment each crop 5 times (rotation and lfipping). This resulted in 118, 440 training and 4180 testing images. 2.2.2. Etch-a-Cell: Fat Checker ( )
Etch-a-Cell: Fat Checker is a cell biology project hosted on Zooniverse.org3 to identify lipid droplets in electron microscopy data. The Zooniverse project presents 2D slices of the data to volunteers who annotate the outline 2.3. Experimental Design of the lipid droplet. The lipid mask is generated by ag- In this work, we investigate the potential of cross-domain gregating the annotations by multiple volunteers based transfer learning by training 5 models. The first 3 on consensus. The data set consists of 2341 image-mask models are trained from scratch – Λ , Λ , and pairs and each image is 1200 × 1200 pix in shape, with Λ – using 100% of their corresponding data sets 3 channels. We split the sample into 2, 106 training and , , and , respectively. Next, we train 235 testing sets. We transform these images and masks the Λ → and Λ → by transferring the to work with our PatchGAN framework by resizing them weights from the trained Λ and Λ models to to 512 × 512 pix and generating five crops (four corners the Λ . By comparing between the baseline Λ to and one center crop). We further augment them by apply- the transfer learnt models Λ → and Λ → , ing three rotations (90, 180, 270 deg) per image, yielding we quantify the impact of performing transfer learning augmented training and testing samples of 42120 and on the accelerated learning of the Λ model from 4700 images, respectively. two distinct feature initializations. During this transfer learning exercise, we also vary the amount of training 2.2.3. COCO-Stuf data used from 10%-100%. the ‘person’ class. This amounts to 63785 training and 2673 testing image-mask pairs.
collection of several real-world images with objects set in various simple to complex scenes, which are annotated by outlines4. [25] further processed the COCO data set to produce dense pixel-wise annotations for them (the COCO-Stuf data set; hereafter COCO). These images and annotated masks vary widely in their shapes, and therefore, we standardize these images by resizing them to a 256 × 256 pix shape. For our PatchGAN training, we limit the training and testing data to those that host
3https://www.zooniverse.org/projects/dwright04/ etch-a-cell-fat-checker 4https://github.com/nightrome/cocostuf
3.1. Training Strategy Our Λ , Λ , and Λ models have been trained for 50 epochs. For the generator, we use the Focal Tversky Loss (FTL; [26]), which is a generalized version of the Tversky Loss (TL) defined in terms of the Tversky Index → = (1 − ) → = ( ) , (1)
For our training, we use = 0.7 and = 0.3. The parameter controls the non-linearity of the TL with respect to the , enabling the learning to focus on easier ( < 1) vs. harder ( > 1) examples. We use = 0.75 during our training. For the discriminator optimization, we use the Binary Cross-Entropy (BCE) loss. Specifically, our total discriminator loss is the average of two components: the discriminator applied on the generated mask (i.e., against a fake label), and applied on the true mask (i.e., the real label). For both the generator and discriminator, we use the Adam optimizer with an initial learning rate 5 × 10− 4 and 1 × 10− 4 respectively, decayed exponentially by = 0.95, applied every 5 epochs. model training runs. As expected, larger training samples provide much better performance, but we also find that the model pretrained on the COCO dataset provides noticeably better performance on the Floating Forests 3.2. Transfer learning strategy data, compared to both Λ → and also Λ . In fact, the Λ → is able to match the performance of the For our transfer learning based model training of Λ model with between 50-75% of the training Floating Λ → and Λ → , we load the weights of the Forests dataset. Λ and Λ models into the freshly initialized Λ In Figure 3, we show examples highlighting the difmodel architecture. To account for the 3 vs 4 channel ference between the generated masks from Λ and mismatch between the Λ , Λ and Λ , we load corresponding masks from Λ → and Λ → . model layer parameters excluding the input layer. For The sharpness of the kelp beds is poorly reconstructed each model, we train 5 diferent versions, using random by the Λ model but is well captured by the transfer subsets of 10%, 25%, 50%, 75% and 100% of the full learnt models (particularly when training Λ → Floating Forests data, to compare TL eficiency gains with more than 75% of the original data). The transfer from having a smaller dataset. For these experiments, learnt models are also better at capturing kelp beds not we also use only the first 6,967 un-augmented images identified in the original consensus data. For example, for re-training. We train the Λ → and Λ → both the ground truth and Λ fail to reveal the kelp models with the same hyper-parameter settings as the beds in the top left of the image, but these are picked up aforementioned “from scratch” models for 50 epochs. well by the transfer learnt models. This is likely due to the large diversity of the features 3.3. Results and discussion in the COCO dataset, making it a much more robust feature extraction network to transfer learn from. InWe find that our Λ , Λ and Λ generally pre- deed, compared to Λ → , the Λ → modeldict the annotation masks reasonably well (Figure 2), detected kelp beds are qualitatively better visually (e.g., qualitatively matching with the ground truths. Figures 3 Figure 3), especially at lower training data sizes. This is and 4 show our transfer learning results. In Figure 4, likely compounded with the lower feature diversity in we show our average validation loss for the diferent both the Floating Forest and Fat Checker data sets, given the fewer number of samples in the training data and low We also thank Lucy Collinson and the Electron Mivariety in target classes. croscopy Science Technology Platform (The Francis Crick Institute, London UK) for their input into this project. 3.3.1. Transfer learning approaches for citizen This work was supported in part by the Francis Crick science datasets Institute which receives its core funding from Cancer ReFor the Zooniverse platform, this study provides an av- search UK (FC001999), the UK Medical Research Council enue to build quick access for projects to use machine (FC001999), and the Wellcome Trust (FC001999). This learning frameworks for simple tasks (e.g., image seg- project has been made possible in part by grant number mentation), by transfer learning from existing models on 2020-225438 from the Chan Zuckerberg Initiative DAF, a small sample of volunteer annotated data sets. How- an advised fund of Silicon Valley Community Foundaever, despite the results presented here, there are still tion (H.S.). This publication uses data generated via the several key questions which need to be answered: Zooniverse.org platform, development of which is funded
Domain dependency: It is unclear how much of the by generous support, including a Global Impact Award performance gained from COCO was a ‘global truth’. from Google, and by a grant from the Alfred P. Sloan That is, whether COCO (or similarly diverse datasets) Foundation. are immediately applicable to out-of-domain data, for all domains, or if there are domain-specific restrictions References which allow these performance gains to occur on data such as Floating Forests. This requires more experiments with increasingly diferent data sets on Zooniverse to investigate the range of performance gains possible.
Task dependency: Previous studies on transfer learning across domains show significant variations in performance across diferent task types. For example, image classification tasks (e.g., [ 12, 17]) show lower gains than image segmentation based tasks (e.g., [18]). We need to further investigate the inherent dificulty associated with diferent tasks on Zooniverse projects, and how effectively they can be transferred between domains. [12], for example, show that significant boosts to performance is only provided by using in-domain transfer learning.
Target data purity: For Zooniverse projects, data labels are generally provided by volunteers and are aggregated based on volunteer consensus. In this study, we found that transfer learning can help mitigate data purity efects, since transfer learnt feature extraction models are generally robust to mislabeled data. The extent to which transfer learning models are sensitive to data purity efects needs to be further investigated.
In conclusion, we find that transfer learning can provide a significant boost to projects that contain similar tasks on Zooniverse. However, the extent to which this can be generalized across the full Zooniverse ecosystem is a question of ongoing study.
The authors would like to thank the Zooniverse volunteers without whom this work would not have been possible. RS, KM, LF, YZ, LT would like to acknowledge partial support from the National Science Foundation under grant numbers IIS 2006894 and OAC 1835530. Partial support by RS, KM, LF, TP, MS, TC, JS is acknowledged through Minnesota Partnership MNP IF#119.09. on, IEEE, 2018. [26] N. Abraham, N. Mefraz Khan, A Novel Focal Tversky loss function with improved Attention UNet for lesion segmentation, arXiv e-prints (2018) arXiv:1810.07842. arXiv:1810.07842.