Classification of Human Actions Using Task fMRI Images

Abstract

In the past few years, the topic of brain signal analysis has become very popular in neuroscience. There are several approaches to imaging the brain. Classification of human activities with task fMRI is an important part of finding effective connectivity in human brain. This article is devoted to developing of an approach to constructing a classifier for human actions. Detailed definition of the basic notions used in analyzing fMRI images is provided. A review of datasets and methods for classifying fMRI images is presented with recommendations. Also, brief description of major international projects involved in brain analysis is provided. In conclusion, workflow and way forward to implementations is examined with description of proposed libraries to use for analysis, filtering, preprocessing, reading and writing fMRI and fitting classification models with it.

Keywords

• task fMRI analysis
• data intensive analysis
• human action classification

-

Neuroimaging is the common name for several methods that allow visualization of the structure, functions, and biochemical characteristics of the brain [ 1 ]. At the same time, neuroimaging techniques do not require surgical intervention and direct contact with the internal organs, since these technologies made it possible to non-invasive visualization of the structure and functionality of the brain, becoming a powerful tool for research and for medical diagnostics with the development of technology and computational methods.

Functional neuroimaging is used to measure aspects of the brain to understand the relationship between the activity of certain areas of the brain with specific mental functions.

There are several approaches to collect data about human brain for latter analysis: • Computed tomography (uses a series of x-rays aimed at the head from a large number of different directions); • Diffuse optical tomography (uses infrared radiation, measures the optical absorption of hemoglobin); • Optical Signal modified by an event (using infrared radiation); • Electroencephalography (EEG); • magnetic resonance imaging (MRI) (uses magnetic fields and radio waves without using ionizing radiation); • functional magnetic resonance imaging (fMRI).

Most brain analysis nowadays is implemented on the basis of fMRI images. Functional magnetic resonance imaging is based on the paramagnetic properties of hemoglobin and makes it possible to see changes in the blood circulation of the brain depending on its activity [ 2 ]. The essence of the method is that when certain parts of the brain work, the blood flow in them increases. Changes in blood flow are recorded, and images can tell which parts of the brain are activated when performing certain actions. fMRI image is a 4-dimensional array of voxels (spatial and time). This type of images allows to analyze the activity of various parts of the brain at some time point.

Over the past decades, researchers have managed to accumulate a large amount of fMRI data. fMRI images have a complex descriptive structure and require large resources for their storage, such as high-performance computing systems. Besides that, the amount of data surpasses tens of terabytes of data, requiring special compute-intensive platforms to deal with these datasets. These facts underline the multidisciplinary nature of neuroscience and the need to develop IT methods for it.

There are several types of relationships in the brain – structural, functional and effective connectivity [ 3 ]. Effective connectivity, which describes the amount of information transmitted by information flows in the presence of any stimulus or the absence of incentives per unit of time, is among the most interest in analyzing brain images [ 3 ].

Classification of human actions using task fMRI images is an important part of analyzing effective connectivity. As an example, in Human Connectome Project participants were asked to perform seven tasks related to the following categories: Emotion, Gambling, Language, Motor, Relational, Social and Working Memory. Based on the task fMRI data obtained [ 4 ], a mathematical model is built relating images to a specific task. However, the model lacks accuracy and does not deal with data-intensive platforms.

This work is aimed at development of methods and tools to process large datasets in neurophysiology domain, build classification model to analyze effective connectivity of the brain using task fMRI images. The research is carried out as thesis for Masters Program “Big data: infrastructures and problem-solving techniques” under the department of Computational Mathematics and Cybernetics of Moscow State University. 2

Problem Statement and Formalization of Application Domain

Specification of the application domain is depicted on Fig. 1. Effective connectivity describes the causal interaction between units of connectivity (usually brain regions). It is described by the amount of information transmitted over information flows in the presence of any stimulus or absence of incentives per unit of time. The connected unit (region of interest) transmits information signal to another connected unit by information flow, receives information from another unit by information flow.

A problem statement is formulated as follows: present an approach for dealing with large incoming datasets of fMRI, preprocess them, build classification model for preprocessed dataset and validate it on some data. Classification task formulates as follows: using task fMRI images relate them into seven groups of task, which a person was doing during that session. 3

Related Works

Classification Methods

One of the pioneering works on the classification of signals of the human brain is based on testing statistical hypotheses [ 5 ]. Using the t-test, the signal is classified into two classes. There are actual problems of binary classification, for example, to distinguish Alzheimer's patients from healthy people, solved with the help of t-test [ 6, 7 ]. The tcriterion has a number of advantages and disadvantages. The advantages include: ease of calculation; ease of interpretation; resistant to emissions; works with even a small amount of data. The disadvantages of this method include: assumptions that the data have a normal distribution; residues are independent and have a normal distribution.

Later works are based on linear classification methods such as: support vector machine (SVM [ 8, 9 ]), general linear models (GLM), etc. Such classifiers are suitable for solving problem of binary classification. However, the use of linear classification methods imposes significant restrictions on the dataset: it must be linearly separable. In [ 5 ], researchers analyze the brain signal using the support vector machine (SVM [ 8, 9 ]). The advantage of this approach is that linear models are easily interpreted, are trained with small samples, and are not prone to overfitting. Linear models have several disadvantages: they do not approximate complex surfaces; dataset must be independent.

Works on analyzing MRI images based on using neural networks [ 10–12 ] began to appear relatively recently. The main advantages are that neural networks allow to build complex separating surfaces, significantly increasing the quality of the model. The second essential advantage of neural networks is that it allows to implement multi-class classification without significantly complicating the model. Disadvantages are that models are prone to overfitting and require vast computational resources. Also, sometimes the dimension of the learning model is too large.

In [ 4 ] researchers analyze the signal from the brain using deep neural networks. Multiple architectures are built with two and more hidden layers and the quality of work of different architectures is compared. Experiments are performed on a dataset from the Human Connectome project. In [ 13 ], the authors take ready-made convolutional neural net (CNN (LeNet-5)) and successfully classify functional MRI data of Alzheimer’s subjects. Accuracy on test dataset reached 96.85%. Usage of CNN allows to extract useful tags from images and approximate complex structures. Recent studies show that modern architects of convolutional neural networks classify the image more qualitatively than humans.

Based on the result of the study, it is recommended to use CNN.

Related Neurophysiology Projects and Dataset Description

Human Connectome Project. The main goal of the Human Connectome Project (HCP) [ 14 ] is to describe the structural and functional connection in the brain of a healthy young adult. Based on the data collected by HCP, a large number of studies are annually conducted to extract functional and structural dependencies between different parts of the brain. The database contains information about more than 1200 participants. The Human Connectome Project brings together a large group of researchers all around working in the field of neuroscience. All data can be downloaded from the project website for free after registration.

Human Connectome Project has a separate task fMRI dataset published. Each participant during the fMRI session was asked to perform some tasks from the following groups: • Emotion processing: participants were asked to map several images to each other. • Gambling: participants were asked to play a simple card game. • Language processing: after listening to a short audio file, participants were asked to answer simple questions. • Motor: participants were asked to move the divided body part. • Social cognition: short video was presented to the participants and asked to answer the question whether the movements of objects in the clips are related to each other in some way.

Each subject has corresponding behavioral data, including age, weight, etc. Also, each task fMRI has a corresponding design file, which stores meta information about the experiment, the number and name of the experimental conditions recorded, and an indication of the path to the timing files. Timing files contain the time of the stimulus (onset) and its duration (duration). In addition, functional data, masks, files with the time of appearance of stimuli and their duration, files with data about experiments (design files) are stored. The dimension of one image is (91, 109, 91, 274), i.e. total 902629 voxels with 274 values.

Other Projects. The Human Brain Project (HBP) is a large research project to study the structure and analyze the functional connectivity between different parts of the brain. The project involves hundreds of scientists from 26 countries and 135 partner institutions. The goal of this project is to create a joint research infrastructure to enable researchers around the world to develop knowledge in the field of neurobiology, computer technology, and medicine related to the brain.

The BRAIN Initiative project was created at the initiative of the White House in 2013. This project was created as a private-public research initiative. A large number of neuroscientists from 30 different countries are involved in this project. At the first stages of the project, researchers will analyze the activity of neurons in mice and other animals, and at later stages of the project a functional map of dependencies of various parts human brain will be built. It is assumed that these studies will help researchers discover the secrets of brain disorders such as Alzheimer's and Parkinson's, depression.

1000 Functional Connectomes Project is a database of functional MRI images taken at rest. The purpose of this project is to collect fMRI images during rest. When visualizing the brain during rest, random low-frequency oscillations of large amplitude occur, which correlate in different functionally related areas. Based on the data obtained, researchers build maps of interaction between different areas of the brain. The database contains data from more than 1,400 participants collected independently in 35 international centers 4

Proposed Approach

Workflow

Workflow for proposed approach is depicted on Fig. 2. First, input data is preprocessed using NIPY [ 15 ] library. Next, regressors and contrasts (artifacts for handling task fMRI) are constructed. Next, using these artifacts and preprocessed data, the classification model is built. There are several libraries to work with CNN. Later, model is validation against testing dataset.

It is assumed that preprocessing step and construction of contrasts and regressors are built with PySpark [ 16 ] library in distributed manner.

Causality Model as Input for Classification

One of the main ideas of the approach, which differs it from other, is to use output of Dynamic Causal Modeling (DCM) [ 17 ] as the input to classification model. Dynamic causal modeling (DCM) is a general Bayesian structure for drawing conclusions about hidden neural states based on measurements of brain activity. DCM provides a posteriori estimates of neurobiological interpreted values, such as the effective strength of synaptic connections between neuronal populations and their context-dependent modulation (i.e., how experiment factors influence these values). In other words, with the help of DCM, it can be understood how a specific change in conditions during an experiment affects the activation of brain area.

DCM is stated as (linear or non-linear) differential equations. They describe hidden dynamics of neural populations. DCM models seek to ensure being neurophysiological interpreted.

The idea of using DCM as input for further classification is following. DCM is not a theoretical simulation of neuronal processes in its pure form, but a method that includes both a theoretical calculation (model prediction) and a validation on real data (implemented using Bayesian inversion). The key feature of the DCM method is its dependence on experimental data. Its equations take into account the influence of experimental manipulations on the dynamics of the system: the experimental conditions are included in the model as input data that either controls the local responses of the system or changes the connections. So, e.g., if a person was watching a video during fMRI, DCM will tend to seek the activation of the brain area responsible for visual cognition. This knowledge cam vastly increase classification accuracy.

Data Neuroimaging Format

Task fMRI neuroimage is a four-dimensional array of voxels, three dimensions describe the position of voxels in space, the fourth – in time. A voxel has an index in the three-dimensional spatial array of the fMRI neuro-image and has n values for each t (t=1...m) of the fourth dimension of the fMRI neuro-image (i.e., is a time series).

NIFTI [ 18 ] allows to store data in several ways: (1) as in ANALYZE in 2 files (1 file is a header file with the extension. hdr; 2 file – the data itself in .img format); (2) or all in one file with the extension. nii. NIFTI also supports working with compressed data (.gz). The first 4 measurements out of 7 are predefined to represent spatial and temporal coordinates (1–3 spatial, 4 temporal, 5–7 adjustable).

Header structure has size of 348 bytes. Some header fields are: ─ Information about data collection (Dim info): char dim_info – stores the directions of frequency, phase coding, the direction in which the volume increased when receiving data; ─ Image dimensions: short dim [ 8 ] contains information about image dimensions. dim [i] represents the length of the i-th dimension; ─ Intent-fields: short intent_code is a code showing the statistical nature of the data, some codes require additional parameters, which are either indicated in the float intent_p * fields (if applicable to the picture as a whole), or form the 5th dimension (if these parameters are different for each voxel). The readable intent name can be stored in the char intent_name [ 16 ] field; ─ Data type: int datatype shows the type of stored data; short bitpix contains information on the number of bits per voxel; ─ Slice acquisition: char slice_code, short slice_start, short slice_end and float slice_duration store information about the fmri time distribution, and should be used together with char dim_info containing fieldslice_dim. The short slice_start and short slice_end fields indicate which layer is the first and last for a particular mri. Layers out of range are considered added to the file (and not received with mri, usually contain 0). The float slice_duration field indicates the amount of time needed to produce a single layer; ─ Voxel dimensions: The float pixdim [ 8 ] contains the dimension of each voxel, by analogy with short dim [ 8 ]. But the values in float pixdim [0] must be equal to –1 or 1; ─ Voxel Offset (Voxel o set): The int vox_o ff set field indicates the beginning of the data itself from the beginning of the file (for data contained in .nii file), for a pair of files (.hdr / .img), the field must contain 0 if there is no additional data other than the picture in .img not contained; ─ Data scaling: The values stored in each pixel can be linearly scaled in different units.

(fields float scl_slope and float scl_inter); ─ Display range (Data display): For files that store scalar data, the cal_min and cal_max fields define the intended display range when the image is opened; ─ Measurement units: Both temporal and spatial units used in dim [i] (i=1..4) (and for pixdim) are stored in the char xyzt_units field. Bits 1-3 are used for spatial measurements, 4–6 – for temporary, 7–8 – are not used; ─ Image orientation information (Orientation information): In NIFTI, it is possible to uniquely store orientation information. The file standard assumes that the voxel coordinates correspond to the center of this voxel. It is assumed that the system of world coordinates is "RAS +". The format represents 3 different methods of mapping voxel coordinates (i, j, k) to world (x, y, z). The main one is that the world coordinates are determined by scaling the voxel size.

Image Analysis and Processing Tools

The NIPY library consists of several parts that enable the user to perform not only simple operations with fMRI images such as reads and writes, but also analysis algorithms. This library includes the following projects: ─ Nipype provides a unified interface for working with fMRI images; ─ NiBabel is module that allows you to work with a large number of medical and neurovizualizable file formats such as GIFTI, NIfTI1, NIfTI2, CIFTI-2, MINC1, MINC2, AFNI BRIK/HEAD, MGH and ECAT as well as Philips PAR/REC. This library allows you to both read and write the listed file formats. This library has a Python interface that makes it quite simple and easy to use. The library’s website provides detailed installation information and examples with explanations on the use of the library ─ PyMVPA is a set of algorithms that are intended for statistical image analysis. In this package, implemented algorithms for classification, clustering and regression, created unified interfaces for interacting with standard data analysis libraries such as scikit-learn, shogun, MDP, etc

OpenCV [ 19–21 ] is first of all this computer vision library, there are several thousand high-performance image processing algorithms implemented in this library. This library is distributed under the BSD license, therefore the code of this library can be modified and used in commercial projects. The OpenCV library has a modular structure. Researchers use this library to pre-process MRI images and extract functions from MRI images.

Recently, a lot of articles appeared trying to classify fMRI images using convolutional neural networks. There are many different libraries and software products that implement neural network architectures. The Keras library is one of the most popular. This library is written in the Python programming language, with operations performed on TensorFlow. For a training of a convolutional neural network, a huge number of trained images are required. Keras contains within itself the architecture of popular convolutional neural networks, which were trained in ImageNet [ 22–24 ]. 5

Conclusion

Classification problem is stated for task fMRI. Specification of application domain is provided. A review of datasets and methods for classifying fMRI images is presented with recommendations. Also, brief description of major international projects involved in brain analysis is provided. In conclusion, workflow and wayforward to implementations is examined with description of proposed libraries to use for analysis, filtering, preprocessing, reading and writing fMRI and fitting classification models with it.

Acknowledgements

This work is supervised by Dmitry Kovalev, Federal Research Center “Informatics and Control” of Russian Academy of Sciences.

References

1. Duncan , J.: Neuroimaging methods to evaluate the etiology and consequences of epilepsy . Epilepsy Research , 131 - 140 ( 2002 ).

2. Shulman , R.G. , Rothman , D.L. , Behar , K.L. , and Hyder , F. : Energetic basis of brain activity implications for neuroimaging . Trends Neurosci , 489 - 495 ( 2004 ).

3. Schlösser , R. , Gesierich , T. , Kaufmann B. , Vucurevic G. , Hunsche S. , and Gawehn , J.: Altered effective connectivity during working memory performance in schizophrenia: a study with fMRI and structural equation modeling . NeuroImage , 751 - 763 ( 2003 ).

4. Koyamada , S. , Shikauchia , Y. , Nakaea , K. , Koyamaa , M. , and Ishiia , S. : Deep learning of fMRI big data: a novel approach to subject-transfer decoding . Stat.ML, arXiv:1502.00093v1 ( 2015 ).

5. Mahmoudi , A. , Takerkart , S. , Regragui , F. , Boussaoud , D. , and Brovelli , A. : Multivoxel Pattern Analysis for fMRI Data: A Review. Computational and Mathematical Methods in Medicine ( 2012 ).

6. Friston , K. , Frith , C. , and Liddle , P. : Comparing functional (PET) images: the assessment of significant change . Journal of Cerebral Blood Flow and Metabolism , 690 - 699 ( 1991 ).

7. Hall , D. and Miller , H. : The Theory of Stochastic Processes. 1nd Edition . Routledge ( 1977 ).

8. Cortes , C. and Vapni , V.: Support-vector networks . Machine Learning , 273 - 297 ( 1995 ).

9. Vapnik , V. : The Nature of Statistical Learning . 2nd Edition . Springer ( 1995 ).

10. Karnowski , T.: Deep machine learning a new frontier in artificial intelligence research . Computational Intelligence Magazine , 13 - 18 ( 2010 ).

11. Grady , C. , Sarraf , S. , and Saverino , C. : Age differences in the functional interactions among the default, frontoparietal control and dorsal attention networks . Neurobiology of Aging ( 2016 ).

12. Shelhamer , E. , Donahue , J. , Karayev , S. , Long J. , and Girshick , R.: Convolutional architecture for fast feature embedding . Proceedings of the ACM International Conference on Multimedia , 675 - 678 ( 2014 ).

13. Sarraf , S. and Tofighi , G.: Classification of Alzheimer's disease using fMRI data and deep learning convolutional neural networks . cs.CV, arXiv:1603.08631v1 ( 2016 ).

14. Barch , D.M. , Burgess , G.C. , Harms , M.P. , Petersen , S.E. , Schlaggar , B.L. , and Corbetta , M. : Function in the human connectome: task-fMRI and individual differences in behavior . Neuroimage , 169 - 189 ( 2013 ).

15. NIPY - neuroimaging software. https://nipy.org/

16. Drabas , T. and Lee , D. : Learning PySpark . 1nd Edition . Packt Publishing ( 2017 ).

17. Friston , K. : Dynamic causal modeling and Granger causality comments on: The identification of interacting networks in the brain using fMRI: Model selection, causality and deconvolution . Neuroimage 58 , 303 - 305 . ( 2011 ).

18. Gorgolewski , K. , Auer , T. , Calhoun , V. , Craddock , C. , Das , S. , and Duff , E.: The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments . Scientific Data ( 2016 ).

19. OpenCv - Open Source Computer Vision Library. https://opencv.org/

20. Bradski , G. and Kaehler , A. : Learning OpenCV: Computer Vision with the OpenCV Library . 1nd Edition. O'Reilly Media ( 2008 ).

21. Garrido , G. and Joshi , P.: OpenCV 3.x with Python by example: make the most of OpenCV and Python to build applications for object recognition and augmented reality . 2nd Edition . Packt Publishing ( 2018 ).

22. Keras - open-source neural-network library . https://keras.io/

23. Gulli , A. and Pal , S. : Deep Learning with Keras: Implementing deep learning models and neural networks with the power of Python. 1nd Edition . Packt Publishing ( 2017 ).

24. Williams , A. : Deep Learning with Keras: Introduction to Deep Learning with Keras . 2nd Edition. CreateSpace Independent Publishing Platform ( 2017 ).