=Paper=
{{Paper
|id=Vol-2523/paper38
|storemode=property
|title=
Classification of Human Actions Using Task fMRI Images (short paper)
|pdfUrl=https://ceur-ws.org/Vol-2523/paper38.pdf
|volume=Vol-2523
|authors=Dmitrii Sergeev
|dblpUrl=https://dblp.org/rec/conf/rcdl/Sergeev19
}}
==
Classification of Human Actions Using Task fMRI Images (short paper)
==
https://ceur-ws.org/Vol-2523/paper38.pdf
Classification of Human Actions Using Task fMRI
Images
Dmitrii Sergeev1
1Moscow State University, Moscow, Russia
SerDimIgor@gmail.com
Abstract. In the past few years, the topic of brain signal analysis has become
very popular in neuroscience. There are several approaches to imaging the brain.
Classification of human activities with task fMRI is an important part of finding
effective connectivity in human brain. This article is devoted to developing of an
approach to constructing a classifier for human actions. Detailed definition of the
basic notions used in analyzing fMRI images is provided. A review of datasets
and methods for classifying fMRI images is presented with recommendations.
Also, brief description of major international projects involved in brain analysis
is provided. In conclusion, workflow and way forward to implementations is ex-
amined with description of proposed libraries to use for analysis, filtering, pre-
processing, reading and writing fMRI and fitting classification models with it.
Keywords: task fMRI analysis, data intensive analysis, human action classifi-
cation.
1 Introduction
Neuroimaging is the common name for several methods that allow visualization of the
structure, functions, and biochemical characteristics of the brain [1]. At the same time,
neuroimaging techniques do not require surgical intervention and direct contact with
the internal organs, since these technologies made it possible to non-invasive visuali-
zation of the structure and functionality of the brain, becoming a powerful tool for re-
search and for medical diagnostics with the development of technology and computa-
tional methods.
Functional neuroimaging is used to measure aspects of the brain to understand the
relationship between the activity of certain areas of the brain with specific mental func-
tions.
There are several approaches to collect data about human brain for latter analysis:
• Computed tomography (uses a series of x-rays aimed at the head from a large num-
ber of different directions);
• Diffuse optical tomography (uses infrared radiation, measures the optical absorption
of hemoglobin);
• Optical Signal modified by an event (using infrared radiation);
• Electroencephalography (EEG);
• magnetic resonance imaging (MRI) (uses magnetic fields and radio waves without
using ionizing radiation);
Copyright © 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
395
• functional magnetic resonance imaging (fMRI).
Most brain analysis nowadays is implemented on the basis of fMRI images. Functional
magnetic resonance imaging is based on the paramagnetic properties of hemoglobin
and makes it possible to see changes in the blood circulation of the brain depending on
its activity [2]. The essence of the method is that when certain parts of the brain work,
the blood flow in them increases. Changes in blood flow are recorded, and images can
tell which parts of the brain are activated when performing certain actions. fMRI image
is a 4-dimensional array of voxels (spatial and time). This type of images allows to
analyze the activity of various parts of the brain at some time point.
Over the past decades, researchers have managed to accumulate a large amount of
fMRI data. fMRI images have a complex descriptive structure and require large re-
sources for their storage, such as high-performance computing systems. Besides that,
the amount of data surpasses tens of terabytes of data, requiring special compute-inten-
sive platforms to deal with these datasets. These facts underline the multidisciplinary
nature of neuroscience and the need to develop IT methods for it.
There are several types of relationships in the brain – structural, functional and ef-
fective connectivity [3]. Effective connectivity, which describes the amount of infor-
mation transmitted by information flows in the presence of any stimulus or the absence
of incentives per unit of time, is among the most interest in analyzing brain images [3].
Classification of human actions using task fMRI images is an important part of ana-
lyzing effective connectivity. As an example, in Human Connectome Project partici-
pants were asked to perform seven tasks related to the following categories: Emotion,
Gambling, Language, Motor, Relational, Social and Working Memory. Based on the
task fMRI data obtained [4], a mathematical model is built relating images to a specific
task. However, the model lacks accuracy and does not deal with data-intensive plat-
forms.
This work is aimed at development of methods and tools to process large datasets in
neurophysiology domain, build classification model to analyze effective connectivity
of the brain using task fMRI images. The research is carried out as thesis for Masters
Program “Big data: infrastructures and problem-solving techniques” under the depart-
ment of Computational Mathematics and Cybernetics of Moscow State University.
2 Problem Statement and Formalization of Application Domain
Specification of the application domain is depicted on Fig. 1. Effective connectivity
describes the causal interaction between units of connectivity (usually brain regions).
It is described by the amount of information transmitted over information flows in the
presence of any stimulus or absence of incentives per unit of time. The connected unit
(region of interest) transmits information signal to another connected unit by infor-
mation flow, receives information from another unit by information flow.
396
Fig. 1. Specification of neurophysiology application domain
A problem statement is formulated as follows: present an approach for dealing with
large incoming datasets of fMRI, preprocess them, build classification model for pre-
processed dataset and validate it on some data. Classification task formulates as fol-
lows: using task fMRI images relate them into seven groups of task, which a person
was doing during that session.
3 Related Works
Classification Methods
One of the pioneering works on the classification of signals of the human brain is based
on testing statistical hypotheses [5]. Using the t-test, the signal is classified into two
classes. There are actual problems of binary classification, for example, to distinguish
Alzheimer's patients from healthy people, solved with the help of t-test [6, 7]. The t-
criterion has a number of advantages and disadvantages. The advantages include: ease
of calculation; ease of interpretation; resistant to emissions; works with even a small
amount of data. The disadvantages of this method include: assumptions that the data
have a normal distribution; residues are independent and have a normal distribution.
Later works are based on linear classification methods such as: support vector ma-
chine (SVM [8, 9]), general linear models (GLM), etc. Such classifiers are suitable for
solving problem of binary classification. However, the use of linear classification meth-
ods imposes significant restrictions on the dataset: it must be linearly separable. In [5],
researchers analyze the brain signal using the support vector machine (SVM [8, 9]).
The advantage of this approach is that linear models are easily interpreted, are trained
397
with small samples, and are not prone to overfitting. Linear models have several disad-
vantages: they do not approximate complex surfaces; dataset must be independent.
Works on analyzing MRI images based on using neural networks [10–12] began to
appear relatively recently. The main advantages are that neural networks allow to build
complex separating surfaces, significantly increasing the quality of the model. The sec-
ond essential advantage of neural networks is that it allows to implement multi-class
classification without significantly complicating the model. Disadvantages are that
models are prone to overfitting and require vast computational resources. Also, some-
times the dimension of the learning model is too large.
In [4] researchers analyze the signal from the brain using deep neural networks. Mul-
tiple architectures are built with two and more hidden layers and the quality of work of
different architectures is compared. Experiments are performed on a dataset from the
Human Connectome project. In [13], the authors take ready-made convolutional neural
net (CNN (LeNet-5)) and successfully classify functional MRI data of Alzheimer’s
subjects. Accuracy on test dataset reached 96.85%. Usage of CNN allows to extract
useful tags from images and approximate complex structures. Recent studies show that
modern architects of convolutional neural networks classify the image more qualita-
tively than humans.
Based on the result of the study, it is recommended to use CNN.
Related Neurophysiology Projects and Dataset Description
Human Connectome Project. The main goal of the Human Connectome Project
(HCP) [14] is to describe the structural and functional connection in the brain of a
healthy young adult. Based on the data collected by HCP, a large number of studies are
annually conducted to extract functional and structural dependencies between different
parts of the brain. The database contains information about more than 1200 participants.
The Human Connectome Project brings together a large group of researchers all around
working in the field of neuroscience. All data can be downloaded from the project web-
site for free after registration.
Human Connectome Project has a separate task fMRI dataset published. Each par-
ticipant during the fMRI session was asked to perform some tasks from the following
groups:
• Emotion processing: participants were asked to map several images to each other.
• Gambling: participants were asked to play a simple card game.
• Language processing: after listening to a short audio file, participants were asked to
answer simple questions.
• Motor: participants were asked to move the divided body part.
• Social cognition: short video was presented to the participants and asked to answer
the question whether the movements of objects in the clips are related to each other
in some way.
Each subject has corresponding behavioral data, including age, weight, etc. Also,
each task fMRI has a corresponding design file, which stores meta information about
the experiment, the number and name of the experimental conditions recorded, and an
indication of the path to the timing files. Timing files contain the time of the stimulus
(onset) and its duration (duration). In addition, functional data, masks, files with the
398
time of appearance of stimuli and their duration, files with data about experiments (de-
sign files) are stored. The dimension of one image is (91, 109, 91, 274), i.e. total 902629
voxels with 274 values.
Other Projects. The Human Brain Project (HBP) is a large research project to study
the structure and analyze the functional connectivity between different parts of the
brain. The project involves hundreds of scientists from 26 countries and 135 partner
institutions. The goal of this project is to create a joint research infrastructure to enable
researchers around the world to develop knowledge in the field of neurobiology, com-
puter technology, and medicine related to the brain.
The BRAIN Initiative project was created at the initiative of the White House in
2013. This project was created as a private-public research initiative. A large number
of neuroscientists from 30 different countries are involved in this project. At the first
stages of the project, researchers will analyze the activity of neurons in mice and other
animals, and at later stages of the project a functional map of dependencies of various
parts human brain will be built. It is assumed that these studies will help researchers
discover the secrets of brain disorders such as Alzheimer's and Parkinson's, depression.
1000 Functional Connectomes Project is a database of functional MRI images taken
at rest. The purpose of this project is to collect fMRI images during rest. When visual-
izing the brain during rest, random low-frequency oscillations of large amplitude occur,
which correlate in different functionally related areas. Based on the data obtained, re-
searchers build maps of interaction between different areas of the brain. The database
contains data from more than 1,400 participants collected independently in 35 interna-
tional centers
4 Proposed Approach
Workflow
Workflow for proposed approach is depicted on Fig. 2. First, input data is preprocessed
using NIPY [15] library. Next, regressors and contrasts (artifacts for handling task
fMRI) are constructed. Next, using these artifacts and preprocessed data, the classifica-
tion model is built. There are several libraries to work with CNN. Later, model is vali-
dation against testing dataset.
It is assumed that preprocessing step and construction of contrasts and regressors are
built with PySpark [16] library in distributed manner.
Fig. 9. Workflow for constructing classifiers for human activity
Causality Model as Input for Classification
One of the main ideas of the approach, which differs it from other, is to use output of
Dynamic Causal Modeling (DCM) [17] as the input to classification model. Dynamic
causal modeling (DCM) is a general Bayesian structure for drawing conclusions about
399
hidden neural states based on measurements of brain activity. DCM provides a poste-
riori estimates of neurobiological interpreted values, such as the effective strength of
synaptic connections between neuronal populations and their context-dependent mod-
ulation (i.e., how experiment factors influence these values). In other words, with the
help of DCM, it can be understood how a specific change in conditions during an ex-
periment affects the activation of brain area.
DCM is stated as (linear or non-linear) differential equations. They describe hidden
dynamics of neural populations. DCM models seek to ensure being neurophysiological
interpreted.
The idea of using DCM as input for further classification is following. DCM is not
a theoretical simulation of neuronal processes in its pure form, but a method that in-
cludes both a theoretical calculation (model prediction) and a validation on real data
(implemented using Bayesian inversion). The key feature of the DCM method is its
dependence on experimental data. Its equations take into account the influence of ex-
perimental manipulations on the dynamics of the system: the experimental conditions
are included in the model as input data that either controls the local responses of the
system or changes the connections. So, e.g., if a person was watching a video during
fMRI, DCM will tend to seek the activation of the brain area responsible for visual
cognition. This knowledge cam vastly increase classification accuracy.
Data Neuroimaging Format
Task fMRI neuroimage is a four-dimensional array of voxels, three dimensions de-
scribe the position of voxels in space, the fourth – in time. A voxel has an index in the
three-dimensional spatial array of the fMRI neuro-image and has n values for each t
(t=1...m) of the fourth dimension of the fMRI neuro-image (i.e., is a time series).
NIFTI [18] allows to store data in several ways: (1) as in ANALYZE in 2 files (1
file is a header file with the extension. hdr; 2 file – the data itself in .img format); (2)
or all in one file with the extension. nii. NIFTI also supports working with compressed
data (.gz). The first 4 measurements out of 7 are predefined to represent spatial and
temporal coordinates (1–3 spatial, 4 temporal, 5–7 adjustable).
Header structure has size of 348 bytes. Some header fields are:
─ Information about data collection (Dim info): char dim_info – stores the directions
of frequency, phase coding, the direction in which the volume increased when re-
ceiving data;
─ Image dimensions: short dim [8] contains information about image dimensions. dim
[i] represents the length of the i-th dimension;
─ Intent-fields: short intent_code is a code showing the statistical nature of the data,
some codes require additional parameters, which are either indicated in the float in-
tent_p * fields (if applicable to the picture as a whole), or form the 5th dimension (if
these parameters are different for each voxel). The readable intent name can be
stored in the char intent_name [16] field;
─ Data type: int datatype shows the type of stored data; short bitpix contains infor-
mation on the number of bits per voxel;
─ Slice acquisition: char slice_code, short slice_start, short slice_end and float
slice_duration store information about the fmri time distribution, and should be used
together with char dim_info containing fieldslice_dim. The short slice_start and
400
short slice_end fields indicate which layer is the first and last for a particular mri.
Layers out of range are considered added to the file (and not received with mri, usu-
ally contain 0). The float slice_duration field indicates the amount of time needed to
produce a single layer;
─ Voxel dimensions: The float pixdim [8] contains the dimension of each voxel, by
analogy with short dim [8]. But the values in float pixdim [0] must be equal to –1 or
1;
─ Voxel Offset (Voxel o set): The int vox_o ff set field indicates the beginning of the
data itself from the beginning of the file (for data contained in .nii file), for a pair of
files (.hdr / .img), the field must contain 0 if there is no additional data other than the
picture in .img not contained;
─ Data scaling: The values stored in each pixel can be linearly scaled in different units.
(fields float scl_slope and float scl_inter);
─ Display range (Data display): For files that store scalar data, the cal_min and
cal_max fields define the intended display range when the image is opened;
─ Measurement units: Both temporal and spatial units used in dim [i] (i=1..4) (and for
pixdim) are stored in the char xyzt_units field. Bits 1-3 are used for spatial measure-
ments, 4–6 – for temporary, 7–8 – are not used;
─ Image orientation information (Orientation information): In NIFTI, it is possible to
uniquely store orientation information. The file standard assumes that the voxel co-
ordinates correspond to the center of this voxel. It is assumed that the system of
world coordinates is "RAS +". The format represents 3 different methods of mapping
voxel coordinates (i, j, k) to world (x, y, z). The main one is that the world coordi-
nates are determined by scaling the voxel size.
Image Analysis and Processing Tools
The NIPY library consists of several parts that enable the user to perform not only
simple operations with fMRI images such as reads and writes, but also analysis algo-
rithms. This library includes the following projects:
─ Nipype provides a unified interface for working with fMRI images;
─ NiBabel is module that allows you to work with a large number of medical and neu-
rovizualizable file formats such as GIFTI, NIfTI1, NIfTI2, CIFTI-2, MINC1,
MINC2, AFNI BRIK/HEAD, MGH and ECAT as well as Philips PAR/REC. This
library allows you to both read and write the listed file formats. This library has a
Python interface that makes it quite simple and easy to use. The library’s website
provides detailed installation information and examples with explanations on the use
of the library
─ PyMVPA is a set of algorithms that are intended for statistical image analysis. In
this package, implemented algorithms for classification, clustering and regression,
created unified interfaces for interacting with standard data analysis libraries such as
scikit-learn, shogun, MDP, etc
OpenCV [19–21] is first of all this computer vision library, there are several thou-
sand high-performance image processing algorithms implemented in this library. This
library is distributed under the BSD license, therefore the code of this library can be
401
modified and used in commercial projects. The OpenCV library has a modular struc-
ture. Researchers use this library to pre-process MRI images and extract functions from
MRI images.
Recently, a lot of articles appeared trying to classify fMRI images using convolu-
tional neural networks. There are many different libraries and software products that
implement neural network architectures. The Keras library is one of the most popular.
This library is written in the Python programming language, with operations performed
on TensorFlow. For a training of a convolutional neural network, a huge number of
trained images are required. Keras contains within itself the architecture of popular
convolutional neural networks, which were trained in ImageNet [22–24].
5 Conclusion
Classification problem is stated for task fMRI. Specification of application domain is
provided. A review of datasets and methods for classifying fMRI images is presented
with recommendations. Also, brief description of major international projects involved
in brain analysis is provided. In conclusion, workflow and wayforward to implementa-
tions is examined with description of proposed libraries to use for analysis, filtering,
preprocessing, reading and writing fMRI and fitting classification models with it.
Acknowledgements
This work is supervised by Dmitry Kovalev, Federal Research Center “Informatics and
Control” of Russian Academy of Sciences.
References
1. Duncan, J.: Neuroimaging methods to evaluate the etiology and consequences of epilepsy.
Epilepsy Research, 131–140 (2002).
2. Shulman, R.G., Rothman, D.L., Behar, K.L., and Hyder, F.: Energetic basis of brain activity
implications for neuroimaging. Trends Neurosci, 489–495 (2004).
3. Schlösser, R., Gesierich, T., Kaufmann B., Vucurevic G., Hunsche S., and Gawehn, J.: Al-
tered effective connectivity during working memory performance in schizophrenia: a study
with fMRI and structural equation modeling. NeuroImage, 751–763 (2003).
4. Koyamada, S., Shikauchia, Y., Nakaea, K., Koyamaa, M., and Ishiia, S.: Deep learning of
fMRI big data: a novel approach to subject-transfer decoding. Stat.ML, arXiv:1502.00093v1
(2015).
5. Mahmoudi, A., Takerkart, S., Regragui, F., Boussaoud, D., and Brovelli, A.: Multivoxel
Pattern Analysis for fMRI Data: A Review. Computational and Mathematical Methods in
Medicine (2012).
6. Friston, K., Frith, C., and Liddle, P.: Comparing functional (PET) images: the assessment
of significant change. Journal of Cerebral Blood Flow and Metabolism, 690–699 (1991).
7. Hall, D. and Miller, H.: The Theory of Stochastic Processes. 1nd Edition. Routledge (1977).
8. Cortes, C. and Vapni, V.: Support-vector networks. Machine Learning, 273–297 (1995).
9. Vapnik, V.: The Nature of Statistical Learning. 2nd Edition. Springer (1995).
10. Karnowski, T.: Deep machine learning a new frontier in artificial intelligence research.
Computational Intelligence Magazine, 13–18 (2010).
402
11. Grady, C., Sarraf, S., and Saverino, C.: Age differences in the functional interactions among
the default, frontoparietal control and dorsal attention networks. Neurobiology of Aging
(2016).
12. Shelhamer, E., Donahue, J., Karayev, S., Long J., and Girshick, R.: Convolutional architec-
ture for fast feature embedding. Proceedings of the ACM International Conference on Mul-
timedia, 675–678 (2014).
13. Sarraf, S. and Tofighi, G.: Classification of Alzheimer’s disease using fMRI data and deep
learning convolutional neural networks. cs.CV, arXiv:1603.08631v1 (2016).
14. Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., and Corbetta, M.:
Function in the human connectome: task-fMRI and individual differences in behavior. Neu-
roimage, 169–189 (2013).
15. NIPY – neuroimaging software. https://nipy.org/
16. Drabas, T. and Lee, D.: Learning PySpark. 1nd Edition. Packt Publishing (2017).
17. Friston, K.: Dynamic causal modeling and Granger causality comments on: The identifica-
tion of interacting networks in the brain using fMRI: Model selection, causality and decon-
volution. Neuroimage 58, 303–305. (2011).
18. Gorgolewski, K., Auer, T., Calhoun, V., Craddock, C., Das, S., and Duff, E.: The brain
imaging data structure, a format for organizing and describing outputs of neuroimaging ex-
periments. Scientific Data (2016).
19. OpenCv – Open Source Computer Vision Library. https://opencv.org/
20. Bradski, G. and Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Li-
brary. 1nd Edition. O’Reilly Media (2008).
21. Garrido, G. and Joshi, P.: OpenCV 3.x with Python by example: make the most of OpenCV
and Python to build applications for object recognition and augmented reality. 2nd Edition.
Packt Publishing (2018).
22. Keras – open-source neural-network library. https://keras.io/
23. Gulli, A. and Pal, S.: Deep Learning with Keras: Implementing deep learning models and
neural networks with the power of Python. 1nd Edition. Packt Publishing (2017).
24. Williams, A.: Deep Learning with Keras: Introduction to Deep Learning with Keras. 2nd
Edition. CreateSpace Independent Publishing Platform (2017).
403