Hand activity and speech comprise the most important modalities of human-to-agent interaction. So a multimodal interface can achieve more natural and effective human-agent interaction. In this paper, we suggest a novel technique for improving the performance of accelerometer-based hand activity recognition system using fusion of speech. The speech data is used in our experiment as the complementary sensor data to the acceleration data in an attempt to improve the performance of hand activity recognizer. This recognizer is designed to be capable of classifying nineteen hand activities. It consists of 10 natural gestures, e.g., 'go left', 'over here' and 9 emotional expressions by hand activity, e.g., 'I feel hot', 'I love you'. To improve performance of hand activity recognition using feature fusion, we propose a modified Time Delay Neural Network (TDNN) architecture with a dedicated fusion layer and a time normalization layer. Our experimental result shows that the performance of this system yields an improvement of about 6.96% compared to the use of accelerometers alone.
Interface technology using activity or gesture is one of the key functions for agent
system in ubiquitous computing environment. In general, accelerometers are currently
among the most widely studied wearable sensors for activity or gesture recognition,
thanks to their accuracy in the detection of human body movements, small in size, and
reasonable power consumptions[
In the study of Bharatula and others[
We use a human cognitive-based technique for improving the performance of
activity recognition. Humans do not depend only on their hearing in order to
recognize information. This fact is illustrated by the McGurk effect[
Currently, the recognition of human input using data fusion has been partially
achieved in a lip-reading system. The fusion algorithm can be carried out either at the
feature-level or the class-level[
Figure 2 shows the block diagram of the feature-level fusion. Two feature vectors
of accelerometer and speech signal are combined into a joint feature vector, and the
joint feature vector is used as input vector of fusion classifier. As already mentioned,
the feature fusion uses a single classifier to fuse two modalities. Several approach
have been proposed : fuzzy logic, Artificial Neural Network(ANN), Hidden Markov
Model(HMM), hybrid ANN-DTW(Dynamic time warping), hybrid ANN-HMM,
genetic algorithm, Support Vector Machines (SVM) etc[
Fig 2. The fusion at feature level
In this paper, we propose a feature fusion method in an attempt to improve the performance of accelerometer-based hand activity recognition. To develop hand activity recognizer with high performance, we present a modified TDNN architecture with a dedicated fusion and time normalization layer. 2
In this section, details of hand activity and the feature extraction module of speech and acceleration used in this paper are described. 2.1
The Speech Feature Extraction(SFE) module extracts feature vectors from the speech
signal. This module is comprised of the following components : an End Point
Detection(EPD) module based on the frame energy, a Feature Extraction(FE) module
based on Zero-Crossing with Peak Amplitude (ZCPA)[
The ZCPA model is more robust in noisy environments than other popularly used feature extraction methods, such as LPCC or MFCC. It is composed of cochlear bandpass filter, zero-crossing detector and peak detector. And frequency information is obtained by the zero-crossing detector, and intensity information is also incorporated by the peak detector. RASTA processing of speech is a bandpass modulation filtering, operating on the log spectral domain. Slow channel variations should in principle be removed. Finally, the SFE module captures 16 features per frame. 2.2
The acceleration data of the subject was collected using two MTx(Xsens technologies) accelerometers. These 3-axis accelerometers are accurate to 1.7G with tolerances within 0.2%. The accelerometers were mounted on wrist and sampled at 100Hz. Figure 3 shows MTx accelerometer with 3-axis and the attached type on the wrist.
In our experiment, we examine natural gestures and emotional expressions by hand activity. A natural gesture is defined as an action that everyone can understand in human-to-human communication. The supported natural gestures are as follows: ‘go right’, ‘go left’, ‘go up’, ‘go down’, ‘over here’, ‘go away’, ‘catch’, ‘release’, ‘open’ and ‘close’.
So an emotional expression by hand activity is defined as an action of hand according to the emotion change such as ‘I love you’, ‘I feel cold’, ‘I feel hot’, ‘I feel so-cold’, ‘I feel so-hot’, ‘I feel real-hot’, ‘I feel real-cold’, ‘Handshaking’ and ‘Goodbye’. But the linguistic definitions of emotional expression are often ambiguous. To address ambiguities in hand activity labels, test subjects were provided with image descriptions of each hand activity and short sentence descriptions. Figure 5 shows an example of image descriptions of each hand activity. Table 1 lists Korean utterance of each hand activity along with its short sentence description.
In 1989, it was shown that neural network model yields a high performance in the
speech recognition. The main goal of TDNN was to have a neural network
architecture for non-linear feature classification invariant under translation in time or
space. TDNN uses time-delay steps to represent temporal relationships. The
translation invariant classification is realized by sharing the connection weights of the
time delay steps[
We explain modified TDNN architecture to improve performance of hand activity recognition system. One of most difficult challenges in the feature-level fusion is the synchronization between accelerometer and speech data. In our system, speech features are extracted with the dimensions 64x16(where 64 is the number of frames and 16 is the number of coefficients). So the accelerometer-based hand activity features are extracted with the dimensions 120 x 6. Therefore, the method chosen to synchronize between the two feature spaces has a significant effect on the improvement of the recognition rate. We solve the synchronization problem by using a dedicated fusion layer and time normalization layer. Figure 6 is the modified TDNN architecture for data fusion. at fusion layer is given as.
z F'j f ( wSiFj zSi
wAiFj z Ai ) (NS Fj 1)
( NA Fj 1) Si Fj
Ai Fj (3) where Fj is the index of node at fusion layer, A shows the hand activity features. S denotes the speech feature and f is a sigmoid fusion. So N is the number of windows at TN layer, i is the index of node at TN layer, j is the index of node at fusion layer. z is the output value of TN layer, z` is the output value of fusion layer and w is weight.
In figure 6, The input layer of the Speech Network(SN) has 16 feature values at each 10ms interval, 64 frames, and overlap windows of 3 frames. And the input layer of the Accelerometer-based Activity Network(AAN) has 6 feature values at each 10ms, 120 frames, and overlap windows of 59. So the TN of the SN consists of 62 frames, 8 units per frame and overlap windows of 5 frames. And the TN of AAN consists of 62 frame, 3 units per frame and overlap windows of 5 frame. In the case of the SN, the 48 units of this input layer are fully interconnected to a layer of 8 TN units. In the case of the ANN, the 354 units of the input layer are fully interconnected to a layer of 3 TN units. Finally, the fusion layer consists of 58 frames and 4 units per frame. 4
The experimental data consists of 19 utterances recorded by a male. The subject is directed to perform each activity at a time accompanied by corresponding speech in a quiet office environment. For training, 50 sets of activity and speech data are used, and the other 50 sets are used as test patterns. Speech is recorded by a SHURE microphone and the accelerometer-based activity is recorded by two MTx on his wrists. To train gesture and speech, the data set is provided to the system at the learning rate of 0.1.
Table 2 compares the performance of the proposed fusion system to the system that uses accelerometer alone. In table 2, when the signal-to-noise ratio(SNR) decreases, the Fusion method does not degrade as much as Accelerometers alone case. An accelerometer is one of the most useful wearable sensors for activity recognition. The accuracy of the previous work only using accelerometer was around 85% ~90%, which may be good enough for some applications. In this paper, we present a multisensory-based fusion recognition system having more enhanced performance than accelerometer-based activity recognition. In our work, we designed a hand activity recognizer that can classify acceleration data and utterance into nineteen activities : 10 natural activities and 9 emotional expressions by hand activity.
To improve performance of hand activity recognition system, we propose the modified TDNN architecture with a dedicated fusion layer and time normalization layer. Using the proposed fusion layer, we solved the synchronization problem in feature-level fusion. Our experiment shows performance improvement of 6.96% when compared to an activity system using only accelerometer.