In recent years, lip reading has attracted extensive research attention as deep learning shows great potential in computer vision. In this work, we proposed Multi-Dilation Temporal Convolutional Networks (MD-TCN) to predict individual words in lip reading tasks. Although Temporal Convolutional Networks (TCNs) have lately demonstrated promising importance in a variety of video sequence tasks, ordinary TCNs' bottom layers still have tiny receptive fields and are unable to reproduce complicated temporal dynamics in scenarios of lip-reading tasks. To tackle this problem, we use dual dilated convolution in the network instead of typical dilated convolution to capture more powerful temporal features. Furthermore, our method incorporates a self-attention block after each convolutional layer to further enhance the classification and screening capabilities of the model. On the lip-reading in the wild (LRW) dataset, our MDTCN Model achieves 85.7 percent accuracy and is an effective method for individual word prediction.
Lip Reading, also known as Visual Speech Recognition, is a task of recognizing words based on the movement of the speaker's lips without audio assistance. Due to the complexity of Lip Reading, human lip readers need to take a long period of professional training to ensure the accuracy, which usually requires a high cost. Machine Lip Reading has become a very hot topic in video processing due to its relative low cost and even higher accuracy than human lip readers. And with the development of informatization, a real-time and fast lip-reading solution is required in many scenarios. Especially for speech recognition in a high-noise audio signal environment, the fusion of video signal and speech signal can greatly improve the accuracy of recognition, and the robustness of the system can also be greatly improved. At the same time, because of the nature of the lip-reading task, it is easy to apply the model to other video recognition tasks, such as action recognition and emotional semantic analysis.
This paper will build an end-to-end Visual Speech Recognition model and test it on both English
and Mandarin data to better evaluate the model performance. Researchers generally divide Visual
Speech Recognition into two steps: the front-end structure of visual feature extraction and the back-end
structure of time series information recognition [
This article will be divided into five main sections. The second part of the article will review the classic literature on Visual Speech Recognition and the deep neural network architecture used in this paper, such as discussing previous research on Visual Speech Recognition and temporal convolutional networks. The third part of the article will focus on the feature extraction model of 3D-CNN + denseResnet18 and the sequence model of TCN, and discuss their advantages. The fourth part of the article provides an in-depth discussion of the experiments, introducing the dataset, experimental setup, experimental results, and a discussion of the results. Finally, the fifth section of the article will conclude by discussing the limitations of this study and directions for future research.
Before the deep learning gold rush, Lip Reading was mostly accomplished by depending on
manually derived features, such as Discrete Cosine Transform (DCT) [
Though RNN-like neural networks such as GRU and LSTM have been widely employed for time series applications, sequential models with higher parallelism and faster training have also received extensive attention in recent years. Lea et al. [15] first proposed Temporal Convolutional Networks (TCNs) for video action segmentation. The encoder and decoder of this network are both two-layer onedimensional convolutional blocks. In 2018, Bai et al. [16] suggested an effective and simple TCN structure that surpassed RNN models in a variety of time series problems. Martinez et al. [17] first proposed Multi-scale TCN architecture to mix up long-term and short-term information, which improved robustness of network over time domain. Although the Temporal Convolutional Network introduced in [16] is a causal one-way model, it can also be adapted to an acausal structure in practical classification tasks (such as our lip-reading task). The work of [9] adopts a non-causal TCN structure and introduces densely connected layers to improve the performance of the network on complex datasets. However, these models may not consider the autocorrelation within the series. Many studies [18, 19] have shown that considering the autocorrelation of sequential models can effectively increase the prediction accuracy. As a result, we propose a TCN-embedded temporal self-attention strategy to increase the capture of sequence autocorrelations.
The main framework of our technique is depicted in Fig. 1. The input is a raw video from a dataset with the shape × × × , where denotes the temporal dimension and , denotes the input video's height and width, respectively. After we adapt a face detection to input video, it can easily be transformed and cropped to gray-scale mouth Region of Interests (RoIs). We start by using a 3D convolutional layer to approximately extract the spatial-temporal features with the shape of × × 1 × 1 × 1, where 1 and 1 are the modified height and width, and 1 is the feature channel number, as described in [17]. We apply a 2D ResNet-18 [20] on top of this layer to generate features with the form × × 2 × 2 × 2. To summarize the information of lip characteristics and compress the dimension to × × 2, the following layer uses spatial average pooling. After the pooling layer, the temporal dynamics are modeled using our suggested Multi-Dilation TCN (MD-TCN). To complete the temporal information into 4 channels, the output tensor ( × 3) is routed through another average pooling layer. The Softmax layer that follows predicts single word probability.
TCN is much better than other sequential models in parallel processing and training speed. However, due to the size limitation of the convolution kernel, the receptive field of TCN is usually limited. In other words, it is difficult to comprehensively consider information with a long interval. The most basic TCN method [15] usually solves this problem by increasing the number of hidden layers and adding dilation layer by layer. Although the top layers may have a broad receptive field, the lowest levels still have a very small receptive field. Furthermore, because of the significant dilation factor of upper layers in TCN, convolutions must be applied at very distant time steps.
Inspired by [21], we adopt a dual dilated convolution (DDC) to replace the traditional dilated convolution. Two convolutions with different dilation factors are combined in DDC (shown as the orange and green blocks in Fig. 1). Smaller levels of the first convolution (orange block in Fig. 1) have a lower dilation factor, which increases exponentially as the number of layers increases. The second convolution (green block in Fig. 1) starts with a strong dilation factor in the lower levels and gradually decreases as the number of layers grows. Finally, in order to ensure that the output shape is similar to standard TCN, we use a 1*1 convolution layer to transform the shape. Since higher layer don’t have the problem of conception field size, we only use DDC in the lower layers of the MD-TCN, while we still use traditional Single Dilation Convolution in the higher layers of the network.
In the meantime, because the size of the conventional TCN convolution kernel is constant, all activations of a certain layer have the same temporal receptive field. As a result, this kind of network generally cannot consider both long-term and short-term information. We expect that the network will be able to capture receptive fields over a range of time scales, allowing short-term and long-term data to be combined for feature encoding. We use a TCN with several convolution kernels to do this. Each temporal convolution in this multi-kernel TCN variation now has many branches, each with a distinct kernel size. Each convolutional layer therefore combines data from many time scales.
In view of the above two points, we finally generate the Multi-Dilation Temporal Convolutional Networks (MD-TCN) to replace the standard TCN (shown in Fig. 1). In this network, we create four temporal branches with kernel sizes of 1, 3, 5, and 7, respectively. And in each branch, we also use dual dilated convolution to replace ordinary dilated convolution. Hence each layer of this MB-TCN has eight branches. After each convolution, we employ Batch Norm layers [24] to speed up training converge, and we apply dropouts [25] using dropping probability of 0.5 for regularization. Meanwhile, as in standard TCN, we also reuse two identical convolutional layer in each MB-TCN to achieve better model effectiveness.
In our experiments, we adopt a total of four convolutional layers structure, because this setup can balance the training speed and accuracy. Also, the number of layers of the hyperparameter DDC in our model is set to 2, which is proved to be the best value in our experiments, and the specific experiments will be shown later in the discussion of hyperparameters.
Each lip position in a lip motion model is frequently linked to other positions in the sequence. If each word corresponds to a position, the lips will likewise be in a regular posture. This prompted us to create a method that would allow us to choose the most relevant context for the features we needed to extract. As a result, we propose a temporal attention strategy built in TCN for assigning weights to contextual information at each time step in an adaptable manner. We incorporate a self-attention block after each MB-TCN to account for the autocorrelation of lip-reading sequences.
Our studies were done using the Lip Reading in the Wild (LRW) [11] dataset, which is the biggest publicly accessible dataset for lipreading individual English words. The LRW dataset has a vocabulary of 500 English words. Each video sequence segment in LRW has a length of 1.16 seconds and was recorded from over 1000 speakers in a BBC show (29 video frames). This dataset contains 538 766 sequences, which are separated into 488 766/25 000/25 000 for training, validation, and testing. Due to the enormous number of speakers and the wide changes in lighting conditions, head positions, and speech speeds, this dataset is also one of the most difficult dataset in Lip Reading.
We preprocessed the video similar to the methods introduced in [17]. We first detected face marks and did face alignments for every single video. Then we can easily crop the videos into the size of 96 × 96 and converted them to grayscale. In order to simulate different lighting and positions between different videos, we did a bunch of data argumentations such as random horizontal flip, random brightness jitter 20% and random contrast jitter 20%. Finally, to avoid over-fitting to training dataset, we randomly select 1 to 3 frames in a video and randomly delete or copy them. Thus our model can be more powerful to fit different application scenarios.
Since the easy part of the dataset is often correct even for the simplest model, the accuracy of the model is ultimately determined by the most difficult part of the dataset. We observed that pretrain the whole model on a relatively small sub-dataset is also an effective way to adjust hyperparameters, test model performance and accelerate training. We extract the 50 hardest words based on the current stateof-the-art open-source model [17] for LRW to create the sub-dataset. As a result, we use this pretraining method since it may significantly speed up training.
The whole model is trained end-to-end, with all weights initialized using the pretrained model, as illustrated in Fig. 1. With a batch size of 32, an initial learning rate of 0.0004, and a weight decay of 1e4, we train for 90 epochs. To acquire the weights at the best performing point, we measure the accuracy using the validation set provided by LRW. We adopt Adam [22] and cross entropy as optimizer and loss function. The learning rate is gradually reduced from its original value to zero using the cosine scheduling [23].
Our approach is conducted in the PyTorch framework 3.8.10 [27]. We used a 1080Ti GPU for our experiments. The LRW sub-dataset takes roughly 5 hours to train an end-to-end hyperparametric validation model. To train a single model from beginning to end, it takes roughly 5 days. Our network is lighter than other works, and our training time is significantly lowered.
On the LRW sub-dataset, we analyze several structural parameters of the MD-TCN model in order to find the best performing one. In particular, we validate the effect of the selection of DDC layers on the results while freezing other hyperparameters, such as the total number of layers and the kernel size. To accelerate training, we only train on LRW sub-dataset for 20 epochs and fix the weight of the frontend (ResNet).
72.8 72.6 73.5 73.4
To determine the best DC-TCN structure, we evaluated the effect of number of dual dilated layers on the LRW sub-dataset, while keeping the values of other hyperparameters constant. As shown in Table 1, it is noticed that the best performance is achieved when the number of DDC layers is 4, i.e., all dilated layers are dual dilated layers. The performance is significantly increased when the number of DDC layer is greater than or equal to two. We note that with more than two layers, increasing DDC layers will barely improve the test accuracy, so we decided to use two DDC layers in the subsequential experiments for a good balance between accuracy and speed.
For effectiveness of attention block, we also do an experiment on whether the Self-Attention Block is enabled. Here we do this experiment by using the DDC layers of 2 to ensure consistency with the final experimental hyperparameters. The result shows that when using Self-Attention Block, the training speed slow down but can achieve a better test accuracy. It is worth stating that since the sub-dataset was selected from the most difficult 10% of LRW and we only trained for 20 epochs, the results shown here are not as accurate as they could be.
On the LRW dataset, we compare the performance of our technique versus several baseline methods in Table 3. On the LRW dataset, our technique achieves an accuracy of 85.7 percent, which has a 0.2 percent increase over other similarly structured networks [10].
We chose the LRW sub-dataset as the training dataset for both hyperparameter tuning and pretraining because of accelerated training. However, we did not investigate whether the LRW sub-dataset can truly reflect the LRW dataset and whether the hyperparameters that perform well in the LRW subdataset also perform well in the LRW dataset. Therefore, we extracted the results of the completed training model to investigate how accurate the part of the results corresponding to the sub-dataset.
Although our model is fast and has very good accuracy, there is still much room for optimization. First, although the LRW sub-dataset is selected from the LRW, the data might be different in distribution, so the network architecture applicable to the sub-dataset may not be optimal on the LRW. Therefore, with sufficient arithmetic power, it is better to select hyperparameters directly on the LRW to guarantee optimal results. Second, in this paper, we improve the accuracy by constructing a dual dilation layer, where the dilation can also be carefully designed like using more than two dilation in one layer. Third, we have taken the data pre-processing approach of adding and deleting frames for data argumentation, but this may affect the speaker's rhythm. We would like to take the approach of directly increasing or decreasing the video length (without changing the frame rate) to simulate different speech rates of the speaker, which has the potential to significantly improve the model robustness.
In this work, we propose MD-TCN for isolated word recognition. Our model has a very good performance on LRW by solving the problem of sparse perception field and data autocorrelation. We also adopted a novel method for data argumentation that randomly copy and delete frames to improve model robustness. Finally, we use sub-dataset to select hyperparameters and accelerate training. [8] Themos Stafylakis, Muhammad Haris Khan, and Georgios Tzimiropoulos, “Pushing the boundaries of audiovisual word recognition using residual networks and lstms,” Computer Vision and Image Understanding, vol. 176, pp. 22–32, 2018. [9] Pingchuan Ma, Brais Martinez, Stavros Petridis, and Maja Pantic, “Towards practical lipreading with distilled and efficient models,” ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. [10] Feng, Dalu, Shuang Yang, Shiguang Shan, and Xilin Chen. "Learn an effective lip reading model without pains." arXiv preprint arXiv:2011.07557 (2020). [11] Chung, Joon Son, and Andrew Zisserman. "Lip reading in the wild." Asian conference on computer vision. Springer, Cham, 2016. [12] Yang, Shuang, Yuanhang Zhang, Dalu Feng, Mingmin Yang, Chenhao Wang, Jingyun Xiao, Keyu Long, Shiguang Shan, and Xilin Chen. "LRW-1000: A naturally-distributed large-scale benchmark for lip reading in the wild." In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1-8. IEEE, 2019. [13] T. Stafylakis, M. H. Khan, and G. Tzimiropoulos. Pushing the boundaries of audiovisual word recognition using residual networks and lstms. Computer Vision and Image Understanding, 176:22–32, 2018. [14] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman. Lip reading sentences in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3444–3453. IEEE, 2017. [15] Lea, Colin, Rene Vidal, Austin Reiter, and Gregory D. Hager. "Temporal convolutional networks: A unified approach to action segmentation." In European Conference on Computer Vision, pp. 4754. Springer, Cham, 2016. [16] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018. [17] Martinez, Brais, Pingchuan Ma, Stavros Petridis, and Maja Pantic. "Lipreading using temporal convolutional networks." In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6319-6323. IEEE, 2020. [18] Wan, Renzhuo, Shuping Mei, Jun Wang, Min Liu, and Fan Yang. "Multivariate temporal convolutional network: A deep neural networks approach for multivariate time series forecasting." Electronics 8, no. 8 (2019): 876. [19] Dai, Rui, Luca Minciullo, Lorenzo Garattoni, Gianpiero Francesca, and François Bremond. "Selfattention temporal convolutional network for long-term daily living activity detection." In 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-7. IEEE, 2019. [20] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016. [21] Li, Shi-Jie, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall. "Ms-tcn++: Multistage temporal convolutional network for action segmentation." IEEE transactions on pattern analysis and machine intelligence (2020). [22] Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014). [23] Loshchilov, Ilya, and Frank Hutter. "Sgdr: Stochastic gradient descent with warm restarts." arXiv preprint arXiv:1608.03983 (2016). [24] Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." In International conference on machine learning, pp. 448456. PMLR, 2015. [25] Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. "Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15, no. 1 (2014): 1929-1958. [26] Stafylakis, Themos, and Georgios Tzimiropoulos. "Combining residual networks with LSTMs for lipreading." arXiv preprint arXiv:1703.04105 (2017). [27] Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen et al. "Pytorch: An imperative style, high-performance deep learning library." Advances in neural information processing systems 32 (2019). [28] Petridis, Stavros, Themos Stafylakis, Pingehuan Ma, Feipeng Cai, Georgios Tzimiropoulos, and Maja Pantic. "End-to-end audiovisual speech recognition." In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 6548-6552. IEEE, 2018. [29] Wang, Chenhao. "Multi-grained spatio-temporal modeling for lip-reading." arXiv preprint arXiv:1908.11618 (2019). [30] Zhang, Yuanhang, Shuang Yang, Jingyun Xiao, Shiguang Shan, and Xilin Chen. "Can we read speech beyond the lips? rethinking roi selection for deep visual speech recognition." In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 356363. IEEE, 2020. [31] Xu, Bo, Cheng Lu, Yandong Guo, and Jacob Wang. "Discriminative multi-modality speech recognition." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14433-14442. 2020.