In the last few years, the technological advancement of wearable cameras has led to an increasing interest in egocentric (first-person) vision, thanks to its ability to capture activities from the user's perspective with applications in a variety of diferent tasks, from human-object interaction to action prediction and anticipation. In contrast, continuous head movement, variations in lighting conditions and diferences in the way humans complete the same task represent a source of bias that strengthens the coupling between the model's predictions and the training domain, afecting its ability to generalize to new environments. Several Domain Adaptation (DA) techniques have been proposed to make models more robust. Among these, Unsupervised Domain Adaptation (UDA) combines labeled source data and unlabeled target data to close the gap between diferent domains. However, real-world applications require more flexibility, as target samples are often scarce, unrepresentative or even private. Test Time Adaptation (TTA) appears to be a viable solution to these issues, with adaptation performed directly at test time under the simple assumption that input samples provide clues on the actual distribution of the target domain which could be used to improve predictions. With TTA, models undergo multiple adaptation steps at test time by minimizing an adaptation loss on target data and updating normalization statistics. This work provides a comparative analysis of multiple adaptation techniques on the EPIC-Kitchens dataset. Experiments indicate strong accuracy improvements over the unadapted baselines, suggesting that TTA efectively improves model performance in dynamic environments.
1. Introduction puter vision have spread to countless tasks, from image classification and object detection, to action recognition and video generation. Recently, the availability of light and cheap wearable cameras has made first-person video recording much more accessible, marking the birth of egocentric vision. The first-person perspective opens up mans interact with the world, paving the way for more human-aware robots. In this work, we focus on Egocentric Action Recognition (EAR), addressing one of the main issues that make this task challenging on egocentric videos.
Model predictions tend to be deeply correlated with IIT HPC Infrastructure for the Availability of High Performance Computing (Franklin) and CINECA Award through the ISCRA Initiative, in part by FAIR - Future Artificial Intelligence Research, and in part by European Union Next-Generation EU through PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 is dificult to achieve. Even though UDA is not realistic in the egocentric setting, the idea of adapting models to the target samples is powerful.
In this work, we address the application of Test Time new opportunities for a more in-depth study of how hu- too large. With Unsupervised Domain Adaptation (UDA), 3. Method adaptation process to the test phase, under the simple assumption that input samples provide clues on the actual distribution of the target domain which could be used to improve predictions. Compared to UDA, TTA allows for continuous adaptation which is suitable for the dynamic nature of egocentric vision.
The TTA methods addressed in this work can be divided in two groups: backpropagation free and loss based methods. The former are based uniquely on the update of the parameters of the network that do not require gradient updates, like the normalization statistics of the Batch Normalization layers of the network. Updates of the normalization statistics ensure that the activation maps within the internal layers of the network remain in the usual range seen during training. The second group, lossbased methods, are based on the minimization of a loss function computed over the model predictions or on the intermediate outputs of the network. This work provides an overview of these TTA methods with applications on action recognition and on optical flow estimation.
The goal of TTA is to update the predictive hypothesis learned by the model at test time, directly on the test data.
Unlike UDA and DG, TTA does not require a training process with access to source and target data (Table 1).
As labels are not available for the target domain, it is assumed that the two label spaces coincide. In addition, the absence of target labels limits the number of techniques that can be applied to improve the quality of the model assumptions. TTA approaches can be classified into two main categories, depending on whether or not they require gradient updates through backpropagation. 2. Related Works The first approach is based on updating the Batch Normalization statistics collected by the model at training Video Domain Adaptation The goal of Unsuper- time. Replacing normalization statistics collected on the vised Domain Adaptation (UDA) is to learn how to bridge training data with online estimates of the target data has the domain gap between a source labeled domain and a been shown to improve the robustness of the model in non-annotated target domain directly from data. UDA presence of covariance shift [ 8]. The second approach, in video tasks focuses on the alignment of the temporal loss-based methods, use gradient updates on the model. dimension across diferent domains [ 1, 2], on exploiting Among these methods, class losses improve the quality of complementary information between diferent modali- the predictions, e.g. by optimizing some side properties ties [3] and on learning invariant representations across of the class distribution produced by the model, like its domains and modalities [4, 5]. The objective of DG is to entropy. Feature level losses attempts to adapt the model learn predictive functions that are more robust to out-of- by operating on its intermediate outputs. distribution data, using multiple source domains during training but without accessing target data. In this context, RNA-Net [6] introduces a loss to align the feature norms 3.1. Batch normalization of diferent modalities. VideoDG [ 7] aligns the temporal relations between diferent frames across domains.
The first class of methods for TTA derives from the assumption that domain shift primarily results in a deviation from the batch normalization statistics collected at Test Time Adaptation Test Time Adaptation (TTA) training time. A natural solution is to replace the running defers the adaptation phase to test time. Previous works statistics of the normalization layers with the mean and focused on updating the statistics of the normalization variance of the target samples [8, 9]. These update enlayers with those of the target samples [8, 9]. Authors sure that the output of the normalization layers remains of [10] introduce an auxiliary task that is jointly trained in the same range encountered at training time, as this with the main task during training, and finetuned on the is the only portion of the input space explored during target data during the test phase. ViTTA [11] aligns the training and model behavior may become unpredictable statistics of diferent temporal augmentations of the same outside this region. The authors of [13] propose -BN video to those seen during training. T3A [12] computes to mix the source and target statistics, to reduce the dispseudo-prototypes for each class label and classifies sam- crepancy between source and target activations without ples based on the distance from these prototypes. moving too far from the activations seen during training. For each BN layer , a new set of normalization statistics ( (̃ ) , ̃ () )is computed as a weighted average of the old (1) matrix defined as: which represent the probabilities that the samples in the current batch belong to classes and , respectively. More compactly, this is equivalent to computing a confusion
C, = y⋅, y⋅, . where controls the contribution of each sample to give more importance to those with a low entropy, i.e. higher confidence. This coeficient is computed as: (4) (5) (6) (7) = (1 + exp(−ℍ(y,⋅ ))) ∑=1 (1 +exp(−ℍ(y,⋅ ))) where is the batch size and ℍ(y,⋅ ) is the entropy of the predictions. To rebalance the contributions of the diferent classes, the
C is row normalized and the loss is defined as the sum of all the of-diagonal entries of the matrix: ℓ =
1
∑ =1 ≠
∑ C, . (2)
Indeed, the confusion matrix should be ideally close to the identity matrix, indicating low between-class confusion and strong prediction confidence. estimates and the new running estimates: ̂() = (1 − )(̂ ) + (
) ̂ () = (1 − ) ̂ () + √var( ), where ∈ [0, 1] controls the balance between the old and new statistics. This method does not require backpropagation and can be easily combined with the other adaptation techniques.
The entropy of a probability distribution measures the uncertainty of its outcomes. In a classification task, it is preferable for the model to produce predictions with very low entropy as higher entropy indicates confusion in the predictive function learned by the model. A natural solution might be to update the network weights along the direction that minimizes the entropy of the predictions, as an efort to strengthen the network’s confidence and obtain more clear decision boundaries.
ℓ (y) = −
∑ ∑ , log , 1
Minimization of prediction entropy does not guarantee the best solution, as the model may become even more radicalized on incorrect predictions or produce always the same prediction. average prediction .̃ mization (IM) mitigates the detrimental efect of the entropy loss in the presence of an unbalanced test dataset by enforcing diversification in the model predictions. The IM loss ℓ is defined as the sum of the average entropy of the predictions ,⋅ and the negative entropy of the 1
∑ =1 =1 =1 ℓ (y) = − ∑ , log , + ∑ ̃ log ̃ , (3) where is the batch size and the number of classes. The ifrst reduces the uncertainty of the predictions, while the second ensures that they remain globally diferent to avoid the pitfalls of entropy minimization alone.
Minimum Class Confusion (MCC) loss [14] targets a reduction of the pairwise class confusion of the model predictions, i.e. the situation in which a sample is ambiguously classified into two diferent classes class and with an equally high probability. The degree of pairwise confusion between Information Maximization (IM) Information Maxi- formulation can be trivially extended to the TTA setting: tion, e.g. the dot product, of the predictions ⋅, and ⋅, , two classes and is measured using a similarity func- the estimation process itself.
Alignment (RNA) loss [6] tackles the normunbalance problem in the target domain by re-balancing the contributions of the diferent modalities during the testing phase [15]. Since RNA is an unsupervised loss, its ℓ = ( [ℎ( 1)] [ℎ( 2)]
2 − 1) norm. where 1 and 2 are the features extracted from two diferent modalities 1 and 2 and ℎ(⋅)is the usual L2
Optical flow is a robust modality for action recognition but its computation is expensive [16]. Several approaches have been proposed to solve the optical flow estimation task directly from data [17, 18] with real-time performance [17] and may be suitable for online EAR [19]. A EAR pipeline could integrate rough estimates of the optical flow at test time, while still relying on more expensive algorithms for training. However, this introduces a gap in the distribution of the optical flow seen during training, compared to the test phase. In this work, we evaluate TTA techniques to adapt existing models to the optical lfow estimated with diferent approaches and to improve
Optical flow models are typically trained on synthetic datasets [20], which may result in noisy estimates in real-world scenarios. In this section, we propose a loss function inspired by [21] to align the boundaries of the estimated optical flow to the visual edges of the objects in the frame. For horizontal edges, the loss is defined as: 1 3 (1) ℓ ℎ, = ∑ exp (− ∑ | |) | | , (8) while an equivalent loss is defined for vertical edges. 4. Experiments All experiments proposed in this work were conducted on the EPIC-Kitchens-55 dataset [22], following the experimental protocol defined by [ 4], which restricts its analysis to the three largest kitchens in terms of number of labeled samples, hereafter referred to as D1, D2 and D3. Unless otherwise specified, models are pre-trained using RNA-Net [6] on two source domains and and tested on a possibly diferent target domain . Accuracy is averaged across of all unique combinations , ⟶ .
Five clips are uniformly sampled from the video, each consisting of 16 frames adjacent RGB frames and 16 optical flow frames, with stride 2. The visual samples are augmented using random crops, scale jitters and horizontal flips. RGB and Flow feature extractors use an I3D model [23]. Audio is processed from chunks of 1.28-seconds, converted into a log-spectrogram and fed to a BN-Inception model [24]. Features are then fed to a classifier and the outputs of diferent clips are averaged to compute the final predictions.
The target data set is processed in batches of 32 action samples, each consisting of 5 clips. Clips are adapted according to their index within each sample and an adaptation step is performed for each group to update the model parameters, i.e. the first clips are processed, then the second ones and so on. Once all the clips have been processed, the model is evaluated over the same 5 clips and the predictions are averaged.
Audio is the most robust modality, even though the drop Samples are annotated using one of eight verb classes. losses. The only exception is Audio, which sufers a before producing the final predictions. Although multistep adaptation may not be realistically applicable due to the increased latency, it is interesting from a theoretical point of view to analyze the robustness of the adaptation techniques and the adaptability of the diferent modalities. Table 4 shows that RGB+Flow is quite robust to multi-stage adaptation, while the accuracy of RGB+Audio decreases rapidly, as also shown in Fig. 1. Comparison with UDA TTA is much more limited compared to UDA as it exploits only a small batch of target data to adapt the model. On the other side, this allows TTA to adapt the model more specifically for the current batch. Therefore, the adaptation steps are based on a very narrow view of the target domain, as all updates are discarded once a new batch of data is available. The combination of RNA-Net and the proposed TTA techniques outperforms state-of-the-art methods on both RGB-Flow and RGB-Audio (Table 5).
the efect of the L1 smoothness loss on the optical flow frames estimated by FlowFormer [18]. Adaptation is performed online, e.g. without resetting the model after each batch, using batch size 1 and learning rate 0.0001.
The FlowFormer model was pretrained on the Sintel dataset [20]. The output of FlowFormer [18] is clean but still exhibits noisy edges and imperfections. With TTA, the estimated flow becomes more sharp and clean. 5. Conclusion Experiments indicate strong performance drops when models are tested on unseen domains. TTA is able partially cover this gap and to further increase the accuracy of DG models, improving by a significant margin over the best state-of-the-art UDA models when the models are evaluated on unseen domains. Furthermore, TTA demonstrates promising results when applied to the optical flow estimation task and is able to reduce the performance drop experienced by EAR models when tested with the optical flow estimated by real-time approaches.
Systems 34 (2021) 2427–2440. [13] F. You, J. Li, Z. Zhao, Test-time batch statis[1] M.-H. Chen, Z. Kira, G. AlRegib, J. Yoo, R. Chen, tics calibration for covariate shift, arXiv preprint J. Zheng, Temporal attentive alignment for large- arXiv:2110.04065 (2021). scale video domain adaptation, in: Proceedings of [14] Y. Jin, X. Wang, M. Long, J. Wang, Minimum class the IEEE/CVF International Conference on Com- confusion for versatile domain adaptation, in: Euputer Vision, 2019, pp. 6321–6330. ropean Conference on Computer Vision, Springer, [2] A. Jamal, V. P. Namboodiri, D. Deodhare, 2020, pp. 464–480.
K. Venkatesh, Deep domain adaptation in action [15] M. Plananamente, C. Plizzari, B. Caputo, Test-time space., in: BMVC, volume 2, 2018, p. 5. adaptation for egocentric action recognition, in: [3] L. Yang, Y. Huang, Y. Sugano, Y. Sato, Interact before International Conference on Image Analysis and align: Leveraging cross-modal knowledge for do- Processing, Springer, 2022, pp. 206–218. main adaptive action recognition, in: Proceedings [16] B. K. Horn, B. G. Schunck, Determining optical of the IEEE/CVF Conference on Computer Vision lfow, Artificial intelligence 17 (1981) 185–203. and Pattern Recognition, 2022, pp. 14722–14732. [17] D. Sun, X. Yang, M.-Y. Liu, J. Kautz, Pwc-net: Cnns [4] J. Munro, D. Damen, Multi-modal domain adapta- for optical flow using pyramid, warping, and cost tion for fine-grained action recognition, in: Pro- volume, in: Proceedings of the IEEE conference on ceedings of the IEEE/CVF conference on computer computer vision and pattern recognition, 2018, pp. vision and pattern recognition, 2020, pp. 122–132. 8934–8943. [5] X. Song, S. Zhao, J. Yang, H. Yue, P. Xu, R. Hu, [18] Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Chai, Spatio-temporal contrastive domain adap- H. Qin, J. Dai, H. Li, Flowformer: A transformer tation for action recognition, in: Proceedings of architecture for optical flow, in: Computer Vision– the IEEE/CVF Conference on Computer Vision and ECCV 2022: 17th European Conference, Tel Aviv, Pattern Recognition, 2021, pp. 9787–9795. Israel, October 23–27, 2022, Proceedings, Part XVII, [6] M. Planamente, C. Plizzari, E. Alberti, B. Caputo, Springer, 2022, pp. 668–685.
Domain generalization through audio-visual rela- [19] G. Goletto, M. Planamente, B. Caputo, G. Averta, tive norm alignment in first person action recog- Bringing online egocentric action recognition into nition, in: Proceedings of the IEEE/CVF Winter the wild, IEEE Robotics and Automation Letters 8 Conference on Applications of Computer Vision, (2023) 2333–2340.
2022, pp. 1807–1818. [20] D. J. Butler, J. Wulf, G. B. Stanley, M. J. Black, A [7] Z. Yao, Y. Wang, J. Wang, P. Yu, M. Long, Videodg: naturalistic open source movie for optical flow evalgeneralizing temporal relations in videos to novel uation, in: A. Fitzgibbon et al. (Eds.) (Ed.), European domains, IEEE Transactions on Pattern Analysis Conf. on Computer Vision (ECCV), Part IV, LNCS and Machine Intelligence (2021). 7577, Springer-Verlag, 2012, pp. 611–625. [8] Z. Nado, S. Padhy, D. Sculley, A. D’Amour, B. Laksh- [21] R. Jonschkowski, A. Stone, J. T. Barron, A. Gordon, minarayanan, J. Snoek, Evaluating prediction-time K. Konolige, A. Angelova, What matters in unsubatch normalization for robustness under covariate pervised optical flow, in: European Conference on shift, arXiv preprint arXiv:2006.10963 (2020). Computer Vision, Springer, 2020, pp. 557–572. [9] S. Schneider, E. Rusak, L. Eck, O. Bringmann, [22] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, W. Brendel, M. Bethge, Improving robustness A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, against common corruptions by covariate shift T. Perrett, W. Price, et al., Scaling egocentric viadaptation, Advances in Neural Information Pro- sion: The epic-kitchens dataset, in: Proceedings cessing Systems 33 (2020) 11539–11551. of the European Conference on Computer Vision [10] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, M. Hardt, (ECCV), 2018, pp. 720–736.
Test-time training with self-supervision for gener- [23] J. Carreira, A. Zisserman, Quo vadis, action recognialization under distribution shifts, in: International tion? a new model and the kinetics dataset, in: proconference on machine learning, PMLR, 2020, pp. ceedings of the IEEE Conference on Computer Vi9229–9248. sion and Pattern Recognition, 2017, pp. 6299–6308. [11] W. Lin, M. J. Mirza, M. Kozinski, H. Possegger, [24] E. Kazakos, A. Nagrani, A. Zisserman, D. Damen, H. Kuehne, H. Bischof, Video test-time adap- Epic-fusion: Audio-visual temporal binding for egotation for action recognition, arXiv preprint centric action recognition, in: Proceedings of the arXiv:2211.15393 (2022). IEEE/CVF International Conference on Computer [12] Y. Iwasawa, Y. Matsuo, Test-time classifier adjust- Vision, 2019, pp. 5492–5501. ment module for model-agnostic domain generalization, Advances in Neural Information Processing