Introduction

Egocentric vision is a subfield of computer vision that analyses first-person viewpoint vision data captured by a wearable camera. Núñez-Marcos et al. [1] highlights that compared with thirdperson view (exocentric) videos, egocentric videos usually involve rich hand-object interactions. Our framework leverages the observation that different egocentric actions often involve the same objects (e.g., both "Taking cutting board" and "Cutting onion" involve a cutting board) and captures such correlations using pre-trained large language models.

Early work has demonstrated that, in addition to the RGB modality, leveraging multiple modalities such as audio, optical flow, and the bounding box and category of an object help improve a model's capability to understand egocentric videos [2,3,4]. Such efforts have explored the potential of multi-modal knowledge distillation, where the teacher and student models receive different input modalities [5,6,7,8]. Their results show that using the teacher's knowledge from certain modalities for training improves the student's performance on a different HAII5.0: Embracing Human-Aware AI in Industry 5.0, at ECAI 2024, 19 October 2024, Santiago de Compostela, Spain. * Corresponding author. Envelope yizheng.sun@manchester.ac.uk (Y. Sun) Orcid 0009-0004-2600-1236 (Y. Sun); 0000-0002-9923-4346 (H. Li); 0000-0003-3454-2468 (C. Lin); 0000-0001-6693-7531 (R. Batista-Navarro) modality during inference. It is, however, unrealistic to assume that multiple modalities are always available. In contrast, the language modality is usually available because most existing EAR datasets are annotated according to target actions expressed in natural language [9,10,11]. Additionally, the rapid growth and impressive performance of pre-trained Language Models (LMs) on natural language processing (NLP) and computer vision (CV) tasks have been notable [12,13,14,15]. Pre-trained LMs bring broader knowledge of human actions, that can support the language modality.

Extensive research has delved into exploring the potential of learning vision representations through supervision embedded in natural language [16,17,18,19]. Consequently, it is natural to investigate whether LMs can be employed for video action recognition. Siddharth et al. [20] utilised language models to generate textual descriptions of videos, enabling their vision model to comprehend and identify actions more effectively through textual cues. Sun et al. [21] jointly trained video and language modalities, enabling tasks like action recognition to benefit from textual context. While previous studies demonstrated the advantages of integrating the language modality into video learning, they typically fuse video and language modalities together instead of utilising a pre-trained language model's latent knowledge directly. Several considerations drive the advancement of leveraging pre-existing knowledge in modelling. Firstly, language models (LMs) have showcased exceptional capabilities in few-shot and zeroshot transfer learning [22]. Consequently, LMs can be employed effectively with relatively small datasets, as their objective is solely to assist the existing LMs during inference. Secondly, methods based on LMs for video need little or even no training. Through plug-in modules, they can be utilised in a convenient manner [23]. In this study, we take a different route and propose a cross-modal language-vision knowledge distillation framework for EAR.

Figure 1 depicts our framework. The conventional training approach employs one-hot encoding to represent target actions, treating "Taking cutting board" and "Cutting onion" as distinct target classes. Consequently, a vision model perceives these two videos as unrelated due to the lack of consideration for the correlation between the action classes in the one-hot encoding scheme. However, this perspective fails to reflect the inherent relationships within the video data, leading to a lack of generalisation. This is different from a human standpoint, as humans would recognise that both videos share relevant visual features associated with the cutting board object. Conversely, a language model perceives textual action labels such as "Taking cutting board" and "Cutting onion" as relevant, given their shared usage of the word "cut", which better aligns with the video content. To address this discrepancy, our framework leverages a language model as the teacher to capture and incorporate this contextual relevance information into the EAR training process to help improve vision models' general understanding of videos. Furthermore, our framework also follows a multi-task learning approach for capturing correlations between the vision and language representations. We demonstrate that utilising a pre-trained language model as teacher can improve a vision model's performance and generalisation capability on the EAR task.

Contributions (i)

We provide a cross-modal language-vision knowledge distillation framework for EAR. Our framework is highly flexible, and is not constrained in terms of the vision and language models involved. (ii) We demonstrate through experiments that a pre-trained In EAR data, samples include action labels described in natural language along with corresponding video clips. These video clips often exhibit relevant visual features corresponding to different action labels. However, traditional training methods commonly utilise one-hot encoding for action labels, which does not adequately capture this correlation and lacks generalisation. In contrast, our framework applies a language model on textual action labels to better understand the relationships among them, thereby aligning more closely with the inherent information in video data. language model's pre-existing knowledge is beneficial for a vision model's understanding of egocentric vision. (iii) Our experiments show that our framework's performance in terms of accuracy improves upon a baseline approach by up to 2.6%. This superior performance is achieved without adding any additional computation for inference.

Related Work

Natural language supervision for vision learning focusses on learning visual representations from semantic information contained in natural language. Various methods have been introduced to learn visual presentations from text paired with images [16,18,24,19]. Notably, a close work to ours is that of Gomez-Bigorda et al. [17], which projects given textual information into topic classes using Latent Dirichlet Allocation (LDA). They then use the probability distribution of topic classes as a supervisory signal to train a CNN with cross-entropy loss. In our case, we use pre-trained language models to generate the probability distribution and employ standard practice in knowledge distillation to train a transformer-based vision model. Furthermore, most of the aforementioned work are for pre-training visual representations, while our framework is directly applied to downstream tasks such as egocentric action recognition.

Multi-modal knowledge distillation. In the context of multi-modal knowledge distillation, several methods have been introduced in a cross-modal fashion [5,6], where a student and a teacher receive a different modality, respectively. Alternatively, some efforts explored the distillation of knowledge between more than two modalities [7,25,8,26], which have utilised vision and audio-based data such as raw RGB, optical flow and sound waves, etc. In contrast, we focus on knowledge distillation from a teacher model receiving language modality to a student model receiving RGB modality. Compared with vision and audio-based modalities, the strength of using language as a teaching modality comes from modern pre-trained language models, whose pre-existing knowledge contain strong generalisation and understanding capability.

Egocentric action recognition (EAR).

One line of work has focussed on model architecture design to model the interplay between spatial and temporal information within RGB video frames [27,28,29]. Concurrently, another strand of research demonstrated that using object bounding boxes and categories to model hand-object interaction significantly improves EAR performance [30,4]. Recent work showed that utilising multiple modalities demonstrates promising performance [2,3,8]. They utilised vision and audio-based modalities and have used a shared model architecture for different modalities. Notably, the language modality poses unique challenges due to its distinct data format, making direct application of existing methods impractical. Thus, we propose a novel framework aimed at harnessing the language modality specifically for EAR tasks.

Multi-task learning was originally introduced by Caruana [31], where a shared model generates output predictions for multiple tasks on the same input. Recent research highlighted the strong performance of multi-task learning in computer vision tasks [32,33,34]. In our study, we extend this concept to our knowledge distillation framework by incorporating a regression head. This head projects vision latent representations from a student onto pre-trained language latent representations provided by a teacher.

Methodology

This section provides a formal definition of the EAR task and delineates the procedural aspects of our framework, which we refer to as LanViKD. Figure 2 presents an overview of the architecture of LanViKD, which is comprised of two primary stages: Stage 1 entails the preparation of a language model designated as the teacher model, while Stage 2 involves performing cross-modal knowledge distillation.

Egocentric Action Recognition Formulation

Following Radevski et al. [8], we formally define the EAR task as follows. An RGB video clip is in the format of 𝑐 ∈ ℝ 𝑇 ×𝐶×𝑊 ×𝐻 , where 𝑇 is the number of sampled RGB frames, and 𝐶, 𝑊 and 𝐻 represent the number of channels, height and width. An egocentric action recognition dataset 𝔻 = {(𝑐 1 , 𝑤 1 , 𝑦 1 ), ..., (𝑐 𝑁 , 𝑤 𝑁 , 𝑦 𝑁 )} contains 𝑁 video clips 𝑐 𝑖 , together with textual narrations 𝑤 𝑖 describing actions in the clips, and one-hot encoding 𝑦 𝑖 ∈ ℝ 𝐶 + of the narrations. The goal of EAR is to predict ŷ 𝑖 ∈ ℝ + as the action class for a given video clip 𝑐 𝑖 , or alternatively, ( v , n ) ∈ ℝ 2 + as the verb and noun constituting the action in a video. The traditional training target for EAR is the one-hot encoding of actions expressed in text [35,29,36]. However, as shown in Figure 1 some action classes such "taking cutting board" and "cut carrot" share common features with respect to the "cutting board" object in their corresponding RGB video frames. One-hot encoding ignores the this relationship between different action classes. The goal of our work is to utilise this relationship information for EAR training by distilling the knowledge of a pre-trained language model into an RGB video model.

Language Teacher Model Preparation

As shown in Figure 2, given an EAR dataset 𝔻 = {(𝑐 1 , 𝑤 1 , 𝑦 1 ), ..., (𝑐 𝑁 , 𝑤 𝑁 , 𝑦 𝑁 )}, we employ a pre-trained language model capable of processing sequences of text tokens to generate latent representations. Subsequently, we freeze the parameters of the language model and proceed to train a linear projection layer (or two separate linear projections in scenarios involving verb-noun compositional actions) atop the language model. This trained projection layer is tasked with classifying a textual action description 𝑤 𝑖 into its corresponding one-hot encoding index 𝑦 𝑖 (or verb and noun indices, as previously specified). Following training, the linear projection facilitates the generation of a soft probability distribution across all action classes given a textual action description as input. This soft distribution contains valuable semantic information, differing from conventional one-hot encoding. For instance, consider the actions "taking cutting board", which is associated with the noun label "cutting board" encoded as 1, and "cut carrot", labelled with the noun "carrot" encoded as 2. When inputting "taking cutting board" into the language model for noun index classification, it assigns the highest probability to 1 while also allocating a considerable probability to 2. This is due to the shared term "cut" in both textual actions, despite their distinct noun classes. Moreover, this semantic relationship is echoed in the video data, wherein both actions involve the object "cutting board". While one-hot indices categorise these videos into separate, unrelated classes, the probability distribution reflects their semantic connection, aligning more closely with the visual modality.

1-15

Cross-modal Language-Vision knowledge distillation

Once the language teacher model is prepared, we opt for a vision model to serve as the student model, taking RGB video frames as its input. Similar to the teacher model, we apply linear projection(s) atop the student model. The parameters of the teacher model are then fixed, and we proceed with knowledge distillation, as originally proposed by Hinton et al. [37].

Training Objective. As described above, given a dataset 𝔻 = {(𝑐 1 , 𝑤 1 , 𝑦 1 ), ..., (

ℒ 𝐾 𝐿 = 1 𝑁 ∑ 𝑁 𝑖=1 ŷ 𝑡 𝑖 ⋅ (𝑙𝑜𝑔 ŷ 𝑡 𝑖 − 𝑙𝑜𝑔 ŷ 𝑠 𝑖 ).

Following standard practice [37,8], we use a temperature parameter 𝜏 to control the entropy of class probabilities predicted by the teacher ŷ 𝑡 𝑖 = 𝜎 ( ŷ 𝑡 𝑖 /𝜏 ) and the student ŷ 𝑠 𝑖 = 𝜎 ( ŷ 𝑠 𝑖 /𝜏 ), where 𝜎 is the softmax operator. We then scale the KL-divergence loss according to the temperature parameter ℒ 𝐾 𝐿 = ℒ 𝐾 𝐿 ⋅ 𝜏 2 . Additionally, we also minimise the standard cross-entropy objective of class probabilities predicted by the student

ℒ 𝐶𝐸 = 1 𝑁 ∑ 𝑁 𝑖=1 𝑦 𝑖 ⋅ 𝑙𝑜𝑔𝜎 ( ŷ 𝑠 𝑖 ).

In the case of compositional actions containing verbs and nouns, the training objective becomes the sum of corresponding loss terms with respect to the verb and the noun, where ℒ 𝐾 𝐿 = 1 2 (ℒ 𝑛 𝐾 𝐿 + ℒ 𝑣 𝐾 𝐿 ) and ℒ 𝐶𝐸 = 1 2 (ℒ 𝑛 𝐶𝐸 + ℒ 𝑣 𝐶𝐸 ). Furthermore, we apply a multi-task learning approach in LanViKD by adding an extra linear projection layer on top of the student model to generate 𝑜 𝑠 𝑖 . We take the output from the last hidden layer from the teacher ℎ 𝑡 𝑖 , which is the latent representation of the input text given by the original pre-trained language model. We minimise the smooth L1 objective ℒ 𝐿1 to regress 𝑜 𝑠 𝑖 towards ℎ 𝑡 𝑖 .

ℒ 𝐿1 = { 0.5(𝑜 𝑠 𝑖 − ℎ 𝑡 𝑖 ) 2 /𝛽 if |𝑜 𝑠 𝑖 − ℎ 𝑡 𝑖 | < 𝛽 |𝑜 𝑠

𝑖 − ℎ 𝑡 𝑖 | − 0.5 * 𝛽 otherwise Where 𝛽 determines the threshold for switching between L1 and L2 loss, with a value of 1 used in our experiments. We compute the final loss as ℒ = 𝜆 ⋅ ℒ 𝐾 𝐿 + (1 − 𝜆) ⋅ ℒ 𝐶𝐸 + 𝜇 ⋅ ℒ 𝐿1 . We note that the weights sum of ℒ 𝐾 𝐿 and ℒ 𝐶𝐸 is 1 because they are based on the same output linear layer. Instead, we use a separate loss weight for ℒ 𝐿1 because it is based on the linear layer of a separate task. During the inference process, it is important to note that the language teacher model is dispensable. The student vision model operates solely on RGB video frames as its input.

Experimental Setup

In our experiments, our primary objective is to assess the potential benefits of integrating knowledge from a language model into a vision model for the EAR task. Specifically, we ask the following questions: (i) What is the performance of utilising LanViKD on regular EAR data samples, i.e. training and testing samples containing overlapping environments and/or objects.

(ii)

To what extent can a student model, trained using LanViKD, effectively generalise to unseen environments and/or objects not encountered during training? (iii) How does the incorporation of a language model's teaching signal alongside the standard one-hot target affect the training of a student model, and what is the optimal balance between the two? (iiii) How does using the language modality compare to using the audio modality in cross-modal knowledge distillation with the RGB modality? We choose to compare language with audio because it is unlike optical flow and object bounding box/category which need to be computed using external algorithms or models for RGB data [38,39]; audio and language are both raw data sources that are readily available in EAR datasets.

To address questions (i) and (ii), we conduct experiments across various datasets, encompassing those with overlapping environments and objects for both training and validation, as well as those featuring unseen or under-represented elements during validation. For question (iii), we perform experiments with different 𝜆 settings, regulating the ratio of the language model's teaching signal to the traditional one-hot target within the training objective. To address question (iv), we compare our findings with those of Radevski et al. [8], who conducted similar knowledge distillation from audio modality to RGB video modality.

Datasets.

Our experiments are conducted on three datasets: Epic-Kitchens-100 [9], Something-Something V2 [10] and Something-Else [40].

Epic-Kitchens-100 (EK-100) is a large-scale dataset of egocentric videos. It contains 100 hours of non-scripted videos recorded by 37 participants in kitchen environments. The actions depicted in the videos include narrations in the form of English phrases. The training targets are verbs and nouns expressing the actions (e.g. "cutting onion" is an action narration, whose training targets are "cut" and "onion"). There are 300 unique noun classes and 97 unique verb classes. An action is considered to be correctly predicted if both the verb and the noun are correct.

The Something-Something V2 (SSV2) dataset is a large collection of (mostly egocentric) videos that show people performing 174 pre-defined basic actions with everyday objects (e.g. putting something on a surface, moving something up) [10]. Notably, videos in SSV2 initially feature annotations with specific object names, which are then replaced with the word "something" for training targets (e.g., "putting box on a surface" becomes "putting something on a surface").

Something-Else (SthElse) is an alternative data re-split of the original SSV2 [40]. SthElse splits SSV2 in such a way that the training and validation sets contain distinct objects. Therefore, SthElse focusses on using unseen objects during training to measure the generalisation capability of a model.

In a similar vein, we also incorporate the EK-100 Unseen and Tail split. The unseen split is a subset of the EK-100 validation set, which contains videos that are recorded by two participants who did not appear in the training set. The unseen split is specifically designed to measure the ability of models on unseen environments during training. The tail split is a subset containing action classes that have little training samples. Notably, the EK-100 regular split encompasses all samples excluding the unseen split.

Language Backbone. In this study, our language model of choice is MiniLM, featuring 12 layers and a hidden size of 384 [41]. The rationale behind choosing MiniLM stems from its compact architecture and computational efficiency. Despite its smaller size, MiniLM maintains competitive performance over its teacher model, UniLM [42]. For the EK-100 dataset, we utilised the original textual action annotations, consisting of English phrases describing actions, as input to MiniLM. Similarly, for the SthElse dataset, we employed the original annotations, which include object names as inputs to MiniLM. Vision Backbone. Following Radevski et al. [8], we chose the Swin Transformer Tiny version (Swin-T) model as the vision model in LanViKD. Each video clip is represented as a sequence of RGB frames, where each frame is represented by a 3 × 224 × 224 tensor. Swin-T takes a video clip as input and produces a 768-dimension tensor as the latent representation of the video.

Implementation details.

For teacher models, we train the linear head for 10 epochs across all datasets. As for the student models, we train them for 50 epochs on Epic-Kitchens, 40 epochs on SSV2 and 30 epochs on Something-Else. As per Radevski et al. [8], we employ the AdamW optimiser [43], setting the peak learning rate at 1𝑒 − 4. Initially, the learning rate linearly increases for the first 3 epochs and then linear decreases to 0. A weight decay of 5𝑒 − 2 is utilised, along with gradient clipping, limiting the maximum norm to 5. Across all experiments, 𝜏 remains fixed at a value of 3. For EK-100, during training, we select a random starting frame and sample 32 frames with a fixed stride of 2. In inference, frames are sampled in the same manner to cover the central section of the video. For SSV2 and SthElse, 16 frames are sampled to cover the entire video during both training and inference. Standard data augmentation techniques are applied to RGB frames, including random cropping, color jitter, and random horizontal flips (exclusive to EK-100). Consistency is maintained within each video clip by applying the same augmentation methods to every frame. A single temporal crop is employed for inference.

Direct Comparison. In the study by Radevski et al. [8], the Swin-T model was trained on the EK-100, SSV2 and SthElse datasets. A key distinction between their approach and ours is that while they incorporated multiple modalities, including RGB, optical flow, and audio, they did not include the language modality. In contrast, our work leverages only the language modality as the teacher modality. To ensure a direct and fair comparison, we adhered to the same experimental settings as Radevski et al. [8], including the use of the backbone model, data augmentation techniques, and frame sampling methods. Evaluation Metrics. We calculate two widely used metrics, Accuracy@1 (ACC@1) and Accuracy@5 (ACC@5), on the test set, which play pivotal roles in assessing the effectiveness of such systems [44]. By measuring the correctness of predictions within the top-ranked recommendations, both ACC@1 and ACC@5 provide valuable insights into the system's ability to deliver relevant and satisfactory outcomes to users, where ACC@1 quantifies the proportion of correct predictions among the top-1 ranked results. It signifies whether the single highestranked item recommended by the system aligns with the user's preference. On the other hand, ACC@5 expands the assessment to the top-5 ranked results, thereby offering a broader evaluation of the system's performance. Generalisation capability on unrepresented and unseen environments and unseen objects. Table 2 shows the performance on EK-100 unseen and tail splits, which contain unseen and unrepresented environments during training, respectively. It also shows the performance on SthElse, which contains videos involving objects that are unseen during training. These validation sets aim at evaluating a vision model's generalisation capability.

Our observations indicate that distilling knowledge from a language model into a vision model generally enhances the generalisation capability of the latter by up to 1.3% on the EK-100 unseen split and 2.6% on the SthElse dataset. Specifically, for the EK-100 unseen split, LanViKD outperforms the baseline across all three metrics (Noun, Verb, and Action) without the addition of the regression head. Furthermore, incorporating the regression head leads to an additional 1.3% improvement in performance specifically on the metrics for Action. For the EK-100 tail split, LanViKD demonstrates competitive results with the baseline when the regression head is absent. However, with the regression head, although LanViKD exhibits a slight performance decrease in the Verb metric compared to the baseline, it achieves a 0.5% enhancement in the primary metric, Action. Similarly, for the SthElse dataset, LanViKD surpasses the baseline by 2.6% in ACC@1 without the regression head. However, the addition of the regression head marginally diminishes performance by 0.4% compared to its absence. Moreover, Figure 3 shows per-class ACC@1 improvement in relation to the top 10 frequent nouns and verbs within the Table 3 The impact of varying teaching weights on EK-100. 𝜆 = 0.4 represents the weight assigned to KLdivergence loss. A higher value of 𝜆 indicates that the training objective for the student model prioritizes the teaching signals from the language model to a greater extent, while reducing emphasis on the one-hot targets.

Method Teaching Parameters SthElse

Acc@1 Acc@5 LanViKD 𝜆 = 0.

Table 5

Comparison of utilising audio modality and language modality on EK-100 dataset. The baseline is introduced by Radevski et al. [8], which distils knowledge from audio modality to RGB modality during training. In contrast, our approach distils knowledge from language modality to RGB modality during training. Both approaches use only RGB video frames for inference.

EK-100 unseen split dataset, alongside the top 20 frequent actions identified in SthElse.

Teacher's influence on the student. To investigate the influence of the teacher language model on the student model's performance, we set the parameter 𝜆 to 0.4 and 0.8 for the EK-100 and SthElse datasets, respectively. Specifically, this adjustment increases the teaching signal's weight in the training objective from 40% to 80%, while maintaining the regression head. Tables 3 and 4 present a comparative analysis of the model's performance with 𝜆 set at 0.4 and 0.8. The results indicate that increasing 𝜆 to 0.8 leads to a slight improvement on the unseen split of the EK-100 dataset. However, this increase is associated with a significant performance decline on the tail split of the EK-100 dataset and across the SthElse dataset.

Comparison with knowledge distillation on audio modality. We are interested in comparing the utilisation of audio modality for knowledge distillation, as opposed to optical flow (OF) and objects' bounding box and category (OBJ) modalities. Unlike OF and OBJ, which are derived from RGB modality through external algorithms or deep learning models [38,39], audio and text modalities represent raw data from the datasets. This distinction is crucial, as the computation of OF and OBJ may introduce hidden external model knowledge into training, making it uncertain whether all the knowledge distilled into a student is solely from the teacher. In the study by Radevski et al. [8], they trained an audio model on EK-100 audio data alongside an RGB model. Subsequently, they combined these models as a teacher ensemble to train a Swin-T vision student model, which only received RGB video frames. Similarly, in our approach, we leverage knowledge from a language teacher alongside RGB video frames to train a vision student, also receiving only RGB frames; while Radevski et al. [8] utilised audio and RGB modalities for training, we employ language and RGB modalities. Both approaches exclusively use the RGB modality for inference.

It is important to note that the audio modality is exclusive to the EK-100 dataset. Table 5 presents a comparison between knowledge distillation using audio and RGB, and language and RGB modalities. Our findings indicate that training with language and RGB yields superior performance, surpassing training with audio and RGB by up to 1.7% on the EK-100 regular split, while also achieving competitive results on the unseen split.

Conclusion and Future Work

In this work, we propose a knowledge distillation framework, LanViKD, for language and vision (RGB) modalities. Our experiments demonstrate enhancement in performance compared to the baseline model, which is solely trained on one-hot labels utilising only the RGB modality. Additionally, we conduct a comparative analysis between the incorporation of audio modality and language modality for knowledge distillation. Our findings indicate the superiority of the language modality as a teacher for enhancing the learning of the vision-based student.

In our future work, we will investigate the integration of the language modality with additional modalities such as audio, depth, and thermography. We plan to find an approach for aligning multiple modalities and create a comprehensive teacher model with broader knowledge for knowledge distillation, potentially leading to further performance improvement.