This study aims to develop a Text to Pictogram translation system which is used to convert a French text into its corresponding pictogram terms. The proposed system demonstrates the efectiveness of a transformer-based model in translating French text into meaningful pictogram sentences. Google-T5 is utilized and further fine-tuned on a custom dataset of French text to predict corresponding pictogram terms in French. The model underwent ifne-tuning across multiple epochs to optimize performance. Additionally, the trained model was iteratively ifne-tuned to enhance its translation capabilities. Metrics like PictoER score, BLEU score and METEOR score were used to assess the model's performance. The proposed model achived a PictoER score of 13.9, BLEU score of 74.3 and a METEOR score 87.0.
those with language impairments, thereby contributing to the advancement of computational methods in assistive technologies and enhancing the quality of life for individuals with aphasia.
Text-to-pictogram translation is a task which involves translation of natural language text into text
with words for which appropriate pictograms are available. A unified approach to transfer learning in
NLP tasks can be achieved by considering every text processing problem as a “text-to-text” problem,
i.e. taking text as input and producing new text as output [
A shallow linguistic analysis approach can be used to perform linguistic analysis for text to picto
conversion [
Encoder-decoder models perform better than other models on textual similarity tasks [
Google T5 or Text-to-Text Transfer Transformer is an encoder-decoder model which was pre-trained on a multi-task mixture of both unsupervised and supervised tasks. It is known to work well in tasks which require out of the box thinking. This task involves converting French text in various everyday contexts into words which are simpler and have a corresponding pictogram available. The main objective of this approach is to develop a Text-to-Picto translation system using the T5 model.
The dataset that has been used was built from the TCOF corpus, and is stored in JSON format. TCOF
contains interactions between adults, adults and children, and children themselves, covering a wide
range of topics including debates, everyday situations, and medical consultations. This type of text is
representative of the interactions we observe between caregivers (families, medical staf) and individuals
who rely on pictograms due to language impairments [
On further analysis of the dataset, it was found that the data contained 24270 lines of French text with appropriate target text. The average size of the source lines is 54.6 words and the average size of the target text is 53.8 pictogram words.
Tokenization is important for preparing the data for model training. Here, both the target pictogram sequence and the source French text are tokenized using the pre-trained tokenizer from the google-T5 model. The text is split into individual tokens and converted to a numerical representation. Padding and truncating limit the text sequences to have a maximum size of 256 tokens.
The T5 model can be adjusted for particular tasks and comes pre-trained on a large data corpus. Here, the model is adjusted to produce the simpler pictogram terms from oral transcriptions in French. The model is inherently trained in solving text-to-text tasks and hence proves to be eficient for this task. The "t5-base" variant of Google’s T5 model is utilised for this task. t5-base is a snapshot of the T5 model taken after it was trained with 220 million parameters. This makes it flexible and easy to train for the given French text data.
The self-attention mechanism enables the model to identify long-range links and dependencies in the input sequence. To be more precise, the T5 model calculates attention scores between every pair of tokens in the input sequence using self-attention layers. By assigning a diferent weight (attention score) to each token based on relevance to the next token, it makes every token in the sequence able to pay attention to every other token.
The T5 model is based on the traditional transformer architecture comprising of an encoder-decoder structure. The input sequence is processed by the encoder, which also outputs contextual embeddings. The output sequence is subsequently generated by the decoder using these embeddings.
The dataset is loaded from the file and the source text (src) and target sequence (tgt) are extracted for every element and stored as a list, following which it is tokenized. This is then converted to key-value pairs and fed as input to the model. TThe T5 model contains several training arguments, such as the batch size, which is set to 16 and indicates the number of training instances processed in each iteration, and the number of training epochs, which are set to 3, 5, and 6 (where the model is initially trained for 3 epochs and then further trained on the already trained model). Additionally, the save steps are set to 1,000, meaning that model checkpoints are saved every 1,000 steps to ensure training progress is recorded. The learning rate of the optimizer algorithm—such as Adam—is chosen in order to efectively update the model parameters in light of the training data. The goal of this optimization is to increase model performance by minimizing the loss function. The model is trained using the hyperparameters mentioned in Table 1
The sequence of pictogram terms generated by the proposed model is converted to the corresponding pictos sequence using the resources provided by the task organisers.
For example: If the input source text is: "il y a un moment donné elle nous avait dit essayez de pas dire de mots français pendant le truc.", the model generates the following sequence of pictogram terms: "il_y_a un instant donner passé elle nous dire essayer de dire non de mot français pendant le truc". Figure 2 shows the pictos sequence corresponding to the above generated sequence of pictogram terms.
Pandas is used in the project to manipulate data, including loading data from JSON files and structuring it into dataframes. The main deep learning framework, PyTorch, makes it easier to apply and train the T5 model, which creates pictogram sequences from French text. A cloud-based Jupyter notebook environment with GPU-accelerated resources for quicker model training and inference is ofered by Google Colab. Together, these tools improve productivity and eficiency by streamlining the development and experimentation process.
The parameters used for evaluating the model are the Picto-term Error Rate (PictoER) [
Comparing the outcomes from varying the epochs while training yields insightful observations. The error rate decreased from 18 to 17 when training for 5 epochs, suggesting some improvement in performance. When initially trained for 3 epochs and fine-tuned for an additional 3 epochs, the pictoer_score drops significantly from 17.5 to 13.9, suggesting improved generalization and performance on unseen data. This is also reflected in the BLEU score which measures the precision of n-grams and the METEOR score which focuses on word order, both of which show considerable improvement. This significant improvement underscores the efectiveness of fine-tuning in refining model parameters and enhancing its ability to capture underlying patterns in the data.
The results suggest that the model may not have been able to reach its maximum potential during the first training period. Rather, the model’s representations were gradually improved through the repeated training process, which improved generalization and decreases the error rate. These findings highlight the importance of iterative training strategies and the need for careful experimentation to achieve optimal results. This shows that one may continuously enhance the model’s performance and guarantee its flexibility to a variety of datasets and applications by iteratively fine-tuning it.
In conclusion, this research demonstrates the efectiveness of advanced transformer models, specifically the Google-T5, for the task of translating French text into pictogram sequences. Through iterative ifne-tuning, the model consistently improved in accuracy, demonstrating its ability to handle intricate aspects of language. This was evaluated using metrics like PictoER, BLEU, and METEOR scores. This research emphasizes how transformer-based techniques can improve accessibility and communication for people who use augmentative and alternative forms of communication. Researchers may enhance the model’s performance and guarantee its flexibility to a variety of datasets by iteratively fine-tuning it. In order to further advance the fields of assistive technology and natural language processing, future studies could explore expanding this strategy to other languages and improving the model’s adaptability to diverse linguistic contexts.