The File Forgery Detection tasks is in its first edition, in 2019. This year, it is composed by three subtasks: a) Forged file discovery, b) Stego image discovery and c) Secret message discovery. The data set contained 6,400 images and pdf files, divided into 3 sets. There were 61 participants and the majority of them participated in all the subtasks. This highlights the major concern the scientific community shows for security issues and the importance of each subtask. Submissions varied from a) 8, b) 31 and c) 14 submissions for each subtask, respectively. Although the datasets were small, most of the participants used deep learning techniques, especially in subtasks 2 & 3. The results obtained in subtask 3 -which was the most difficult one- showed that there is room for improvement, as more advanced techniques are needed to achieve better results. Deep learning techniques adopted by many researchers is a preamble in that direction, and proved that they may provide a promising steganalysis tool to a digital forensics examiner.
The File Forgery Detection tasks described in this paper are part of the ImageCLEF
benchmarking campaign [
The File Forgery Detection has started in 2019 as a new task. It is an important and serious issue concerning digital forensics examiners. Fraud or counterfeits are common causes for altering files. Another example is a child predator who hides porn images by altering the image extension and in some cases by changing the image signature. Many proposals have been made to solve this problem and the most promising ones concentrate on the image content. It is also common that someone who wants to hide information in plain sight without being perceived might use steganography. Steganography is the practice of concealing a file, message, image, or video within another file, message, image, or video. Among them, images are the most usual cover medium for hiding data. Thus, the File Forgery Detection is composed by three different subtasks, namely: • Forged File Discovery • Stego Image Discovery • Secret Message Discovery
This paper presents an overview of the ImageCLEF2019 File Forgery Detection subtasks: the own subtask descriptions are in Section 2, the dataset in Section 3, and an explanation of the evaluation framework in Section 4. The participant approaches are described in Section 5, followed by a discussion and the conclusions in Sections 6. 2
The specific objective of these tasks are first to examine if an image has been forged,
and then, if it could hide a text message. Last objective is to retrieve the potentially
hidden message from the forged steganography images. Subtask 1 focuses on file
forgery. A file can be considered forged whether it has an altered extension or signature
(also known as magic bytes). If a file has an altered extension or signature, it is rather
simple to identify it. The problem relies in the case when both a file’s extension and
signature have been altered at the same time. In this case, even the most used digital
forensic software cannot identify a file as forged. Subtask 2 concerns the discovery of
stego images. Images are the most widespread cover mediums for steganographic
content. Steganography concerns the hiding of information into a cover medium which is
in plain sight, while steganalysis (our main objective in this subtask) tries to detect its
existence (subtask 2) and ideally retrieve the hidden message (subtask 3) [
The participant takes the role of a professional digital forensic examiner collaborating with the police, who suspects that there is an ongoing fraud in the Central Bank. After obtaining a court order, police gain access to a suspect’s computer in the bank with the purpose of looking for images proving the suspect guilty. However, police suspects that the suspect managed to change file extensions and signatures of some images, so that they look like PDF (Portable Document Format) files or other types. It is probable that the suspect has used steganography software to hide messages within the forged images that can reveal valuable information. The considered subtasks are defined as follows: • Subtask 1: perform detection of altered (forged) images (both extension and signature) and predict the actual type of the forged file. • •
Subtask 2: identify the altered images that hide steganographic content. Subtask 3: retrieve the hidden messages (text) from the forged steganographic images. 3
The data set consists of 6,400 forged images and pdfs, divided into 3 groups as shown in Table 1. Every group of images was used for a specific task. All participants had access to the training data sets along with their respective ground truth. The test sets were distributed without the ground truth.
Training set for forged file discovery (i.e. subtask 1) consisted of 2400 files: 1200 of them were true pdf files, whilethe rest seem to be pdf files, but they actually were images (equally distributed among jpg, png, and gif image types). Conversion to pdf files was made by changing their extension to pdf and their signature (the first four bytes) to 25 50 44 46. Training set for stego image discovery (i.e. task 2) consisted of 1000 images of jpg format: 500 of these images were clean, while the rest were stego (Figures 1,2).
Training set for secret message discovery (i.e. task 3) contained 1000 images of jpg format: 500 of them were clean, while the rest contained different text messages (although, the same one for every 100 images). A Least Significant Bit (LSB) insertion technique was used to insert text messages, concerning the presumed dialogue the suspect had with his abettor.
For assessing the performance, classic metrics were used: a) b)
Precision, Recall, and F-measure for Task 1 and Task 2.
Edit distance for Task 3.
In pattern recognition, information retrieval, and binary classification, Precision is the fraction of relevant instances among the retrieved instances. For the task 1, Precision could be defined as the fraction of actual detected altered images among all the images detected as altered:
Precision = nº of actual detected altered images
Total detections of altered images For the task 2, Precision could be defined as the fraction of actual detected images with hidden messages among all the detected images with hidden a message: Precision= º
ℎ ℎ ℎ ℎ of relevant instances. among all the altered images: Recall is the fraction of relevant instances that have been retrieved over the total amount For the task 1, Recall could be defined as the fraction of actual detected altered images Recall = º For the task 2, Recall could be defined as the fraction of actual detected images with hidden messages among all the images with hidden a message:
Recall = º F-measure is the harmonic mean of Precision and Recall, mathematically expressed as For the task3, the edit distance is adopted, which is defined as follows. Given two strings, a and b, on an alphabet Σ (e.g. the set of ASCII characters), the edit distance d(a,b) is the minimum-weight series of edit operations (Insertion, Deletion, Substitution) that transforms a into b.
This section shows the results achieved by the participants in the three subtasks. Table 1 contains the results of subtask 1, Table 2 contains the results of subtask 2, and Table 3 contains the results of subtask 3. 5.1
Results for subtask 1 Six runs were submitted by four groups to this subtask. Table 1 shows the details of the results, while Figure 1 summarizes the F-measure, Precision and Recall per run. The correspondences between run IDs and participant names are given in Table 1. Twenty six runs were submitted by six groups to this subtask. Table 2 shows the details of the results, while Figure 2 summarizes the F-measure, Precision and Recall per run. The correspondences between run IDs and participant names are given in Table 2. 0.8 0.6 0.4 0.2 0 Eleven runs were submitted by two groups to this subtask. Table 3 shows the details of the results, while Figure 3 summarizes the edit (Levenshtein) distance per run. The correspondences between run IDs and participant names are given in Table 3. runID 27447 26933 27162 27438 26904 26898 26896 26899 27446 27445 The security task was introduced in ImageCLEF 2019. The number of the registered teams/individuals and the submitted runs showed that the security challenges receive a significant attention and that they are interesting and challenging. Most participants signed to all three tasks, although this was not mandatory. This fact highlights the importance of each task. The majority of the approaches exploited and combined deep learning techniques, achieving very good results. The third task has been the most challenging one, in which the participants had to retrieve hidden messages from the images. The third task results have also shown that there is room for improvement, as more advanced techniques need to be used for better results. The analysis of the specific task results indicates that the training set was small for the specific problem, i.e., the extraction of the hidden messages. To leverage the power of advanced deep learning algorithms towards improving the state-of-the-art in steganalysis, we plan to increase the data set. We also plan to narrow down the application of the challenges, e.g., focus in steganalysis, probably in another domain.