Introduction

Generative Artificial Intelligence (Gen-AI) is a specialised field of Artificial Intelligence (AI) that deals with the generation of human-like texts, the creation of images from written descriptions and the production of videos based on predefined instructions [1]. Today, the potential of these Gen-AI tools is the subject of considerable debate on various key issues, ranging from the quality and authenticity of the content created to the ethical implications of their use [2,3,4,5]. However, the ability of Gen-AI to produce original materials in various forms has made a significant impact in various sectors such as creative industries, manufacturing, design, entertainment, and education [2,5,6].

Many researchers working for companies and academia have focused their efforts on developing efficient and accessible Gen-AI tools for content creation. Most notably, OpenAI's GPT series, which began with its first release in 2018 and was followed by GPT-2, GPT-3 and GPT-4, as well as its conversational variant, ChatGPT, have significantly impacted the landscape of text generation [1,7]. GPT is built on the principles of Large Language Models (LLMs) [8] which are designed to process and generate natural language text. The outstanding performances of LLMs, in synthesizing complex information, whether in the form of text or images, stems from the use of advanced techniques such as positional encoding and attention mechanisms [8]. Moreover, the main core of LLMs are complex neural network architectures like Transformers which represent the state-of-the-art for numerous natural language tasks [9,10].

Later, in 2021, OpenAI continued to push the boundaries of generative AI by releasing DALL⋅E, a tool capable of generating images based on textual descriptions [11]. While GPT focuses on generating coherent and contextually relevant texts based on input prompts, DALL⋅E integrates linguistic and visual information and extends this capability to visual content generation [11]. The tool employs the same Transformer architecture as GPT-3 [11]. Unlike traditional models that handle either text or images, DALL⋅E is a multimodal model [12,11], meaning that it can understand both types of data, integrating them in creative ways.

The achievements reached by GPT and DALL⋅E have significantly influenced the text-to-video domain, culminating in the remarkable capabilities showcased by OpenAI's Sora [13,14,15] realised in 2024. Sora is an AI model capable of creating realistic and imaginative scenes from text instructions [16,14]. Similarly to GPT and DALL⋅E, Sora can analyze text and understand intricate user directives. The process of generating videos is based on a diffusion Transformer architecture [9,10,17]. This process begins with a video resembling static noise and progressively refines it by removing the noise over many steps introducing details based on the provided text prompt [17]. Sora is notable for its capability to create up to 1-minute long videos, ensuring a strict adherence to user text instructions while delivering high visual quality and maintaining strong visual coherence, thus allowing users to provide visual contents from, even complex, text narratives [13,16]. The results of the productions made with Sora are highly realistic and applicable to a multitude of professional fields [15,14,18] using a prompt that can be as specific and detailed as the user wishes.

These outstanding results might become controversial. On the one hand, these Gen-AI tools offer tremendous benefits, such as enhancing creativity [19]. On the other hand, since a false perception of the world is possible by making indistinguishable what is real from what has been produced with Gen-AI tools [20,21], they also raise concerns about misinformation leading to the spread of fake news. Therefore, social awareness is necessary for the verification of exposed stimuli [22], as well as personal judgement in recognising AI-generated materials [23]. From an ethical point of view, the use of these models can lead to privacy violations, as they might inadvertently reveal sensitive information embedded in their training data. Furthermore, they can perpetuate and even amplify biases present in their training datasets, leading to unfair or discriminatory outcomes [24,25,26]. The deployment of Gen-AI thus demands careful consideration of these factors, emphasizing transparency, accountability, and the implementation of robust ethical guidelines to mitigate potential harms [27].

This study aims to evaluate the attraction, composition, and realism of Gen-AI videos compared to real videos. According to the manual written by Achi [28], multimedia data must adhere to some standards in terms of audiovisual recordings, such as lighting, colour, or scale, which are the most relevant attributes for visual realism [29]. As a case study, we focus on two landscape videos available on Sora's website. The first video is a recreation of Santorini (Greece), while the second one showcases the Amalfi Coast (Italy). Additionally, we consider two real videos reporting the same content. Through a detailed survey, the study measures attraction via parameters such as illumination, saturation, colourfulness, brightness, and sharpness. Composition is assessed by evaluating the video quality, the presence of shadows, focus, perspective, and shot range. Furthermore, the level of realism of the Sora videos is assessed by determining if the videos appear natural, contain fine details, and resemble drone footage, as well as whether respondents recognize the location and believe the video to be real. Finally, we identify which aspects of attraction and which compositional elements most significantly affect the perceived realism of the Gen-AI videos. For these reasons, we seek answers to the following research questions:

• R.Q.1. How do respondents perceive the attraction and composition of AI-generated videos compared to real videos depicting the same landscapes and environments? • R.Q.2. What are the key attraction and composition elements that influence the perceived realism of Gen-AI videos of landscapes?

The paper is organized as follows. Section 2 outlines the methodological design of the survey conducted. In Section 3, we analyze the obtained results. Finally, Section 4 discusses these results, offering valuable insights into the research questions and concluding with a discussion of the study's limitations.

Research methodology

The research objective is to evaluate the attraction, composition and realism of Gen-AI videos compared to real videos, and therefore a quasi-experiment will be conducted. For this reason, an ad hoc survey including either real or Gen-AI videos of landscapes, has been developed. Since the possibilities of producing videos using Gen-AI tools are still limited and the release of Sora is imminent, two landscape videos have been selected from those available on the Sora website. The methodological approach of this research is to make a first approach in the field of Generative AI in video generation and to study how it affects its use in university students. The first is a recreation of Santorini (Greece) and the second shows the Amalfi Coast (Italy). For the selection of the real videos, a search was carried out on Youtube, selecting those that are in the same location and have similar audiovisual characteristics [30,31]. All videos were customised to have the same duration and resolution. Figure 1 shows a frame for each video. In the validation process, the experts considered the videos to be similar in both similarity and format. The design of a quasi-experiment is based on the construction of two groups, a control group and an experimental group, to which a stimulus is exposed. [32] The quasi-experiment design is based on the collection of information by means of a self-administered online survey [33]. To ensure that the quasi-experiment design fits the research objective, it undergoes validation by expert judges (n=12). Experts in the field of computer science communication and artificial intelligence were selected for this purpose. They were provided with a guide explaining in detail the procedure and method of the experiment, as well as the questions in the questionnaire for collecting information. The purpose of this process is to ensure that there is a degree of agreement in terms of univocity and relevance. [34]. The validation includes the procedure of the quasi-experiment and the questionnaire administered.

The questionnaire (see Table 1) is designed as an information collection system based on the validated framework proposed in [29] for the human characterisation of visual realism in images. Since the questionnaire focuses on variables related to videos, the structure of the questions is modified accordingly, while the variables related to realism, attraction, and composition are maintained. The questionnaire is structured in two sections. The first section is designed to collect socio-demographic variables (P1-P5). The second section contains two videos to be evaluated independently in terms of realism, attraction and composition. In particular, the questions P6-P11 aim at evaluating the level of attraction, P12-P15 address the key elements of composition and, finally, P16-P20 focus on realism. The concepts covered by the survey were explained before the questionnaire was carried out in order to reduce bias in the interpretation of the questions.

Realism

As the chosen method is a quasi-experimental design, the survey is carried out on two groups. The survey for the control group contains two videos of real landscapes recorded by a professional as stimuli, while the survey for the experimental group contains two landscape videos produced by Gen-AI as stimuli. A non-probabilistic sample is selected as the results of the study aim to collect data to get more insights about the phenomenon of video production with Gen-AI tools. University students with a background in visual arts, theatre and music are therefore taking part in the study. Allocation to the individual groups was randomised and in proportion to each other. The data was collected in April 2024. Data was collected from n=62 participants, 28 from the control group and 34 from the experimental group. An online survey was used for data collection. All participants are young university students in the field of computer science, communication and new technologies.

Results

In this section, we report some statistics derived from the survey for both the control and experimental groups in order to seek for answers to the research questions R.Q.1 and R.Q.2 outilined in Section 1. The answers to the socio-demographic questions (Q1, Q2, Q3, Q4, Q5 in Table 1) show that the average age of the participants is 21 years, of which 68% are female and 27% are male. The 81% of respondents are not employed and 71% of them are Italian. Concerning the variables attraction, composition, and realism, the results are presented below in percentages, distinguishing between real videos and videos generated with Sora.

We first focus on the attraction variable by analyzing the answers to Q6-Q12. We found significant differences in terms of distribution between real and AI videos when evaluating the "illumination" (Q6). The bar plots are shown in Figure 2. Participants consider this item, in the case of the Sora's Santorini video, to be predominantly unnatural or slightly unnatural (91%) in the case of the Santorini video by Sora, while in the case of real Santorini video, the lighting is considered more natural and only 32% of respondents consider it slightly unnatural.

For the rest of the items, such as saturation (Q7), colour (Q8), brightness (Q9) or sharpness (Q10), the differences between the real videos and Sora are more neutral and less significant. In the case of saturation, for example, participants mostly considered the four videos to be quite saturated (50% for the real video of Santorini and 44% for the Sora's one). The same applies to colour, where participants mostly rated the videos as slightly colourful, regardless of whether it was a real or AI-generated video (57% in the case of the real Amalfi video; 50% in the case of the real Santorini video; 71% in the case of the AI Amalfi video and 47% in the case of the AI Santorini video). In terms of sharpness, the majority of participants felt that all videos were neither sharp nor blurred. Regarding the quality of the videos (Q11), most of them are categorised as of medium quality, with no significant differences between the distribution of the Sora's videos and the real ones. In summary, if we compare the Sora and real videos, in the case of Santorini there are major differences, especially in the lighting, and less in the saturation or colour elements. We point out that Sora's video of Santorini stands out as very colourful and saturated compared to the other stimuli. In the case of Amalfi, the differences of the distribution between Sora and real videos in terms of saturation, colour and lighting are small.

We now focus on the composition variable by analyzing the answers to Q13-Q16 which assess the degree of focusing, the perspective and the camera distance. Concerning audiovisual features such as the degree of focusing (Q13), the majority of participants consider real videos better focus if compared to Sora's ones. More precisely, the percentage of participants perceiving the videos well focused are: 57% for the real video of Amalfi; 64% for the real video of Santorini; 41% in the video of Sora of Amalfi and 47% in the video of Sora of Santorini. Another important compositional element that was analyzed is perspective (Q14). The barplots are shown in Figure 3. For the real videos, most of the respondents believe that the perspective appears natural. For the Sora videos, however, the range of answers is wider, which indicates that the perspective can be perceived as neither natural nor unnatural. Finally, concerning the camera distance (Q16) all the videos are perceived with a moderately or very distant view.

By analyzing the answers to the questions about the realism variable (Q16, Q17, Q18, Q19, Q20), it turns out that the majority of participants recognise Sora's video about Santorini as false (35%) or probably false (32%). In the case of the AI generated video of Amalfi, participants predominantly (32%) recognise it as probably true. As with the real videos shown in the control group, the majority of participants recognise the Santorini video as probably true (43%). For the Amalfi video, 36% of participants think it is probably false and 32% think it is probably true (Q20). The results are shown in Figure 4. For both the real video of Amalfi and its AI-generated counterpart, participants mostly do not recognize the location. In contrast, participants do recognise the location of Santorini for the most part in both experimental groups (Q16). The results are shown in Figure 5.

Discussion and Conclusions

The results of the experiment are based on the assumption that Gen-AI tools for video generation, in particular Sora, are capable of generating realistic content that is practically impossible to differentiate from a real video [35].

In order to answer to R.Q.1. "How do respondents perceive the attraction and composition of AI-generated videos compared to real videos depicting the same landscapes and environments?", in the Section 3 we reported the results obtained by analyzing some items of the attraction and composition variables. The results reveal distinct perceptions of attraction and composition between AI-generated and real videos, thus highlighting both technological limitations and areas of potential improvement in AI video generation. Concerning the attraction variable, we observed the lighting in AI-generated videos, particularly Sora's Santorini video, was largely deemed unnatural, in contrast with the more natural lighting perceived in real videos. Despite this, other attributes such as saturation, colour, brightness, and sharpness showed less pronounced differences, indicating that AI-generated videos can achieve a comparable aesthetic quality in these areas. These results suggest that Sora can replicate certain visual aspects, but it can struggles with replicating natural lighting, which is a critical component of realism [29]. Concerning the composition, the main differences between AI-generated videos and real videos were observed regarding the perspective item. The perspective in AI videos was also perceived less consistently, often seen as neither entirely natural nor unnatural.

We now try to seek answers to R.Q.2. "What are the key attraction and composition elements that influence the perceived realism of Gen-AI videos of landscapes?". It is evident that attraction variable influences the perceived realism of Sora videos. The results indicate that participants could easily distinguish the AI-generated Santorini video due to its unnatural illumination, despite slight differences in saturation and colour. This highlights that elements such as illumination, saturation, and colour are key factors in recognizing an AI-generated video. Conversely, when these elements are closely matched between real and AI-generated videos, as seen with the Amalfi videos, it becomes more challenging for participants to identify the AI-generated content. In these cases, the majority of participants did not recognize the Sora-generated videos as artificial, suggesting that a high degree of similarity in these compositional and attraction elements can enhance the perceived realism of AI-generated videos. The key compositional elements influencing the perceived realism of Gen-AI videos of landscapes include the perspective. Specifically, the natural appearance of perspective in real videos contrasts with the broader range of perceptions for AI-generated videos, where perspectives were often seen as neither natural nor unnatural. Overall, while AI-generated videos are making strides in matching the visual appeal of real videos, significant challenges remain in achieving complete realism, particularly in aspects like lighting and focus that contribute heavily to the perceived naturalness of a scene. These insights underscore the importance of further advancements in AI video synthesis to enhance the authenticity and visual coherence of generated content.

In terms of location recognition, Sora is able to generate highly realistic videos that resemble real locations, as demonstrated in the cases of Santorini and Amalfi [21,36]. Participants were more likely to recognize Santorini because iconic elements, such as the blue domes, were accurately recreated. In contrast, Amalfi lacks such iconic features, making it less recognizable. The era of Sora has just begun and this initial research on the realism of its videos clearly indicates that they are already difficult to distinguish from reality. However, this research suggest that for video generation to achieve results that closely mimic reality, factors such as attraction and composition must be considered must be considered to increase the level of realism. Additionally, there is a need to educate viewers on the potential for AI-generated videos. Consequently, regulations have been developed to control such content, and various studies on Sora emphasize the importance of addressing ethical risks related to misinformation [18,37]. This study faces several limitations. First of all, the accessibility of the Sora tool. At the time of the experiment, Sora was not available to the general public and the study had to rely on default videos provided by the tool, without the ability to explore its full capabilities through detailed instructions. Furthermore, the study exclusively utilized the Sora tool, as other text-to-video AI tools were deemed less effective in producing realistic results. Consequently, Sora is currently regarded as the most efficient tool for generating highly realistic videos using artificial intelligence. Since there is no research of this kind, this experiment is a first approximation to the work of creating videos with AI tools and the results are not generalisable to the rest of the population. To strengthen the findings of this research, replication of the experiment with a larger and more diverse sample of participants across various educational and professional backgrounds is necessary, as well as to compare and study other types of videos that are generated with the Sora tool.