Informational videos serve as a crucial source of conceptual and procedural knowledge for many people. While it is important to make informational videos instructive and engaging, editing such videos (e.g., trimming, overlaying text/image, etc.) can be dificult and time-consuming. Especially for novice video editors, who often struggle with expressing and executing their editing ideas. We present ExpressEdit, a system that facilitates editing informational videos via natural language text and sketching directly on the video frame by interpreting multimodal editing commands and suggesting applicable edits. Powered by a multimodal technical pipeline, the system interprets (1) temporal, (2) spatial, and (3) operational references in an NL editing command and spatial references from sketching. This work ofers insights into building multimodal interfaces for video editing.
CEUR ceur-ws.org
Informational videos are videos that introduce, explain, or demonstrate conceptual or procedural knowledge [1, 2, 3]. They encompass a broad range of topics such as cooking, health, programming, and craft, and can be produced in various formats (e.g., lecture, tutorial, q&a, demonstration, etc.) [4, 5, 6, 7]. They have become a popular source of knowledge for many people due to their rich and engaging content [5].
However, editing informational videos is a tedious task that involves carefully organizing the footage, removing unnecessary parts, and finding and incorporating additional media assets [4]. While popular commercial tools for video editing ofer all the necessary instruments to implement a variety of edits, for novices, these tools are dificult to learn and use, as they require great manual efort and have steep learning curves [ 8, 9].
We investigate how multimodality – natural language (NL) and sketching – can be leveraged in the informational video editing scenario. We conducted a formative study with 10 video editors with diverse levels of expertise and collected 176 expressions of video editing requests in nEvelop-O CEUR Workshop Proceedings the form of NL texts, sketches, and media assets. Focusing on edit expressions that initiate edits rather than revise or adjust applied edits, we found that editors feel comfortable expressing their general editing requests through NL text and use sketching on top of the frame to indicate specific locations or regions of interest.
Based on the findings from the formative study, we built ExpressEdit, a multimodal interactive
system for editing informational videos. It supports the expression of video editing requests
through NL text and sketching on top of a frame, and is powered by a computer vision and large
language model-based technical pipeline that comprehends and executes the edits by extracting
and interpreting three types of references from the multimodal command (Figure 1d,e): (
To learn about (
We recruited five novices who had edited at least 2 videos and five experienced editors who had edited at least 20 videos and 5 informational videos to cover a diverse range of editing commands since edit expressions such as attention to detail and vocabulary used can vary depending on the participant’s editing expertise. We chose five archived informational live streams as raw footage for the study as they are usually unedited and closely resemble a continuous stream of raw footage, which allows for tasks closer to real-world video editing settings. To allow participants to describe their edit commands in both text and sketch, we used Google Slides1, a popular slide authoring tool. We chose the tool because of its functionalities of adding text, images, and shapes, which could be used in expressing edit commands. Participants were also allowed to take a screenshot of a frame of the video and sketch over it. After the main task, we conducted a short semi-structured interview to learn about the participant’s experience performing the given task.
As a result, the collected 176 multimodal editing commands and the interview results revealed several patterns of multimodal video editing request expressions and the role of each modality provided within the tool. We found that the participants comfortably described video editing intents using various modalities: NL text, sketch, image, and graphics. In almost all the commands, the participants referenced moments (e.g., timestamps, visual content, verbal content) in the video where edit should be applied using NL text. Furthermore, they used both NL text and sketching on top of the frame to reference the spatial location of the edits within the frame. The participants used NL text to describe edit operations and their parameters. They mentioned the exact names of the operation (e.g., cut, text, image, etc.) or mentioned the main purpose or intended efect of the edit (e.g., highlight, emphasize, focus). For parameters of the edit operations, they gave either precise numbers or descriptions (e.g., large text, slow zoom). Lastly, participants frequently iterated on their commands to make them clearer by revisiting and redefining their expressions, for example, to make them more precise or to keep the consistency between several editing commands.
Based on formative study findings, we designed ExpressEdit, a multimodal video editing tool for editing informational videos. Our system interprets the user’s editing command in the form of NL&S and suggests a set of edits with temporal location (where in the video timeline), spatial location (where in the video frame), and edit operation & parameters (which edit and how). To better understand the generated edits and iterate on the NL&S command, users can examine the breakdown of the parsing results as well as the reasoning for the temporal & spatial location of each generated edit. Additionally, users can manually adjust the generated edits by the system or create their own edits.
(a) The examine panel (b) The edit result panel (c) The summary panel
To illustrate the envisioned user scenario of ExpressEdit, let’s follow Lia, a businesswoman and a YouTube creator who wants to edit her video about entrepreneurship. She recorded a talking-head video that she wants to make more concise, engaging, and informative using ExpressEdit.
To start editing the video, Lia uploads her recorded footage to ExpressEdit and comes up with the first edit she wants to implement. She presses the Add Tab button on the Tabs list and creates a new layer on top of the video where she can apply edits. Edits within a single layer will be of a single edit operation and cannot temporally intersect with each other.
Lia decides to add text captions whenever she mentions valuable advice or tips. In the Edit
description (Figure 1d), she types “whenever there is a mention of advice or a tip, put it as a
big white text with a transparent background on the bottom part of the frame”. Additionally, Lia
specifies the exact part of the frame where the text should appear using the Sketch function
(Figure 1e) and draws the bounding box on the bottom half of the frame. She presses Enter to
process the NL&S request and ExpressEdit provides (
To determine if the system understood her NL&S editing command, Lia examines the parsing results for the NL part. She looks at Parsing Results (Figure 2a) and sees that the parts of the NL command have colored backgrounds. To see the breakdown of the parsing results, she presses the Reference Breakdown and ExpressEdit shows Temporal (when) (i.e., “whenever there is a mention of advice or a tip”), Spatial (where) (i.e., “bottom part of the frame”), Edit (what) (i.e., “put it as a big white text”), and Parameters (how) (i.e., “big white text” and “transparent background”) references within the NL command, which assures her that the parsing was accurate.
To decide on generated edits to apply, Lia quickly glances over the generated edits. For each generated edit, she looks at the Reasoning (Temporal (when) and Spatial (where) aspects) and Preview (Figure 2b) to decide if she wants to apply the edit. Additionally, the system allows her to test the edit by quickly “turning it on and of” by toggling the radio button. After making all the decisions on generated edits, Lia can press the Get More Edits button to ask for more edits for the same command or manually adjust/add her own edits.
ExpressEdit supports seven edit operations: (
ExpressEdit is implemented as a Web-based React2 application. The backend server is based on Flask3, which hosted the videos along with their transcript and processed users’ requests. We obtained all the videos and transcripts from YouTube using the youtube-dl package 4. 2React 3Flask 4youtube-dl
We designed our pipeline to interpret natural language and sketch (NL&S) descriptions of an edit (Figure 3). In order to facilitate the real-time interactions, we perform pre-processing to extract frame-level and clip-level metadata from the video that ExpressEdit uses to reason about video context. The metadata consists of segmentations of frames (based on Segment Anything model [10]) for every 1-second of the video and textual description of each 10-second clip (based on InternVideo [11] and BLIP-2 [12]) that are then summarized using GPT-3.5 [13]. The overview of the pipeline can be found in Appendix A.
We first parse the NL request and divide the language command into (
We interpret the temporal references using GPT-4 by decomposing them into “positional”, “transcript-based”, and “video-based” references, compiling all segments of the video that match these references, and pass the candidate segments (i.e., edits) further along the pipeline.
To interpret the spatial references in NL we first categorize them based on their dependency on the visual content of the video using GPT-4. For visual content-dependent references such as NL references to a specific object in the video, we extract representative frames for each candidate edit and then obtain the candidate spatial locations based on the segmentation that has the highest cosine similarity with the given NL text reference and sketch (if provided) using CLIP [15]. If no visual content-dependent references are detected and no sketch is given, we designate the top-left corner of the frame as the candidate spatial location. Then, we refine and resize each candidate spatial location with the visual content-independent spatial references (e.g., left, top, etc.) using GPT-4.
Lastly, using GPT-4, we identify the edit operations that are most suitable for the request based on edit operation references and our system’s available edit operations: “text”, “image”, “shape”, “blur”, “cut”, “crop”, or “zoom”. Then, we identify the video editing parameters corresponding to each predicted edit operation for candidate edits based on identified edit parameter references. To get text and images that are more appropriate to the video for “text” and “image” operations, we provide video context to guide the GPT-4 generation process of text and search query, respectively.
We conducted a pilot user study with three novice video editors to (
During the pilot studies, we observed that novice video editors could intuitively use ExpressEdit to generate multiple edits based on their natural language and sketch (NL&S) command. When requesting new edits, the participants first confirmed that the NL&S editing command was understood correctly with examine panel (Figure 2a) that appears before any generated edits. Then, the edit result panel (Figure 2b) and ‘turning on and of’ feature allowed novices to quickly judge and add edits to the video. Furthermore, they were able to manually revise the generated edits by adjusting the edit operation and parameters. Additionally, the participants appreciated the convenience of applying edits with NL&S which efectively alleviated the manual eforts of making edits from scratch and allowed them to try out several edit ideas eficiently.
Since the pilot study lasted only 25 minutes, the participants made a few NL&S requests and spent the rest of the time applying the generated edits. Thus, we did not observe any notable use of the summary panel (Figure 2c), probably, due to its role of facilitating the management of many edits and the navigation between several NL&S requests. Additionally, because of the iterative nature of video editing, the participants had to constantly scroll through multiple panels whenever they wanted to turn on or of a specific generated edit. Since this behavior 5The video used for pilot study: How To SURVIVE As An Entrepreneur made them feel less eficient, they wished for more efective support for navigation between generated edits.
We propose ExpressEdit, a multimodal video editing system that allows users to edit videos
using natural language (NL) text and sketching on top of a video frame. The design of our
system is motivated by findings from the formative study and the analysis of 176 multimodal
expressions of edit requests. ExpressEdit can comprehend video editing commands and generate
edits based on a technical pipeline comprised of CV and large language models that extracts
and interprets (
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-01347,Video Interaction Technologies Using Object-Oriented Video Modeling). Machinery, New York, NY, USA, 2014, pp. 4017–4026. URL: https://dl.acm.org/doi/10.1145/ 2556288.2556986. doi:10.1145/2556288.2556986. [6] P. J. Guo, J. Kim, R. Rubin, How video production afects student engagement: an empirical study of MOOC videos, in: Proceedings of the first ACM conference on Learning @ scale conference, L@S ’14, Association for Computing Machinery, New York, NY, USA, 2014, pp. 41–50. URL: https://dl.acm.org/doi/10.1145/2556325.2566239. doi:10.1145/2556325. 2566239. [7] M. Bétrancourt, K. Benetos, Why and when does instructional video facilitate learning? A commentary to the special issue “developments and trends in learning with instructional video”, Computers in Human Behavior 89 (2018) 471–475. URL: https://www.sciencedirect. com/science/article/pii/S0747563218304102. doi:10.1016/j.chb.2018.08.035. [8] T. Jokela, K. Mäkelä, M. Karukka, Empirical observations on video editing in the mobile context, in: Proceedings of the 4th international conference on mobile technology, applications, and systems and the 1st international symposium on Computer human interaction in mobile technology, Mobility ’07, Association for Computing Machinery, New York, NY, USA, 2007, pp. 482–489. URL: https://dl.acm.org/doi/10.1145/1378063.1378140. doi:10.1145/1378063.1378140. [9] G. Chandler, Cut by cut: editing your film or video, Michael Wiese Productions, Studio
City, CA, 2004. [10] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, R. Girshick, Segment Anything, 2023. URL: http://arxiv.org/ abs/2304.02643. doi:10.48550/arXiv.2304.02643, arXiv:2304.02643 [cs]. [11] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, S. Xing, G. Chen, J. Pan, J. Yu, Y. Wang, L. Wang, Y. Qiao, InternVideo: General Video Foundation Models via Generative and Discriminative Learning, 2022. URL: https://arxiv.org/abs/2212. 03191v2. [12] J. Li, D. Li, S. Savarese, S. Hoi, BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, 2023. URL: http://arxiv.org/abs/2301. 12597. doi:10.48550/arXiv.2301.12597, arXiv:2301.12597 [cs]. [13] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language Models are Few-Shot Learners, 2020. URL: https://arxiv.org/abs/2005.14165v4. [14] OpenAI, GPT-4 Technical Report, 2023. URL: http://arxiv.org/abs/2303.08774. doi:10.
48550/arXiv.2303.08774, arXiv:2303.08774 [cs]. [15] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning Transferable Visual Models From Natural Language Supervision, 2021. URL: http://arxiv.org/abs/2103.00020. doi:10.48550/ arXiv.2103.00020, arXiv:2103.00020 [cs].