connotations of video material, and (2) annotations that are coarse-grained, i.e., focusing on keyframes and video fragments as opposed to full length videos. The main findings of the study are used to facilitate the automatic creation of narratives in the digital humanities exploratory search tool DIVE+1.
Social media provide a mainstream environment to produce,
share and comment on video material, which constitutes the
largest and still growing portion of Web content
Humanities scholars use AV archives
1http://diveplus.beeldengeluid.nl/ providing interactive access to multimedia objects enriched with events, people, locations and concepts.
Visualizing, mapping and constructing narratives play
a significant role in humanities research as they help to
contextualize historical material
The focus of this paper is to understand how niches
DIVE+ accommodates the digital hermeneutics approach by
means of proto-narratives, i.e., relations between events and
their participating entities. To support the creation of such
proto-narratives, we gathered events and links between their
participating entities in textual AV content (i.e., description)
through a hybrid machine-crowd pipeline
Nine international humanities master students (age between 21-25) enrolled in an interdisciplinary course about urban street visualization in Amsterdam participated in our niche study. Their task was to explore a dataset of archival AV material and to construct overarching micro-stories, in the shape of sequences of GIFs. A GIF is composed of three keyframes, or a (set of) short video fragment(s). The students were free to explore the dataset and to create GIFs about topics that drew their attention in relation to the city of Amsterdam, or in relation to the course literature.
The dataset consists of archival video material about Amsterdam, part of the Netherlands Institute for Sound and Vision2 (NISV) open collections. We retrieved 624 videos created between 1910-1989 on the NISV portal using the search keyword “Amsterdam”. The dataset consists of news broadcasts, varying in length from 50 seconds to 10 minutes, from which we identified three time periods, as shown in Table 1.
In the study we asked the students to choose a time period in Table 1 and to watch at least 20 videos from that period. The users had one week to complete the entire task, to log their activity3 and: (1) indicate the GIF type, i.e., keyframeor fragment-based; (2) describe each GIF, keyframe and video fragment with keywords; (3) provide the timestamps of the keyframes (keyframe-based GIFs) or the interval of the video fragment (fragment-based GIFs), among others. The students were also asked to prepare a short presentation to describe and motivate (1) the videos and the time period they selected, (2) the selection of keyframes and video fragments and (3) the story that is told in their GIFs.
We present the study results4 and analyze the data gathered from the participating users by focusing on keyframes, video fragments, GIFs and finally, the overarching micro-stories.
The users picked a time period as shown in Table 1. Their choice was informed by either: (1) feeling unknowledgeable about that period or (2) curiosity about a period when their parents were their own current age. In total, 68 videos were used across all the micro-stories and seven videos were used in more than one micro-story. All the overlaps occurred for the users that chose period P3, which is explained by the low number of videos in P3 and the fact that the users were asked to watch at least 20 videos. On the average, each user used eight videos to generate a story, with a minimum of three and a maximum of 20 videos per story.
Each story was composed of around eight GIFs (stdev of five GIFs), with a minimum of four and a maximum of 20 GIFs. In total, 75 GIFs were generated: seven keyframebased GIFs and 68 fragment-based GIFs. Only two users generated keyframe-based GIFs, while all nine users generated fragment-based GIFs. The 68 fragment-based GIFs were generated by remixing and combining 89 video fragments, meaning that around 25% of the fragment-based GIFs were composed of more than one video fragment. On average, 10 video fragments (stdev of 10) were used in each micro-story, with a minimum of two and a maximum of 35 video fragments. Furthermore, eight GIFs were generated by remixing keyframes and video fragments from multiple videos (six keyframe-based and two fragment-based GIFs).
In general, mostly keyframes and fragments from the beginning of the videos were picked (55.45%), followed by keyframes and fragments from the middle (24.55%) and then by keyframes and fragments from the end of the video (20%). When multiple keyframes and fragments from the same video were remixed in the same GIF, the order was always preserved, i.e., the keyframes and the fragments were used in chronological order with respect to the video stream. However, when looking at the entire story, we observe that the users break the natural temporal and linear sequence of videos by starting the story with video fragments or keyframes from the middle or the end part of the videos.
The majority of the GIFs are shorter than six seconds, with only a few longer than 10 seconds. The average length of a story is 43 seconds, with a maximum length of one minute and 48 seconds and a minimum length of 12 seconds. On average, only 3.6% of the videos length was used to generate each story, but, the length of the story is not always proportional with the total length of the videos.
The users focused their micro-stories around themes that were either inspired by the content of the videos, or by the course literature (i.e., visualization of urban spaces). The themes of the stories are: (1) mobility across the city, (2) citizens co-constructing urban spaces, (3) gender relations and (4) how urban routines relate to feelings of alienation in a globalized world. Some users created literal narratives, depicting aeroplanes, trains and bicycles to indicate mobility, while others worked on an abstract level by, for example, juxtaposing fragments of a person in a deep-sea diving suit with shots of a newspaper article lamenting loneliness in the city, to create a story about alienation.
Users reported that creating sequences of GIFs enabled
them to develop more elaborate stories. However, moving
from GIF to GIF does not denote a sequential development
in time, but it is used to zoom out spatially, or to create a
jarring contrast between GIFs and thus, a more abstract story
for example, moving from a GIF about riots in the street, to
a deserted, ruined square in the city, to children repainting
a building, to create a story about urban decay and ideals.
Similarly, the story about gender relation creates a
counterpoint between women undergoing beauty procedures, while
men, in a separate GIF, seemingly loom over them.
The users were asked to provide keywords, tags, for their
GIFs, selected keyframes and video fragments. These tags
represent the users’ interpretation of the multimedia content
comprising their narratives and do not necessarily describe
the content, but act as an interpretation medium for the story.
To determine the type of keywords, we manually evaluated
them using the Panofsky-Shatford model
We classified 207 (168 unique) tags that describe the GIFs and 262 (159 unique) tags that describe the keyframes and fragments composing the GIFs. The majority of the keywords are general, followed by specific and then by abstract keywords. When looking at the facets, we observe that more than 60% of the keywords belong to the what facet. The smallest number of keywords belongs to the when facet, with around 1% in all cases. While the keywords describing the who and where facets are evenly distributed among the keywords describing the GIFs, the amount of keywords describing the keyframes and fragments belonging to the where facet is much greater than the amount of keywords describing the who facet. While at the abstract and general levels a significant amount of keywords belong to the what facet, at the specific level, the users provided more keywords belonging to the where facet, and less for the what facet, showing that users tend to provide specific locations.
In storytelling, people can refer to concepts,
perspectives, opinions that are not physically present in the video,
but are referred to or expressed. As research
The overlap between the visual and the keywords provided by the users is quite low: 33% with the keywords describing stills and fragments and 49% with the keywords describing the GIFs. At the level of general concepts, the tags provided by the scholars overlap in proportion of 99% with the visual tags. This suggests that for (micro)narrative creation, what is visualized - generally - steers the narrative contained in the story. The overlap between user and professional keywords is even lower, 26% for keyframes and fragments and 30% for GIFs. In contrast to the visual tags, the professional tags do contain specific tags which usually refer to places, the where facet. For the facet distribution at the general level, the proportion of overlapping what facets is higher at the level of sequences but lower at the level of the
GIFs when compared to the visual tags. The professionalsuser gap is clearly defined at the level of abstract concepts.
The nichesourcing study aimed to bring insight into storytelling in digital humanities by exploring the interaction and interpretation of micro-narratives remixed using archival AV content. Overall, users tend to generate GIFs by remixing material positioned in the first part of the videos, disregarding the GIF position in the final produced micro-story. The temporal aspect is even more disrupted when users start their narrative with GIFs that contain keyframes and fragments from the middle and the end part of the videos, or when they finish their story with GIFs containing keyframes and fragments from the beginning of videos. Therefore, the original temporal sequence of the video is not relevant when remixing video footage for creative storytelling.
Users ascribe similar interpretations and meanings to their micro-narratives to those contained in visual tags, while they tag the chosen sequences more in terms of their function as a narrative building-block. Although at the GIF level users ascribe similar meaning to the video material as the professionals, they engage in scholarly interpretation on the keyframe level. Thus, the interpretation of meaning in storytelling is, to some extent, developed serendipitously and as a user- and context-centric development, driven by humanities research interests. Time seems - as our facet analysis emphasizes - less important than the where or what facets. Hence, people find events and objects the most relevant when building narratives. General keywords referring to events, objects, places and people almost entirely overlap with visual tags. Thus, the understanding of visual aspects, especially event and concept-centric, is needed to steer the story line.
In summary, humanities scholars need rich enrichments of AV datasets to facilitate the creation of narratives. However, storytelling through video remixing is a creative process that can not rely only on visual aspects. Deep semantic enrichment is needed to cover both implicit and explicit video concepts and perspectives. For exploratory-centric tools such as DIVE+ it is crucial to: (1) provide easy access to already extracted keyframes and video fragments as opposed to expecting the user to watch full videos; (2) provide deep semantic enrichment of keyframes and video fragments focusing on specific and general actors or people, locations, time periods, objects and most importantly events. Events play a central role in narrative development. Since event centrality is already a main aspect of DIVE+, we will focus on also integrating crowd-driven keyframes and video fragments semantics to offer users direct access to relevant information. DIVE+ users should be able to access smaller video granularity of interest and their enrichments, as opposed to watching the entire video and inspecting general video metadata.
The research for this paper was made possible by the CLARIAH-CORE (www.clariah.nl) project financed by NWO and by the Netherlands Institute for Sound and Vision and NWO under project nr. CI-14-25. [Highfield and Leaver 2016] Highfield, T., and Leaver, T. 2016. Instagrammatics and digital methods: studying visual social media, from selfies and gifs to memes and emoji. Communication Research and Practice 2(1):47–62. [Kemman et al. 2013] Kemman, M.; Scagliola, S.; de Jong, F.; and Ordelman, R. 2013. Talking with scholars: Developing a research environment for oral history collections. In International Conference on Theory and Practice of Digital Libraries, 197–201. Springer. [Maccatrozzo et al. 2013] Maccatrozzo, V.;
Van Hage, W. R.; et al. 2013. Crowdsourced evaluation of semantic patterns for recommendations.