User generated content (UGC) is a valuable source for improving the coverage of events such as concerts, festivals or sports events. Integrating UGC in existing professional production workflows is particularly challenging in live productions. UGC needs to be checked for quality in this case, and metadata captured by the mobile device and extracted from the content are relevant for filtering the UGC streams that go into a live production system. We propose a system for capturing live audio and video streams on a mobile device, performing automatic metadata extraction in real-time and indexing the metadata for access by a production system. The system receives an audio, video and metadata stream from the mobile device, and creates additional metadata from the ingested audiovisual content. The metadata (e.g., location, quality) are then used to automatically select and rank streams, either selecting a stream to show to a viewer or a list of streams from which a human operator can select.
INTRODUCTION User generated content (UGC) is a valuable source for improving the coverage of events such as concerts, festivals or sports events. In order to integrate user generated content into existing production workflows, both the quality of UGC needs to be checked and metadata needs to be extracted. Such metadata, together with sensor information from the mobile device, will help the production team to assess the context, quality and relevance of the user contribution.
A particularly challenging scenario are live productions, where such metadata needs to be available with small latency. Live streaming of UGC from mobile devices has recently gained popularity, among others through the use of apps like Meerkat1 or Periscope2. However, these apps provide a stream “as is” for viewing on the web, without integration in production
CAPTURE AND ANALYSIS SYSTEM System Overview Figure 1 shows an overview of the proposed system. The system consists of a dedicated capture app, which sends video, audio and metadata as separate streams. This saves the muxing/demuxing effort and also facilitates distributed processing of different modalities on different machines in the cloud. All data are provided as RTP streams. The processing system (dashed box in the diagram) performs the necessary decoding and transformation for the content, and also includes a set of interconnected analysis modules. These modules may not only use the content as input, but may also use metadata from the device or from other modules. All extracted metadata are provided as streams again, and a logging module listens to these streams and indexes data in the metadata store. The audiovisual streams can be connected to viewers or to an editing system. A web application performs content selection Client app (capture, quality
analysis, sensor metadata) and displays the audiovisual data together with the extracted metadata.
We decided to build on an existing framework with many standard components which is able to handle the decoding of commonly used media formats. Thus, the GStreamer3 open source multimedia framework is used for this purpose. Content Capture The integrated capture application for Android enables users to perform quality analysis while capturing sensor data and streaming captured video. The main features are: (a) audio and video recording, via the built-in microphone and camera respectively, (b) metadata capturing from different sensors available on the device, (c) on-device analysis of captured essence to meet quality constrains, (d) en-/transcoding and packaging of recorded content and (e) the up-streaming functionality to servers for processing.
Raw video and audio data is captured through the camera and microphone of the device and encoded using Android’s MediaCodec API, while at the same time the quality of video frames is analysed. As encoding of video frames typically is more time consuming than for audio frames, a buffer synchronizes both streams. Once the encoding for a frame has finished, it is committed into the buffer and/or sent to an RTP packager. In parallel, a live on-device preview, containing visual qualityrelated notifications, as discussed in the upcoming sections, is presented. Synchronization is done by keeping track of the latest PTS for each stream.
During initialisation of the capture application, various types
of static metadata (such as properties and technical
parameters of the mobile device) are sent to the processing system.
Moreover, together with the content, sensor metadata of the
on-device sensors are captured to support real time quality
analysis, by recording the following sensors: location,
accelerometer, gyroscope, magnetic field, orientation, rotation,
ambient light, proximity and pressure. For example, the
accelerometer can be used to detect fast and shaky movements
of the mobile device. For the analysis of video frames, several
lightweight algorithms, which identify defects in the captured
3http://gstreamer.freedesktop.org
content, were implemented. Thus, contributing users who
have this app installed are capable of performing visual
quality analysis on the mobile device while capturing video and
obtaining direct feedback about the quality of the captured
content. The application continuously measures sharpness, noise,
luminance, exposure and detects the use of brightness
compensation before streaming captured video [
Analysis Framework
The analysis components are implemented as GStreamer
plugins and we create a flexible and powerful analysis chain by
combining the standard GStreamer modules and our
components. The GStreamer framework’s messaging concept assures
the optimal configuration for each plugin. For our purposes
we applied a simplified configuration manner to create
simple plugins with arbitrary number of input and output pads
having different formats though in some cases manual hints
are necessary. Since the standard RTP stream contains only
relative timestamps, the synchronization of audio and video
content from different devices is realized on the basis of the
timestamps of the RTCP stream. A custom plugin handles
the extraction of the timestamps and the difference calculation
between the internal clock and the absolute timestamp (e.g.,
synchronized with a PTP [
Analysis Modules The visual quality of a user generated video is a good indicator for an early decision whether the video might be useful to be considered, e.g. in a production, or whether it can conversely be sorted out due to an insufficient quality. In particular, the quality is an important decision criterion when having a huge amount of data available which should be reduced automatically. In order to obtain an overall quality measure of a user generated video, all available individual quality indicators are considered. The metadata received from the mobile device directly as well as the more complex quality measures obtained after transmitting to the server are fused as described in the following.
As mentioned earlier, the mobile device provides quality estimates of how blurry the content is, how much noise it contains, if there are parts suffering from over or under exposure and if the video was recorded under shaky conditions or not. On the server side, we may get additional measures by using a more complex algorithm for the blurriness and the contained noise. Furthermore, an estimate for macro-blocking artefacts is determined. At first we compute one representative value for each measure and combine them with all others by fusion. All these measures may be available or not, they are optional. The sampling steps for each measure are individual but constant over the whole duration of the video. Noise and blurriness can be measured on both the mobile device or on the server. Since on the server a more complex algorithm can be used, the results may differ a bit. Depending on the use case and the computational complexity that can be afforded, a subset of measures is computed and used for the overall quality measure. Under real-time requirements, we rather use the blur and noise estimation from the mobile device. Independent of the source of the blur and noise measure the computation for the overall quality measure stays the same.
All sampled values are collected individually for each measure and are sorted in increasing order. An appropriate subset of the sorted list is chosen to compute the average value for each measure. It was empirically established that a well correlated measure emerges if the subset is chosen from the higher values (high values indicate lower quality in this case), causing bad quality frames to have a higher influence on the result than good quality frames (so bad quality frames are over weighted compared to good quality frames). Thus, a video where only parts appear as very blurry or have a high level of noise will be rated as being of poor quality. The quality is represented by a floating point number in the range of 0 to 1, where 0 indicates excellent quality and 1 corresponds with very poor quality. This representation is used for each individual measure as well as for the overall quality value.
We have chosen to use the upper 25% of quality scores (i.e., representing the 25% segments with worst quality) to compute an average value for all measures of the involved quality metric. Finally, those values have to be combined. Simply averaging the individual measures is not a good strategy, since having one or two bad quality measures out of our set of five metrics would result in a non appropriate quality measure, distorted by the good quality measures.
The measure which causes the highest impact on the content quality should have the highest impact on the final quality measure. Thus, we apply a weighted sum where the highest values are disproportionately weighted higher.
Metadata Store The data exchange between the analysis platform and the production system is realised via a metadata store. This metadata store is a persistent hybrid repository accessible over a REST interface. Short term data are kept in a Redis4 in-memory data structure store whereas long term data are archived into a MySQL5 database. The repository type is transparent for the client, the difference is only noticeable in the query response time.
The extracted metadata are used for automatic content filtering of the UGC streams, e.g., discarding streams based on overall quality metadata or their location. By querying the metadata store with the appropriate criteria, the relevant streams can be selected for live editing.
CONTENT SELECTION When multiple concurrent live streams are available for an event, automatic and real-time selection of the best quality content is advocated. The selection strategies implemented so far are rule-based. They use the metadata available in the metadata store as input, i.e., the metadata captured on the device and the results from quality analysis on the mobile device and the server. The metadata does not only contain raw sensor and analysis data, but also the annotations of segments where pre-defined minimum quality limits have been violated.
Prior work on automatic video production in [
After filtering, ranking of the remaining streams is applied.
For content-based ranking, we use a strategy that is similar to
approaches that boost diversity in search results: (i) we prefer
streams showing a different area of the event over more of
the same, and (ii) from a group of similar ones we select the
one with the best quality. We use location information, where
available from the metadata of the stream and/or additionally
from determining the visual overlap between streams as
described in [
Results The content selection application is implemented as a web application, which implements the selection rules and also includes an HTML5 metadata viewer (see Figure 2). The metadata store is polled in defined intervals for recent data. The selection rules are executed and the UI is updated accordingly. As described in Section Analysis Modules, both quality annotations done at the mobile client and server are used. For each annotation type, a chart with the continuous quality measure is shown, and an additional event view displays segments that do not meet predefined quality standards. This level of detail is only shown for the currently selected video stream. For other concurrent streams, the overall quality metrics are additionally retrieved from the metadata store and visualised in a compact form. When switching to another stream, the views are switched accordingly. To provide audiovisual content to the HTML5 viewer, the incoming media stream is re-streamed by the analysis platform. This can be done as RTP stream with very low latency (requiring a browser plugin) or providing a stream for consumption by an HTML video player, with possibly higher latency.
CONCLUSION In this paper, we have presented a framework for automating content selection in order to complement professional coverage of live events such as concerts, festivals or sports events with user generated content. We have described a system for capturing live audio and video streams on a mobile device, performing automatic metadata extraction in real-time and indexing the metadata for access by a production system. The system creates additional metadata from the audiovisual content, and all available metadata are then used for automatic filtering and ranking of streams, using a rule-based approach. ACKNOWLEDGMENTS The authors would like to thank Jürgen Schmidt and Mario Sieck from Technicolor. The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement n 610370, ICoSOLE (“Immersive Coverage of Spatially Outspread Live Events”, http://www.icosole.eu).