-IP-Racine is an European commission funded research program addressing metadata compatibility within the digital cinema chain. Results of those efforts are visible at the shooting stage with optimized camera parameters transmission to on-set postproduction equipment. III. PRACTICAL IMPLEMENTATION The camera rig is fitted with sensors and sends pan, tilt, zoom, lens focus, aperture, and Cartesian coordinates (optionally using GPS) through a serial link to the Grass Valley Viper camera. The camera inserts the positional metadata in the HDSDI video stream connection. The image and metadata stream is then recorded on a DVS Clipster workstation. At the postproduction stage, various processes are achieved such as compositing the incoming images with synthetic background or props generated by the eStudio software from Brainstorm. The final images are produced off line but are of very high quality.
Teach of the numerous steps in the movie making chain, he digital cinema workflow is extremely complex. At metadata are produced, stored and transmitted. Unfortunately at each step lots of those metadata are ignored at the input, dropped during a file format conversion or simply lost. When everything was analog, the process was slow enough that people were able to transmit the image related information by hand or by fax. But with digital cinematography, we are modifying images in real time and the old models become unsatisfactory. IP-Racine addresses the problem of real time transmission and use of metadata right at the beginning: within the camera.
II. METADATA IS KEY TO REALTIME FEEDBACK IN VIRTUAL
STUDIO SCENES SHOOTING
A typical ‘effects’ movie is now shot with actors recorded in front of a green screen and the background is added digitally at a later stage. To synchronize both pictures positional data from the camera rig are transmitted to the computer creating the background. The best way to ensure coherency is obviously to included the positional metadata from the camera rig inside the image data stream as ‘embedded metadata’. The enhanced video signal from the camera may be used subsequently by high quality virtual studio software to produce a composite image with the real actors and the synthetic background perfectly synchronized. The same signal can be used directly by a simpler preview computer to display real time images for the director of photography, giving the feedback necessary to assert the validity of the produced content. Obviously, this kind of feedback is a major improvement over traditional methods where dailies were available only the next day.
Work is supported by the European commission under the FP6-IST-511316 contract.
At the same time, the HD-SDI video feed and the metadata are transmitted to a hardware chromakeyer inserting the images over a real time generated background similar to the final one. The resulting images are displayed on stage on a control monitor. Lower level of details and synthetic lighting conditions allow a real time rendering of the computer generated background matching the positional data of the camera. In order to perfectly synchronize the camera position and the virtual background, a small delay is inserted in the camera video stream. This delay is adjusted to the exact duration of the real-time background image generation.
In the digital cinema production workflow a number of heterogeneous types of metadata exist, many of them are represented in specific formats and standards. The scenario described here is a practical example of metadata transmission and use at the beginning of the digital cinema chain. IPRacine is developing and testing transmission of the camera metadata and other metadata generated at later stages within the images themselves. One of the most commonly used image formats is the single image DPX1 format whose origin can be tracked back to the pioneering Cineon film scanner from the early nineties. The DPX format allows the use of several metadata fields but the exact way of handling those fields vary between equipment manufacturers.
An important prerequisite for an integrated metadata workflow as envisaged in IP-Racine is thus to establish interoperability between the different metadata formats that exist throughout the workflow. This includes different types of metadata from low-level (e.g. camera position) parameters to high-level textual descriptions.
Within IP-Racine, components for visualization, editing and conversion of metadata are being developed. The Workflow Metadata Organizer (WMO) tool developed by Joanneum Research serves as an interface to these components and can be used to visualize and modify the metadata that are available throughout the workflow.
The metadata formats to be considered are the headers of commonly used image formats, such as DPX, the elements of the SMPTE Metadata Dictionary (SMPTE RP-210), MPEG-7 [1] and MXF DMS-1 [2], as well as proprietary formats commonly used in production. While embedding of metadata into essence containers is useful during capture, metadata are stored separately in the postproduction content management system in order to allow for efficient search and conversion.
IP-Racine will issue recommendations to the industry and to standard bodies in order to promote metadata interoperability between the various postproduction steps. We hope that the above recommendations and the verification tool will be soon adopted by as many manufacturers as possible. An increase in efficiency in movie production and postproduction is expected. A report on standards for digital cinema metadata and their implementation is available on the IP-Racine project web site at http://www.ipracine.org/documents/documents.html.
Another IP-Racine document of interest is Digital Cinema Perspectives, a state-of-the-art book covering all aspects of production, postproduction, distribution and exhibition as seen by key people of the industry.
1 DPX stands for Digital Picture eXchange, an SMPTE image format described in the “ANSI/SMPTE 268M-2003 Standard for File Format for Digital Moving-Picture Exchange” document.
A number of different metadata are created during the digital cinema production process. Further descriptions are generated using automatic content analysis tools (shot boundary detection, camera motion estimation, re-detection of similar objects, key frame extraction). In order to visualize and modify the various metadata, which are often stored in a number of different formats, the WMO tool and its extensible set of plug-ins can be used to access the metadata descriptions. The software includes a video player which synchronously displays the metadata associated with the images.
In postproduction environments, users deal with a large amount of images, such as newly shot scenes, archive material and computer generated sequences. A large portion of the material, such as production rushes, is unedited and often redundant. Typically only few metadata annotations are available. To support users in navigating and organizing such content, IP-Racine develops a summarization and browsing tool. The tool uses the available metadata from the production as well as automatic content analysis to filter and cluster the content collection. The presentation of the media items and clusters is based on key frames extracted from the content. By selecting relevant clusters and changing the filter and sort criteria, the user refines a query which narrows the selection and locates relevant media items.
A high degree of transparency in data management is essential in the film production business. In IP-Racine DVS Digital Video Systems GmbH develops an application called Spycer™.
Spycer collects production relevant metadata from media files automatically and provides this information to a search tool. Several Spycer™ applications build up a distributed content management network, called SpycerNet, for easy and fast file retrieval.
[1] Information Technology—Multimedia Content Description Interface.
ISO/IEC 15938:2001. [2] SMPTE, Material Exchange Format (MXF) Descriptive Metadata
Scheme - 1, SMPTE 380M-2004. [3] Digital Cinema Perspectives, September 2006, ISBN 2-87111-029-8, available at www.i6doc.com.