he digital cinema workflow is extremely complex. At each of the numerous steps in the movie making chain, metadata are produced, stored and transmitted. Unfortunately at each step lots of those metadata are ignored at the input, dropped during a file format conversion or simply lost. When everything was analog, the process was slow enough that people were able to transmit the image related information by hand or by fax. But with digital cinematography, we are modifying images in real time and the old models become unsatisfactory. IP-Racine addresses the problem of real time transmission and use of metadata right at the beginning: within the camera.
A typical 'effects' movie is now shot with actors recorded in front of a green screen and the background is added digitally at a later stage. To synchronize both pictures positional data from the camera rig are transmitted to the computer creating the background. The best way to ensure coherency is obviously to included the positional metadata from the camera rig inside the image data stream as 'embedded metadata'. The enhanced video signal from the camera may be used subsequently by high quality virtual studio software to produce a composite image with the real actors and the synthetic background perfectly synchronized. The same signal can be used directly by a simpler preview computer to display real time images for the director of photography, giving the feedback necessary to assert the validity of the produced content. Obviously, this kind of feedback is a major improvement over traditional methods where dailies were available only the next day.
Work is supported by the European commission under the FP6-IST-511316 contract. J. Blat is with University Pompeu Fabra, Barcelona, Spain and is the coordinator of the IP-Racine project.
B. Michel is responsible for dissemination inside the IP-Racine project. (phone: +32 498 93 70 58; e-mail: benoit@michel.com).
W. Haas is with Joanneum Research Institute, Austria.
The camera rig is fitted with sensors and sends pan, tilt, zoom, lens focus, aperture, and Cartesian coordinates (optionally using GPS) through a serial link to the Grass Valley Viper camera. The camera inserts the positional metadata in the HD-SDI video stream connection. The image and metadata stream is then recorded on a DVS Clipster workstation. At the postproduction stage, various processes are achieved such as compositing the incoming images with synthetic background or props generated by the eStudio software from Brainstorm. The final images are produced off line but are of very high quality. At the same time, the HD-SDI video feed and the metadata are transmitted to a hardware chromakeyer inserting the images over a real time generated background similar to the final one. The resulting images are displayed on stage on a control monitor. Lower level of details and synthetic lighting conditions allow a real time rendering of the computer generated background matching the positional data of the camera. In order to perfectly synchronize the camera position and the virtual background, a small delay is inserted in the camera video stream. This delay is adjusted to the exact duration of the real-time background image generation.
In the digital cinema production workflow a number of heterogeneous types of metadata exist, many of them are represented in specific formats and standards. The scenario described here is a practical example of metadata transmission and use at the beginning of the digital cinema chain. IP-Racine is developing and testing transmission of the camera metadata and other metadata generated at later stages within the images themselves. One of the most commonly used im-IP-Racine metadata integration in the digital cinema workflow
Josep Blat, Benoit Michel, Werner Haas and the IP-Racine consortium T age formats is the single image DPX1 format whose origin can be tracked back to the pioneering Cineon film scanner from the early nineties. The DPX format allows the use of several metadata fields but the exact way of handling those fields vary between equipment manufacturers. An important prerequisite for an integrated metadata workflow as envisaged in IP-Racine is thus to establish interoperability between the different metadata formats that exist throughout the workflow. This includes different types of metadata from low-level (e.g. camera position) parameters to high-level textual descriptions.
Within IP-Racine, components for visualization, editing and conversion of metadata are being developed. The Workflow Metadata Organizer (WMO) tool developed by Joanneum Research serves as an interface to these components and can be used to visualize and modify the metadata that are available throughout the workflow.
The metadata formats to be considered are the headers of commonly used image formats, such as DPX, the elements of the SMPTE Metadata Dictionary (SMPTE RP-210), MPEG-7 [1] and MXF DMS-1 [2], as well as proprietary formats commonly used in production. While embedding of metadata into essence containers is useful during capture, metadata are stored separately in the postproduction content management system in order to allow for efficient search and conversion.
IP-Racine will issue recommendations to the industry and to standard bodies in order to promote metadata interoperability between the various postproduction steps. We hope that the above recommendations and the verification tool will be soon adopted by as many manufacturers as possible. An increase in efficiency in movie production and postproduction is expected. A report on standards for digital cinema metadata and their implementation is available on the IP-Racine project web site at http://www.ipracine.org/documents/documents.html.
Another IP-Racine document of interest is Digital Cinema Perspectives, a state-of-the-art book covering all aspects of production, postproduction, distribution and exhibition as seen by key people of the industry.
A number of different metadata are created during the digital cinema production process. Further descriptions are generated using automatic content analysis tools (shot boundary detection, camera motion estimation, re-detection of similar objects, key frame extraction). In order to visualize and modify the various metadata, which are often stored in a number of different formats, the WMO tool and its extensible set of plug-ins can be used to access the metadata descriptions. The software includes a video player which synchronously displays the metadata associated with the images.
In postproduction environments, users deal with a large amount of images, such as newly shot scenes, archive material and computer generated sequences. A large portion of the material, such as production rushes, is unedited and often redundant. Typically only few metadata annotations are available. To support users in navigating and organizing such content, IP-Racine develops a summarization and browsing tool. The tool uses the available metadata from the production as well as automatic content analysis to filter and cluster the content collection. The presentation of the media items and clusters is based on key frames extracted from the content. By selecting relevant clusters and changing the filter and sort criteria, the user refines a query which narrows the selection and locates relevant media items.
A high degree of transparency in data management is essential in the film production business. In IP-Racine DVS Digital Video Systems GmbH develops an application called Spy-cer™.