=Paper=
{{Paper
|id=None
|storemode=property
|title=Explorations
in Media Visualization (invited talk)
|pdfUrl=https://ceur-ws.org/Vol-1210/datawiz2014_11.pdf
|volume=Vol-1210
|dblpUrl=https://dblp.org/rec/conf/ht/Reyes-Garcia14
}}
==Explorations
in Media Visualization (invited talk)==
https://ceur-ws.org/Vol-1210/datawiz2014_11.pdf
Explorations in Media Visualization
Everardo Reyes-García
University of Paris 13
99, av. Jean-Baptiste Clément
93430 Villetaneuse, France
everardo.reyes-garcia@univ-paris13.fr
ABSTRACT corpus of complex data which includes visual media such as
In this contribution we explore an emergent approach to data photographs, films, or any other digital images, broadly speaking.
visualization called ‘media visualization’. The main characteristic Of course, images are often used in visualization projects. The
of this practice is to take into account the content of visual media best examples are infographics and geographical maps, whose
directly as a constituent part of the data visualization project. main role is often to contextualize and decorate statistical data.
Media visualization employs and develops image processing Yet it is not common to find a project that analyses and represents
techniques. It contributes to current efforts on the design of data visual features of images themselves.
visualization such as diagrammatical representations, spatial In this contribution we focus on media visualization as an
distribution of elements, combination of colors, or animated emergent approach to take into account visual media as
behaviors. In this paper we describe ‘media visualization’: constituent part of a visualization project. After describing its
principles, requirements and related work. We also show some primary goals and techniques, we will present some examples
examples of media visualization developed by us within the developed by us in order to reflect on our own experiences and to
framework of visual analytics and media art. identify future work.
Categories and Subject Descriptors 2. MEDIA VISUALIZATION
H.5.2 [Information Interfaces and Presentation]: User ‘Media visualization’ is an idea originated in 2005 and currently
Interfaces – prototyping, screen design, interaction styles. developed by the Software Studies Initiative [12]. It refers to the
practice of analyzing visual media through visual media. In other
General Terms words, it consists of making visualizations including the images
Design, Experimentation, Human Factors. being analyzed. In contrast to common data visualizations, where
data is most of the time depicted as symbols and organized in
diagrams, media visualization takes advantage of visual analytics
Keywords and image processing techniques to construct visual spaces of the
Media visualization, visual representation, visual analytics. information analyzed.
In general, a project on media visualization involves two domains:
1. INTRODUCTION digital image processing and information design. The first domain
Research and development in data visualization (here understood is useful to extract and measure visual features from a collection
as un umbrella term for associated notions such as information of images, while the second domain concentrates on the visual
design, visual representation, or even hypermedia models) have representation of the collection of images. A project on media
gained popularity and acceptance for depicting discreet data in visualization assists research on cultural analysis through the
graphical form. Today, we see how some graphical models that identification of patterns by means of visual analytics [3].
once were restricted to particular domains become common and
distributed. Models such as network visualizations (force-directed Images must be understood technically and plastically, and not
graphs, among others), treemaps, and streamgraphs are more and exclusively from the figurative standpoint. Digital images are
more present in diverse professional domains (newspapers, mass series of pixels with chromatic values arranged in a bi-
media, etc.) dimensional matrix. The visible content of the image could be
regarded from two perspectives: figurative or non-figurative (also
Within this diversified context, the kind of data that is visualized known as plastic). Figuratively, the accent is on recognizing
deals most of the time to social records, transactions, preferences, characters, objects, places, etc. On the contrary, the plastic strand
hours, locations, connections, etc. There is also a considerable considers images as fundamentally chromatic values, forms, and
amount of valuable tools and resources to produce data shapes. These three properties constitute the visual features of the
visualization, ranging from scripting libraries (d3.js, sigma.js, image.
etc.) and software applications (Tableau, Gephi, etc). However,
the same cannot be said when we try to analyze and organize a For us, visual features define the realm of materiality and
objectivity of images. These features can be seized and quantified.
In computer science, visual features are the operational units
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
inside image processing procedures such as: analysis, extraction,
not made or distributed for profit or commercial advantage and that classification, retrieval, visualization, and representation [8].
copies bear this notice and the full citation on the first page. To copy
In the case of colors, the process of extracting and measuring
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee. visual properties implies storing in the database different values
HT’14, September 1-4, 2014, Santiago, Chile. that represent chromatic information. We can use for example the
Copyright 2014 ACM 1-58113-000-0/00/0010 …$15.00. HSB color model as the basis for measuring images. In one
column we can have its median hue value, in the second column mosaic. We call a ‘slice’ a thin part of an image, a region that
its median saturation, and in the third column its median slices it all along its X or Y axis. A slice does not show or
brightness. Of course, just as we decided to calculate the median summarizes the entire image, but only a delimited region. The size
value, we could also calculate the average, the mean, the standard of the slice (how thin of thick it is) can be parameterized. For
deviation and other statistical measures. large collections of images, it seems thinner slices are the best
option in order to depict variations and transformations of the
In the case of forms and shapes, the associated data considers the entire corpus of analysis. The visual patterns then are observed by
visible area, particles, fragments, contours, distribution, and differences and variations in the regions generated.
dimensions, among others [8]. Other measures could concentrate
on features such as block differences and variations; entropy; 2.5 Image Plotting
Sobel edge detection; Adaptive Color Quantization; statistics on Image plotting is based on common types of 2D plots that use dots
RGB channels, etc. and lines to represent data along the X and Y axis. An image plot
places, at the crossing coordinate of two values, the image
Besides visual features, each image can also be described
corresponding to those values. So, for example, we can decide to
semantically with metadata. The database can be enriched with
plot images by ‘year’ on the X axis, while the Y axis would
categories: year, designer, photographer, creator, software used,
determined by the median brightness value. In this case, we can
place, technique, etc.
observe variations and evolution in time over the two scales.
Once the database has been assembled, we are now in position to
look for modes of representation of images. As we said, the idea is 2.6 Related Work
to make evident patterns to approach cultural analysis. Current This brief review of emerging media visualization techniques
media visualization techniques require digital images as input data emphasized two of its underlying domains: image processing and
in order to output a new, different, and processed digital image. information design. Both domains have a history outside modern
So far, few techniques have started to delimit the practice of data visualization. For instance, image processing flourished in
media visualization. computer vision, computer graphics, and scientific visualization.
Media visualization takes advantage of tools and techniques from
2.1 Image Pixelation these developments to create its own procedures. Currently, one
Image pixelation consists basically on obtaining the colors of an of the main software environments to extract and measure visual
image and to represent them according to a discreet sequence of features is ImageJ, which is open source and well-known among
mask shapes. The mask shape is often a square (but could also be specialists of medical imaging [4]. Besides a series of scripts and
another geometrical figure such as circles or triangles) and its software on top of ImageJ, other tools are QtImageProcessing,
color is sampled from the original image and organized along its Mondrian (for statistical operations), scripts for MathLab, and
relative position to the image. The size of a unitary shape VisualSense.
determines the degree of pixelation. A bigger size of shape Regarding information design, we observe a close relationship
implies the summarization of more colors from the visual area between media visualization and contemporary art. In fact, some
where it gets its values. existing techniques can be approached from media art. Pixelation,
for example, is related to ‘pixel art’, as introduced by Goldberg
2.2 Image Averaging and Flegal in 1982 to describe the new kind of images being
Image averaging consists on stacking a series of images on top of
produced with Toolbox, a Smalltalk-80 drawing system designed
each other at the same spatial coordinates. It implies that all
for interactive image creation and editing [2]. Image averaging is
images are present in the same visual space, but in order to
related to the work of Sirovich and Kirby on ‘Eigenfaces’ in 1987
observe visual patterns it is necessary to perform a statistical
[11], and more recently, to Jason Salavon, who has produced a
measure of visual features, otherwise only the last image of the
series of images by averaging 100 photos of special moments
series would be visible. A single procedure for image averaging
[10]. For image mosaics, Brendan Dawes presented ‘Cinema
would be to reduce the opacity of each image by n-times its
Redux’ in 2004, a project aimed at showing what he calls a visual
percentage. Another technique would be to output an image where
fingerprint of an entire movie [1]. His main idea was to
each pixel depicts the calculated measure in all the series of
decompose an entire film into frames and then to arrange them as
images.
rows and columns. And image slicing can also be seen as a
2.3 Image Mosaic remediation of slit-scan photography. Among other prominent
Image mosaic, also known as image montaging, consists on slit-scan photographers, William Larson produced, from 1967 to
ordering the corpus of images one after another in a sequential 1970, a series of experiments on photography called ‘figures in
manner. Such as texts and grids, images are arranged in lines and motion’. The trick was to mount a thin slit in front of the camera
columns. The ordering rule could be obtained from measures of lens to avoid the pass of light into the film. Thus the image is only
visual features (for instance going from the brightest to the a part of an ordinary 35mm photograph.
darkest), from metadata (for instance by year) or by order of To conclude this section, we think ‘media visualizations’ have
appearance in the sequence (from the first to the last frame). The been focused so far on visual media: photographs, comics,
resulting image montage shows a rhythm of variations and magazine covers, album covers, film photograms, etc. But we
transformations. In many cases it seems visual patterns are clearer know there are other types of media which are not visual, or not
when there is no space between columns and lines (i.e. images are only visual. There is still work to do on audio, gestures,
only divided by their own size) and when all the images of the performance, tissue, garments, objects, furniture, industrial
corpus have the same dimensions. design, architecture, virtual worlds, and hybrid and multimodal
media. Among other issues, there is more research to be done in
2.4 Image Slicing analyzing and representing sound as sound (sonorisation rather
Image slicing also presents the corpus of images one after another than visualization) and objects as objects. In any case, it is
but there is a fundamental difference in comparison to an image important to remember that media in digital form implies the
transformation of another media form. An image of a painting or motion structure for the web or to physically print it, however
an album cover is a representation of the physical object; and an both techniques require destructive 3D model processing, i.e.
image of a digital image is its encoding, reproduction, reducing geometry by simplification, decimation or resampling.
compression, modification, and rendering. For technical details, a motion structure exported from ImageJ has
an average of 500,000 vertices and more than 1 million faces,
3. EXPLORATIONS IN MEDIA which is a very large amount compared to an optimized 2000-face
model for the web, loaded with the library three.js.
VISUALIZATION
In this section we present our work on media visualization. The The current constraints of the exploration of motion structures in
following projects have been developed mainly as research and web-based environments and as printed objects can be seen as a
experimental practice; like tools for reflection. While putting in similar path to the evolution of the representation of movement.
practice existing techniques and methods for cultural analysis, we Pioneers such as Etienne Jules Marey and Eadweard Muybridge
try to explore new forms of representation and interaction. One of first represented movement with pictures and images themselves,
our strategies has been the exploration of the aesthetics of digital but later Frank Gilbreth abstracted the traces of movement and
information through visual disruptions, that is, by reconfiguring created diagrams made out of lines. At that moment, artists got
the expected functional mode of visual representations [6]. inspiration from both types of representation with the intention to
explore a vocabulary of symbols, myths, and psychic processes.
For the following examples we concentrate on information design
and presentation formats. The first example is more related to the So experimental projects on media visualization contribute to the
exploration of shapes, and the second to the exploration of colors. design of data visualization in two manners. First, through the
The presentation format is studied as a constraint of designing the abstraction of shapes, traces and patterns, it permits discovering
resulting media visualization. We know early projects in media diagrammatical representations, spatial distributions of elements,
visualization were static, in the form of a single high-resolution and combinations of shapes. Second, through the inclusion of
image, which are useful for print and exhibitions. Likewise, first images and the design of exploratory and immersive experiences,
interactive explorations of image collections were done in large it provides insights for investigating animated behaviors,
tiled computer displays (such as the 287-megapixel HIPerSpace at combinations of colors, mix of media, and graphical indicators to
Calit2). But if the presentation format is web-based, we must face improve the comprehension of patterns in non-figurative
the challenge of smaller screens and the speed of network productions.
connection. Similarly, if the presentation format is a 3D shape, the
challenge is on rendering and interacting with 3D models for the
web or even on printing them for analog and manual analysis.
Presentation formats have their own conventions for explaining
visualizations. For media visualizations, we often see texts, lines,
arrows and other indicators that assist the identification and
labeling of patterns. In a conference poster, for example, the
designer can manually layout elements and design symbols and
diagrams to improve comprehension. In a video narrative, titles
and sound facilitate making sense of patterns. In a web-based
context, recent projects combine different views and information
processing techniques such as filtering, searching, and sorting.
3.1 Motion Structures
‘Motion structures’ is an ongoing project initiated in 2011 [7].
The idea is to convert an animated video sequence into a 3D Figure 1. Motion structure from Bill Viola’s Intimate Work
digital model with the intention of revealing patterns of time and
shape. The mode of interaction with a 3D object allows for 3.2 Web-based Media Visualizations
different ways of digital exploration: orbiting around, zooming in Our last example is an exploration in web-based media
and out, and immersive views inside the model. visualization. We developed ‘RockViz’ with the intention to
produce web-based image mosaics and image plots [9].
Our process puts special attention on shapes above other visual
features. To create a motion structure we use a script we wrote for For this project we gathered data about the most significant Rock
ImageJ. Basically, the operations require decomposing an albums according to AllMusic.com. Data was obtained Rovi, the
animated video sequence into a series of separated image files, data service behind AllMusic. The total amount of records was
which are then manipulated as an image stack. Then, several 1994, ranging from Blues to Alternative Rock and Heavy Metal.
image processing techniques occur under the hood: converting The metadata collected was about the artist/group name, album
images to 8-bit format, subtracting background, and rendering the title, release date, and album cover image. Contrary to ‘motion
stack as 3D shape. structures’, for RockViz the principal visual feature to explore
was the chromatic values.
With motion structures we intend to represent the spatial and
temporal transformations of a moving image sequence. The The first step for our media visualization was to download all the
obtained 3D shape encodes the changes of the objects in a frame: images and make them available locally, so we could measure
the different positions, the movement traces, and spatial and their chromatic features. We used ImageMeasure, a script for
temporal relations. The way in which we can interact with an Image, to measure hue, saturation, and brightness values. Then,
object is not limited to ImageJ. The model can be exported and we used Open Refine to handle data, but more importantly to
later manipulated in other 3D software applications such as Maya, apply mathematical formulae to the measure of images and
Sculptris, or MeshLab. Furthermore, it is also possible to export a dynamically calculate their Cartesian position.
The image mosaic was ordered according to, first, median of hue;
second, median saturation, and third, median brightness. To
facilitate the exploration of the dataset, we added a filter engine
that acts upon years, artist name, and album title. Finally, to make
a little faster the loading of images, we produced two versions of
each image: one is scaled to 100 x 100 px. and the other to 500 x
500 px. The small version is used for visual representations and
the larger appears when the user clicks on an image, so she can
observe more details of a single cover. Of course, a deeper study
should consider larger dimensions of images but this was the
largest resolution provided through AllMusic.
For image plots, we calculated spatial positions according to
measures of visual features. We decided to use Open Refine to
dynamically generate the HTML for each image because of two
reasons. First, Open Refine supports algebraic and trigonometric
operations so we could restrain the visual area to fit a resolution of
1024 x 768 px. Second, we originally used JQuery and the
function getJSON to communicate with a JSON database, but the
loading time is very slow for more than a few hundreds of images.
While an image plot requires translations of scale, for instance
years into maximum width in pixels (in our case 1024 px), we
also experimented with different representations inspired by Figure 3. Experimental interactive web-based plot of images
geometric figures. We used the main formula for Cartesian-Polar
transformations. Our first exploration, Figure 2, draws images
around polar coordinates, taking values from median hue and
median saturation, resulting into a chromatic circle-like
4. FUTURE WORK
visualization. Figure 3 disrupt this formula to investigate how So far, the primary goal behind our media visualizations has been
images could be plotted according to different figures. to practice and experiment with existing techniques. The
interpretation of data and the explanation of visual cultural
Web-based media visualizations contribute to information design patterns have been put aside momentarily, but this is precisely one
in recalling the need to adapt large amounts of information to of the clues for our future work. We expect to use our
small screens. Moreover, it raises questions on making efficient visualizations as teaching resource but also to collaborate closely
time-consuming operations for transferring data files. But with historians, filmmakers, media artists, musicologists and other
considering the web as presentation support also demands to domain experts.
reconsider the value of early developments by the hypermedia
community and their potential implementation with contemporary On the other hand, we believe there is still much to do at the level
web technologies. We are thinking specially in the xanalogical of interactivity. Research on hypermedia functionalities needs to
model, where visualizations of transclusions are depicted in a 3D be done for web-based visualizations. In the same line, models of
environment [5]. representation also require to be tested and experienced. While
text-and-number-based visualizations meet an explosion of
models, diagrams, demos, libraries, etc., some of them simply are
not suited for visual media. We believe there are two main
domains where we can get valuable insights: media art and
scientific imagery. For the former, artists often challenge our tools
and our common viewing experience; they practice could be
regarded as very innovative. For the latter, we must remember that
digital images are not exclusive to design, arts and art history, a
wide range of different disciplines use them as well: geography,
astronomy, medical imaging, mathematics, physics, chemistry,
biology, etc.
5. CONCLUSION
In this paper we have made a brief review of the emergent
approach of media visualization: its main principles and
requirements. We identified this approach mainly at the coupling
of image processing techniques and information design. Today we
can list a short gallery of media visualization techniques and
projects that start settling guides and practices. In order to enrich
current research and development on media visualization we
observed that two domains are particularly interesting: on the one
hand, the heritage of hypermedia functionalities, systems,
Figure 2. Experimental interactive web-based plot of images abstractions, and models for web-based projects. On the second
hand, experimental media art and software from other disciplines
equally related to images (others than media studies and art http://softwarestudies.com/cultural_analytics/cultural_analyti
history, for example sciences). cs_2008.doc
In the last part of our contribution we presented two explorations [4] National Institutes of Health. 2014. ImageJ: image
on media visualization. First, ‘motion structures’ an experiment processing and analysis in Java. Online:
on transforming an animated video sequence into a 3D digital http://imagej.nih.gov/ij/
model with the intention of revealing patterns of time and shape. [5] Project Xanadu. Online: http://www.xanadu.com/
Second, ‘RockViz’ a web-based media visualization comprising
almost 2000 rock album cover images and its visualization [6] Reyes, Everardo. 2012. Disrupting 3D Models in
through experimental image plots. Proceedings of the 3rd. Computer Art Congress. Paris:
Europia.
Further work should be conducted in the design visual indicators
to improve the comprehension of media visualizations, which are [7] Reyes, E. 2013. Motion structures. Online:
often non-figurative and difficult to seize. At the same time, the http://ereyes.net/ms/
non-figurative character of resulting processes can be seen as a [8] Reyes, E. 2013. “On Visual Features and Artistic Digital
move towards the symbolism of abstractions. By abstracting Images” in Proceedings of the ACM conference VRIC’13.
shapes, traces and patterns, new models emerge and can be Laval, France.
applied to other domains.
[9] Reyes, E. 2014. RockViz: visualizing the 2k. most significant
rock album covers. Online: http://ereyes.net/rockViz/
6. REFERENCES
[1] Dawes, B. 2004. Cinema Redux. Online: [10] Salavon, J. 2004. 100 Special Moments. Online:
http://brendandawes.com/projects/cinemaredux http://salavon.com/work/SpecialMoments/
[2] Goldberg, and Flegal, R. 1982. “Pixel Art” in [11] Sirovich, L. & M. Kirby. 1987. "Low-dimensional procedure
Communications of the ACM. Vol. 25. No. 12. December. for the characterization of human faces" in Journal of the
New York: ACM Press. Optical Society of America. Vol. 4. No. 3, pp. 519–524.
[3] Manovich, L. 2008. Cultural analytics: analysis and [12] Software Studies Initiative. 2014. Online:
visualization of large cultural data sets. White Paper. http://lab.softwarestudies.com/
Software Studies Initiative. Online: