In this contribution we explore an emergent approach to data visualization called 'media visualization'. The main characteristic of this practice is to take into account the content of visual media directly as a constituent part of the data visualization project. Media visualization employs and develops image processing techniques. It contributes to current efforts on the design of data visualization such as diagrammatical representations, spatial distribution of elements, combination of colors, or animated behaviors. In this paper we describe 'media visualization': principles, requirements and related work. We also show some examples of media visualization developed by us within the framework of visual analytics and media art.
Research and development in data visualization (here understood as un umbrella term for associated notions such as information design, visual representation, or even hypermedia models) have gained popularity and acceptance for depicting discreet data in graphical form. Today, we see how some graphical models that once were restricted to particular domains become common and distributed. Models such as network visualizations (force-directed graphs, among others), treemaps, and streamgraphs are more and more present in diverse professional domains (newspapers, mass media, etc.) Within this diversified context, the kind of data that is visualized deals most of the time to social records, transactions, preferences, hours, locations, connections, etc. There is also a considerable amount of valuable tools and resources to produce data visualization, ranging from scripting libraries (d3.js, sigma.js, etc.) and software applications (Tableau, Gephi, etc). However, the same cannot be said when we try to analyze and organize a Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
HT’14, September 1-4, 2014, Santiago, Chile.
Copyright 2014 ACM 1-58113-000-0/00/0010 …$15.00. corpus of complex data which includes visual media such as photographs, films, or any other digital images, broadly speaking. Of course, images are often used in visualization projects. The best examples are infographics and geographical maps, whose main role is often to contextualize and decorate statistical data. Yet it is not common to find a project that analyses and represents visual features of images themselves.
In this contribution we focus on media visualization as an
emergent approach to take into account visual media as
constituent part of a visualization project. After describing its
primary goals and techniques, we will present some examples
developed by us in order to reflect on our own experiences and to
identify future work.
2. MEDIA VISUALIZATION
‘Media visualization’ is an idea originated in 2005 and currently
developed by the Software Studies Initiative [
In general, a project on media visualization involves two domains:
digital image processing and information design. The first domain
is useful to extract and measure visual features from a collection
of images, while the second domain concentrates on the visual
representation of the collection of images. A project on media
visualization assists research on cultural analysis through the
identification of patterns by means of visual analytics [
For us, visual features define the realm of materiality and
objectivity of images. These features can be seized and quantified.
In computer science, visual features are the operational units
inside image processing procedures such as: analysis, extraction,
classification, retrieval, visualization, and representation [
In the case of forms and shapes, the associated data considers the
visible area, particles, fragments, contours, distribution, and
dimensions, among others [
Besides visual features, each image can also be described semantically with metadata. The database can be enriched with categories: year, designer, photographer, creator, software used, place, technique, etc.
Once the database has been assembled, we are now in position to look for modes of representation of images. As we said, the idea is to make evident patterns to approach cultural analysis. Current media visualization techniques require digital images as input data in order to output a new, different, and processed digital image. So far, few techniques have started to delimit the practice of media visualization.
Image pixelation consists basically on obtaining the colors of an image and to represent them according to a discreet sequence of mask shapes. The mask shape is often a square (but could also be another geometrical figure such as circles or triangles) and its color is sampled from the original image and organized along its relative position to the image. The size of a unitary shape determines the degree of pixelation. A bigger size of shape implies the summarization of more colors from the visual area where it gets its values.
Image averaging consists on stacking a series of images on top of each other at the same spatial coordinates. It implies that all images are present in the same visual space, but in order to observe visual patterns it is necessary to perform a statistical measure of visual features, otherwise only the last image of the series would be visible. A single procedure for image averaging would be to reduce the opacity of each image by n-times its percentage. Another technique would be to output an image where each pixel depicts the calculated measure in all the series of images.
Image mosaic, also known as image montaging, consists on ordering the corpus of images one after another in a sequential manner. Such as texts and grids, images are arranged in lines and columns. The ordering rule could be obtained from measures of visual features (for instance going from the brightest to the darkest), from metadata (for instance by year) or by order of appearance in the sequence (from the first to the last frame). The resulting image montage shows a rhythm of variations and transformations. In many cases it seems visual patterns are clearer when there is no space between columns and lines (i.e. images are only divided by their own size) and when all the images of the corpus have the same dimensions.
Image slicing also presents the corpus of images one after another but there is a fundamental difference in comparison to an image mosaic. We call a ‘slice’ a thin part of an image, a region that slices it all along its X or Y axis. A slice does not show or summarizes the entire image, but only a delimited region. The size of the slice (how thin of thick it is) can be parameterized. For large collections of images, it seems thinner slices are the best option in order to depict variations and transformations of the entire corpus of analysis. The visual patterns then are observed by differences and variations in the regions generated.
Image plotting is based on common types of 2D plots that use dots and lines to represent data along the X and Y axis. An image plot places, at the crossing coordinate of two values, the image corresponding to those values. So, for example, we can decide to plot images by ‘year’ on the X axis, while the Y axis would determined by the median brightness value. In this case, we can observe variations and evolution in time over the two scales.
This brief review of emerging media visualization techniques
emphasized two of its underlying domains: image processing and
information design. Both domains have a history outside modern
data visualization. For instance, image processing flourished in
computer vision, computer graphics, and scientific visualization.
Media visualization takes advantage of tools and techniques from
these developments to create its own procedures. Currently, one
of the main software environments to extract and measure visual
features is ImageJ, which is open source and well-known among
specialists of medical imaging [
Regarding information design, we observe a close relationship
between media visualization and contemporary art. In fact, some
existing techniques can be approached from media art. Pixelation,
for example, is related to ‘pixel art’, as introduced by Goldberg
and Flegal in 1982 to describe the new kind of images being
produced with Toolbox, a Smalltalk-80 drawing system designed
for interactive image creation and editing [
To conclude this section, we think ‘media visualizations’ have been focused so far on visual media: photographs, comics, magazine covers, album covers, film photograms, etc. But we know there are other types of media which are not visual, or not only visual. There is still work to do on audio, gestures, performance, tissue, garments, objects, furniture, industrial design, architecture, virtual worlds, and hybrid and multimodal media. Among other issues, there is more research to be done in analyzing and representing sound as sound (sonorisation rather than visualization) and objects as objects. In any case, it is important to remember that media in digital form implies the transformation of another media form. An image of a painting or an album cover is a representation of the physical object; and an image of a digital image is its encoding, reproduction, compression, modification, and rendering.
In this section we present our work on media visualization. The
following projects have been developed mainly as research and
experimental practice; like tools for reflection. While putting in
practice existing techniques and methods for cultural analysis, we
try to explore new forms of representation and interaction. One of
our strategies has been the exploration of the aesthetics of digital
information through visual disruptions, that is, by reconfiguring
the expected functional mode of visual representations [
‘Motion structures’ is an ongoing project initiated in 2011 [
Our process puts special attention on shapes above other visual features. To create a motion structure we use a script we wrote for ImageJ. Basically, the operations require decomposing an animated video sequence into a series of separated image files, which are then manipulated as an image stack. Then, several image processing techniques occur under the hood: converting images to 8-bit format, subtracting background, and rendering the stack as 3D shape.
With motion structures we intend to represent the spatial and temporal transformations of a moving image sequence. The obtained 3D shape encodes the changes of the objects in a frame: the different positions, the movement traces, and spatial and temporal relations. The way in which we can interact with an object is not limited to ImageJ. The model can be exported and later manipulated in other 3D software applications such as Maya, Sculptris, or MeshLab. Furthermore, it is also possible to export a motion structure for the web or to physically print it, however both techniques require destructive 3D model processing, i.e. reducing geometry by simplification, decimation or resampling. For technical details, a motion structure exported from ImageJ has an average of 500,000 vertices and more than 1 million faces, which is a very large amount compared to an optimized 2000-face model for the web, loaded with the library three.js.
The current constraints of the exploration of motion structures in web-based environments and as printed objects can be seen as a similar path to the evolution of the representation of movement. Pioneers such as Etienne Jules Marey and Eadweard Muybridge first represented movement with pictures and images themselves, but later Frank Gilbreth abstracted the traces of movement and created diagrams made out of lines. At that moment, artists got inspiration from both types of representation with the intention to explore a vocabulary of symbols, myths, and psychic processes. So experimental projects on media visualization contribute to the design of data visualization in two manners. First, through the abstraction of shapes, traces and patterns, it permits discovering diagrammatical representations, spatial distributions of elements, and combinations of shapes. Second, through the inclusion of images and the design of exploratory and immersive experiences, it provides insights for investigating animated behaviors, combinations of colors, mix of media, and graphical indicators to improve the comprehension of patterns in non-figurative productions.
Our last example is an exploration in web-based media
visualization. We developed ‘RockViz’ with the intention to
produce web-based image mosaics and image plots [
The first step for our media visualization was to download all the images and make them available locally, so we could measure their chromatic features. We used ImageMeasure, a script for Image, to measure hue, saturation, and brightness values. Then, we used Open Refine to handle data, but more importantly to apply mathematical formulae to the measure of images and dynamically calculate their Cartesian position.
The image mosaic was ordered according to, first, median of hue; second, median saturation, and third, median brightness. To facilitate the exploration of the dataset, we added a filter engine that acts upon years, artist name, and album title. Finally, to make a little faster the loading of images, we produced two versions of each image: one is scaled to 100 x 100 px. and the other to 500 x 500 px. The small version is used for visual representations and the larger appears when the user clicks on an image, so she can observe more details of a single cover. Of course, a deeper study should consider larger dimensions of images but this was the largest resolution provided through AllMusic.
For image plots, we calculated spatial positions according to
measures of visual features. We decided to use Open Refine to
dynamically generate the HTML for each image because of two
reasons. First, Open Refine supports algebraic and trigonometric
operations so we could restrain the visual area to fit a resolution of
1024 x 768 px. Second, we originally used JQuery and the
function getJSON to communicate with a JSON database, but the
loading time is very slow for more than a few hundreds of images.
While an image plot requires translations of scale, for instance
years into maximum width in pixels (in our case 1024 px), we
also experimented with different representations inspired by
geometric figures. We used the main formula for Cartesian-Polar
transformations. Our first exploration, Figure 2, draws images
around polar coordinates, taking values from median hue and
median saturation, resulting into a chromatic circle-like
visualization. Figure 3 disrupt this formula to investigate how
images could be plotted according to different figures.
Web-based media visualizations contribute to information design
in recalling the need to adapt large amounts of information to
small screens. Moreover, it raises questions on making efficient
time-consuming operations for transferring data files. But
considering the web as presentation support also demands to
reconsider the value of early developments by the hypermedia
community and their potential implementation with contemporary
web technologies. We are thinking specially in the xanalogical
model, where visualizations of transclusions are depicted in a 3D
environment [
So far, the primary goal behind our media visualizations has been to practice and experiment with existing techniques. The interpretation of data and the explanation of visual cultural patterns have been put aside momentarily, but this is precisely one of the clues for our future work. We expect to use our visualizations as teaching resource but also to collaborate closely with historians, filmmakers, media artists, musicologists and other domain experts.
On the other hand, we believe there is still much to do at the level of interactivity. Research on hypermedia functionalities needs to be done for web-based visualizations. In the same line, models of representation also require to be tested and experienced. While text-and-number-based visualizations meet an explosion of models, diagrams, demos, libraries, etc., some of them simply are not suited for visual media. We believe there are two main domains where we can get valuable insights: media art and scientific imagery. For the former, artists often challenge our tools and our common viewing experience; they practice could be regarded as very innovative. For the latter, we must remember that digital images are not exclusive to design, arts and art history, a wide range of different disciplines use them as well: geography, astronomy, medical imaging, mathematics, physics, chemistry, biology, etc.
In this paper we have made a brief review of the emergent approach of media visualization: its main principles and requirements. We identified this approach mainly at the coupling of image processing techniques and information design. Today we can list a short gallery of media visualization techniques and projects that start settling guides and practices. In order to enrich current research and development on media visualization we observed that two domains are particularly interesting: on the one hand, the heritage of hypermedia functionalities, systems, abstractions, and models for web-based projects. On the second hand, experimental media art and software from other disciplines equally related to images (others than media studies and art history, for example sciences).
In the last part of our contribution we presented two explorations on media visualization. First, ‘motion structures’ an experiment on transforming an animated video sequence into a 3D digital model with the intention of revealing patterns of time and shape. Second, ‘RockViz’ a web-based media visualization comprising almost 2000 rock album cover images and its visualization through experimental image plots.
Further work should be conducted in the design visual indicators to improve the comprehension of media visualizations, which are often non-figurative and difficult to seize. At the same time, the non-figurative character of resulting processes can be seen as a move towards the symbolism of abstractions. By abstracting shapes, traces and patterns, new models emerge and can be applied to other domains.