Sheetmusic: Making music from spreadsheets Thomas Levine csv soundsystem _@thomaslevine.com ABSTRACT 1.2 Data-driven music videos The spreadsheet provides an intuitive paradigm for the ex- Combining the visual and auditory senses, we can plot data pression of musical scores. Musical scores can be expressed in the form of music videos. One example of this is the FMS as data tables, with each record corresponding to a place in Symphony (figure 1). In the FMS Symphony, each beat time and each column corresponding to a note or instrument. of music corresponds to a business day during the period Sheetmusic is a plugin for Gnumeric that provides music se- between 2005 and 2013, the pitch of one instrument corre- quencing spreadsheet functions. Tools like Sheetmusic pro- sponds to the United States interest rate, the pitch of an- vide intuitive music composition interfaces for people who other instrument corresponds to the distance to the United are used to data analysis software. Moreover, they help us States debt ceiling, and the activation of certain flourishes plot data with the sense of sound. corresponds to changes in the balance of the United States treasury. These data are also represented visually, through the combination of an animated line plot and a Chernoff face Keywords [1]. music, spreadsheets, gastronomification, data analysis 1.3 Food 1. NON-VISUAL DISPLAYS OF QUANTITA- Why stop at just vision and hearing? We can plot data as TIVE INFORMATION food and use all five senses. One example of this is Census Data analyists often use visualization as a means for plotting Spices, a set of spices that represent different neighborhoods data, but there are other approaches! based on demographics collected by the United States Cen- sus [5]. 1.1 Data sonification 2. OUR TOOLS FOR MUSICAL PLOTTING Just as data can be expressed visually, data can also be We at csv soundsystem have been exploring multisensory expressed in sound. As demonstration of this, Ferguson & data plotting methods, including music videos and food. In al. [4] created auditory analogs for simple visual plots, such our production of data-driven music videos, we have recog- as the dotplot and boxplot. nized a need for data analysis software and music software to be more strongly integrated. We wanted a more seamless Visual plots are far more common than auditory plots. Why transition between modeling and music, and we wanted it is this? My hunch is that our technology for visual rendering to be easier for data analysts to work with music. We have is simply much further advanced; printing technology has developed tools like Sheetmusic to bridge this gap. been around for centuries, and writing has been around for millenia. With this history, we have also developed advanced To use the language of the Grammar of Graphics, [8] we theory related to the visual plotting of data. Audio recording have abstract data and concrete plot elements, and we define is a comparably recent invention, and our theory around aesthetics that provide mappings between the abstract data auditory plotting is accordingly less developed. and the concrete elements. The primitive plot elements that we use for music are things like key, rhythm, pitch, and In my view, we separate data sonification from data visual- interval. ization only because of technological constraints; there isn’t a fundamental difference between these two processes. 2.1 Data tables We’ve found that the tabular representation of data aligns very well with typical representations of music. Our data music tools work by mapping these two concepts to each other. We can think of data tables as collections of similar things, with the same sorts of information being collected about each thing. In tidy data tables [7] each row corresponds to an observation (a thing), and each column corresponds to a Figure 1: Here is a frame from the FMS Symphony video. I unfortunately can’t play the accompanying song in this paper. variable. We add more rows to the table as we observe more The data analyst can use conventional spreadsheet approaches things, and we add more columns to the table as we collect for composing music. For example, the following function more information about each thing. can be used to produce a major scale in a spreadsheet col- umn. We can think of music as a composition of many different sounds over time, with sounds coming from many different instruments. In musical scores we represent time as move- =IONIAN_SCALE("A4") ment from left to right, and we represent different notes played at the same time by different dots on a staff, The Once you have a major scale in one column, you can easily staff becomes wider as the song gets longer, (They are often make chords with a spreadsheet functions like this. spread across multiple pages.) and we add more dots as we add more notes (figure 2). =MAJOR_THIRD_INTERVAL(B1) Rather than composing music as traditional sheet music, we can use a table-editing program of our choice to compose this sort of table. Our data music software simply adds musical If you put this in cell B1, A1 and B1 will form a major third functions to table containers in various data analysis tools. interval. Sheetmusic is our offering for spreadsheets, but we also have 3.3 Rendering music libraries for R data frames [3] and Pandas data frames [2]. Once we have composed our piece, we can select the appro- priate cells, specify the key and time signatures of the piece, 3. HOW TO USE SHEETMUSIC and export it as MIDI or sheetmusic. Let’s divide Sheetmusic’s features into two groups. The first group is spreadsheet functions for music synthesis—these It is possible, of course, to convert to any number of music are functions like CHORD_PROGRESSION that take spreadsheet formats, just as we can convert spreadsheets to any num- cells or values as input and return values to other spread- ber of data table formats. Only MIDI and sheetmusic are sheet cells. The second group is functions for rendering the implemented at present, but you can indirectly convert to music to external devices, including MIDI and sheetmusic. many formats by first saving as MIDI and then converting from MIDI to your output format of choice. 3.1 Organization of the spreadsheet Sheetmusic expects that the spreadsheet be organized as fol- 3.4 Musical plots lows. Each column corresponds to a musical track, and dif- I’ve discussed how we can use Sheetmusic for conventional ferent tracks can have different music instruments. Row music composition. To use it as a plotting tool, we simply corresponds to a beat (of time). Each cell contains the fre- have to map our abstract data to musical notes. Sheetmusic quency of sound to be played, represented in scientific nota- provides the FROMWHOLENUMBER function to enable this. If we tion (C4, D4, &c.). imagine an infinitely wide piano with the C0 as the left-most note, FROMWHOLENUMBER starts at C0 and walks i keys to the 3.2 Composing music right, where i is the argument passed to FROMWHOLENUMBER. Figure 2: A spreadsheet is displayed alongside some corresponding ordinary sheet music, with a corresponding row/beat highlighted. After using ordinary spreadsheet modeling functions to ma- nipulate data, a user may scale and round the data appropri- ately and then run FROMWHOLENUMBER to convert them into notes. 4. RELEVANCE I hope that I’ve shown how data can be plotted in the form of music. I would be remiss not to discuss the merits of this plotting method. 4.1 Easier composition of music When we plot data as music, we effectively let data compose music for us. We still have to choose datasets that will produce interesting music and map the data to the music appropriately, but the randomness of the data can provide the various subtleties of music that we would otherwise have to design ourselves. 4.2 Data literacy When we start using data analysis software for other things, we blur the line between data analysis and other things. Data analysis seems very magical to many people. When we represent data as familiar things like music, people seem to be a bit less scared of data analysis. 4.3 Expressing high-dimensional datasets The use of multiple senses may also allow for the expres- sion of high-dimensional datasets. Tufte advocates for the production of visuals that express the multivariate nature of Figure 3: Using Sheetmusic to render a spreadsheet the world.[6] I think that the use of multiple senses has the as sheetmusic potential to facilitate the expression of more easily express many variables at once, and this may aid in the identification of high-dimensional relationships. 5. REFERENCES [1] B. Abelson, J. Bialar, B. DeWilde, M. Keller, T. Levine, and C. Podkul. FMS Symphony, 2013. [2] csv soundsystem. Data music for big data analysis, 2013. [3] csv soundsystem. ddr: Data-driven Rhythms in R, 2013. [4] S. Ferguson, W. Martens, and D. Cabrera. Statistical Sonification for Exploratory Data Analysis. [5] H. Kang-Brown. Making Census Data Taste Like New York City, 2013. [6] E. R. Tufte. The Visual Display of Quantitative Information. Graphics Press, Cheshire, CT, USA, 1986. [7] H. Wickham. Tidy data. http://vita.had.co.nz/papers/tidy-data.pdf. [8] L. Wilkinson. The Grammar of Graphics. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2005.