Visual Interaction Design for Experiencing and Engaging with a Large Chronological Table Kumiyo Nakakoji, Yasuhiro Yamamoto, and Yusuke Kita Kyoto University, Japan kumiyo@acm.org, yxy@acm.org, yusuke.kita@design.kyoto-u.ac.jp Abstract The goal of our project is to design visual interactivity for a computational environment, which helps a user browse interconnected historical elements in a variety of manners. We argue that a computational environment to help people explore historical material needs to address the following three aspects: a network of changes, intertwined lines as trajectories, and associatable multiple proximities. The paper presents Timezoom, a prototyped environment, as a poof of concept to illustrate the three aspects. We use the chronological table of the history of the city of Kyoto, Japan, composed of 11,960 timestamped textual data elements. 1 Introduction Timezoom is for users to engage themselves in the historical data by dynamically changing the way the data is presented as their understanding and interests gradually develop. Our design is based on our previous work on the studies of design and design provenance [4][5] as well as the inspiration by the notion of “lines” described by Ingold [3]. Our project aims at designing the visual interactivity of a computational environment for a large amount of timestamped data, so that users (i.e., data analysts, design researchers, and historians) in the early stage of their research would find potentially interesting hypothesis, become aware of some implications, come up with a few keywords for further investigation, or encounter some room for new interpretations, by browsing the data through interacting with such an environment. We use the chronological table as an example of the historical data, which was originally published by the city of Kyoto in 1976 [1]. Kyoto, which used to be the capital of Japan between the 8th and 19th centuries AD, has a long history. The book consists of 505 pages of the chronological table covering years between AD 511 and 1971. There are 11,960 timestamped (year and month dated) textual descriptions as entries in the table. We scanned the table with the admission by the city of Kyoto, and generated the formatted table in a CSV format by using OCR and manually fixing recognition errors. M. Düring, A. Jatowt, J. Preiser-Kapeller, A. van den Bosch (eds.): Proceedings of the 3rd HistoInformatics Workshop, Krakow, Poland, 11 July 2016, published at http://ceur-ws.org 2 The Three Aspects This section describes the three aspects we argue for in designing interactive environments for users to explore a large amount of timestamped textual historical data in the early stage of their research. (1) A Network of Changes. By viewing design as “changing existing situations into preferred ones” [7, p.111], our design studies in the domain of city development and urban design has focused on the change a design provokes [4]. The subsequent changed situation leads to other problematic situations and potentials in different contexts, thus resulting in re-designs. The world remains changing through reiterating such design activities and a series of designs take place like chain reactions. Rather than focusing on the provenance of a single designed artifact, we focused on multiple artifacts that are thus interdependent on one another through the changes they provoke. “Expanded phenomena” may often be observed in our history as well as in our daily life, without explicit recognition of their underlying design activities [4]. Based on such understanding, we proposed a model and a method for depicting the history by focusing on changes of objects [4]. The proposed framework uses a network of changes of artifacts, persons, and situations. The temporal network structure, named C-network (Change-network), consists of a large number of C-units (Change units), each of which is composed of the timestamped three layers: A (Actor/Agency), B (Background) and C (Changed object, from C1 to C2). Each C-unit may be linked to one or more other C-units through one of its three layers, forming a network of changes. Figure 1 illustrates one of the results of our case study that uses our model to depict the development of streets in Kyoto. The city of Kyoto currently has double grid street system consisting of many narrow streets (4 to 10 meters wide) and some wide streets (15 to 30 meters wide). Some of the wider streets have been the results of widening narrower streets in order to introduce a tram network at the time of modernization in the early 20th century (the tram systems were afterward removed in the late 20th century). Those narrow streets emerged during the 9th to 12th century, when people living in the city of Kyoto (then called Heiankyo) started to occupy parts of then-existing wide streets and cultivated or built houses there without government permission due to the decline of the governmental power caused by political confusion. At the time Heiankyo was built in late 700's, Heiankyo had streets as wide as 12 to 84 meters because it was modelled after the Chinese cosmology by Emperor Kanmu. Figure 1: The Historical Development of the Streets of Kyoto Depicted as a Network of Changes (2) Intertwined Lines as Trajectories. We are inspired by the concept of “lines” described by Ingold [3], who uses a line to depict the life of a person, of a thing, of a city, or a wide variety of phenomena such as music, speech, or travel. A line is drawn from left to right where the horizontal axis corresponds to the flow of time (Figure 2). A number of lines, each of which corresponds to a person, an object, music or a book, co-exist by sharing the same period of time (the same vertical area), and some of them may be intertwined, producing knots. Such knots represent meeting points, encounters, or interactions among the intertwined lines. One of the interesting aspects of using the notion of lines to express trajectories of people and objects is where the relationships among people and objects are represented as knots, and the collection of the large number of intertwined lines form what Ingold calls a meshwork. In contrast to the network of changes we described above, the notion of lines naturally expresses continuation, duration, and co-existence. Figure 2: Lines from a Meshwork (3) Associatable Multiple Proximities. The goal of the DPS (Design Practice Streamliner) [5] system was to help a designer re-experience the trace of thoughts during a recorded design meeting to investigate how the decision had been drawn, what consequences it had led to, how the questions being raised had been addressed, or how the problem had been resolved, by extracting the relevant portions of recorded meeting captured in video, transcripts, and whiteboard stroke data by using timestamps to glue segments in different media together. When a user selects a small rectangular region on the whiteboard stroke data browser, DPS identifies a set of timestamped strokes that are included in the specified area, and produces a short video consisting of the video segments that have the same set of timestamps. Those timed segments are often scattered at different time periods along the timeline, as the designers went back and forth among different areas on the whiteboard during the meeting. When the user specifies a phrase as a search query in the transcript browser, the system locates one or more sentences among the timestamped transcripts that have the queried phrase, and likewise produces a short video consisting of the video segments that have the same set of timestamps, as well as highlights strokes on the whiteboard data browser to show which areas of the whiteboard were drawn when uttering the queried phrase. The interaction design of DPS is such that it would help the user search for points of concern through spatial proximity over the whiteboard stroke data browser, symbolic proximity over the transcript data browser, and through temporal proximity over the video browser. Starting with exploring an area in the proximity of one type of media, the user is then presented with the associated, possibly scattered data points in other media by DPS. Thus, DPS pulls out streams of design thinking from the recorded design history, which may likely be intertwined, entangled, and weaved in the design process across multiple types of representations, especially during the early stages of software design. 3 Timezoom: A Prototyped Environment Timezoom (Figure 3) has been prototyped as a proof of concept for the three aspects. It is currently going through a series of revisions. Timezoom displays timestamped text data in the 3-D space using the Web browser. The textual description of the highlighted data element is displayed at the top of the window, preceded by the year and the Japanese calendar year, and followed by the cluster ID’s (see below). Each timestamped text data element is diagramically visualized as a horizontal thin blue line at the corresponding height of the time (vertical) axis; the length reflects the length of its text description for the sake of giving visual identity to each data element. Figure 3: Timezoom Timezoom provides the following three ways to experience proximity: Chronological proximity along the timeline. The environment allows a user to browse a timestamped text data element one by one in the chronological order within the scope of the proximity, which gives us a feeling of linearity among the successively visited data elements. The vertical axis represents the timeline, where the time proceeds from the top to the bottom. There always a single highlighted data element in the visualization, which is emphasized with red. The user may click on one of the chronological-track buttons (triangles located on the right side of the window) to change the highlight by traversing all the data elements temporally forward (i.e., the next one), or backward (i.e., the previous one). Symbolic proximity through the searching mechanism. The user may search for a particular word or phrase among the text descriptions of the data elements. The identified data elements, the text descriptions of which contain the queried word or phrase, are visually highlighted with green. Clicking on one of the two sought-track buttons (the two triangle buttons at the bottom of the window) traverses the located search results. Semantic proximity through the clusters. The horizontal axis (the width in Figure 3) is currently mapped to the clusters of timestamped text data elements in the order of their sizes. Clustering may be done by a variety of ways, and we currently apply the Micro-Clustering algorithm [9] to identify clusters among the timestamped text data using words composing the text description of each data element. When one data element is highlighted, one of its belonging clusters is also highlighted, and all the other data elements that belong to the highlighted cluster are visually stressed by changing its color from light grey to dark grey (Figure 3). As each cluster is positioned along the horizontal axis, the data elements of the same cluster are aligned vertically. The user may click on one of the cluster- track buttons (triangles located on the lower right side of the window) to traverse all the data elements belonging to the currently highlighted cluster. While visiting data elements one by one, one may change the course of line tracking by switching to another proximity. While following a series of the data elements within a certain cluster, one may switch to one of the different clusters the highlighted element belongs to (by selecting the cluster ID). Or, one may switch to simply following the data elements in the chronological order without being bound to any clusters. One may find a phrase appeared in the textual description of the data element interesting and search for the phrase and start browsing the searched results. Such switching proximity context may serve as a knot to intertwine the two lines that the user has been engaging in. 4 Conclusion Timezoom is an environment for the early stage of investigating historical data, which is likely to precede more robust search (such as Singh et al. [6]) on the more formally represented historical data. Timezoom is under development and we are working on evolving Timezoom in several ways. In addition to bringing in elements of aesthetically meaningful visualizations such as works by Sugiura [8] and Grafton and Rosenberg [2], we will associate the geographical information with each data element so that we can use geographical proximity to experience a series of events within the spatial context. This would especially be interesting to browse the Kyoto history data as Kyoto has very old streets and buildings, which have been extended and demolished in a variety of manners through the long history. Acknowledgements The research presented in this paper is supported by CREST, JST. We would like to thank Takeaki Uno, Nobuto Matsubara, and Tomohiro Sakaguchi for their technical advice and support. References 1. City of Kyoto (Eds.), The History of Kyoto, Vol.10, Gakugeishorin, 1976 (in Japanese). 2. Grafton, A., Rosenberg, D., Cartographies of Time: A History of the Timeline 1st Edition, Princeton Architectural Press, 2010. 3. Ingold, T., Lines: A Brief History, Routledge, 2007. 4. Kita, Y., Nakakoji, K., Monnai, T., Depicting the History as Expanded Phenomena: An Approach to Wide, Longitudinal Design Studies, Proceedings of IASDR2015 INTERPLAY, Brisbane, Australia, 2015, 1084-1098. 5. Nakakoji, K., Yamamoto, Y., Matsubara, N., Shirai, Y., Toward Unweaving Streams of Thought for Reflection in Early Stages of Software Design, IEEE Software, Special Issue on Studying Professional Software Design, Vol.29, No.1, 2012, 34-38. 6. Singh, J., Nejdl, W., Anand, A., History by Diversity: Helping Historians search News Archives, Proceedings of CHIIR 2016, ACM, New York, NY, USA, 2016, 183-192. 7. Simon, H.A., The Sciences of the Artificial (3rd ed.), Cambridge: The MIT Press., 1996. 8. Sugiura, K. et al., Experiments in Time Distance Map, Diagram Collection by Kohei Sugiura, Kashima Shuppankai, 2014 (in Japanese). 9. Uno, T., Maegawa, H., Nakahara, T., Hamuro, Y., Yoshinaka, R., Tatsuta, M, Micro-Clustering: Finding Small Clusters in Large Diversity, CORR abs, 2015, 1507.03067.