Revealing and Interpreting Crowd Stories in Online Social Environments Chris Kiefer, Matthew Yee-King and Mark d’Inverno Department of Computing, Goldsmiths, University of London m.yee-king@gold.ac.uk Abstract Music Circle is an online social platform, aimed at explor- ing ways of understanding and enhancing learning through The underlying patterns in large scale social me- community feedback. For six weeks in the summer of 2014 it dia datasets can reveal valuable information for in- was used to support a Coursera MOOC, ’Creative Program- teraction designers and researchers, both as part of ming for Digital Media and Mobile Apps’1 . A substantial realtime interactive systems and for post-hoc anal- and detailed log of student interactions was collected. While ysis. Music Circle is a social media platform aimed there were specific questions that could be asked of the data, at researching the role of community feedback in such a large and complex set of interactions could most likely online learning environments. A large dataset was hold some interesting and unexpected results, and it seemed collected when the platform was used as part of pertinent to follow a bottom-up approach to data analysis, by a Massive Open Online Course (MOOC). We de- letting patterns emerge rather than imposing them. To this veloped a novel analysis technique for observing purpose, a set of techniques was developed that attempted to global patterns in the behaviour of students. The elucidate the broad patterns and temporal dynamics of crowd technique employs network theory techniques to behaviour that occurred during the period of the MOOC, to view student activity as an interconnected com- transform the raw data into a format that would give the re- plex system, and observes the temporal dynamics search and design teams a deeper understanding of student of network metrics to create timelines which are interactions within the Music Circle environment. clustered into groups using unsupervised learning A novel approach was developed, which leveraged network methods. This approach highlighted global trends analysis and machine learning techniques to cluster tempo- and groups of outliers that needed further attention ral data. We outline the development of this technique and or intervention. present and critique the results. The following sections ad- dress the research questions that were encountered during this development process: how can social media data be encoded 1 Introduction into a human readable form that describes temporal patterns Online social activity has become a fundamental part of many in actor behaviour? How can network analysis techniques en- interactive systems, either explicitly as part of their intended hance this encoding? What are good ways to present this data design, or implicitly as part of external and pervasive social for interpretation by stakeholders? media networks. With social activity comes large or mas- We present this research as a technique for eliciting infor- sive scale data, which describes interactions between individ- mation from large datasets for analysis by stakeholders and uals mediated through a variety of possible formats. These domain experts, rather than as a process which will supply datasets can reveal stories about individuals and groups that absolute answers concerning student behaviour. In this light, may be of high significance to stakeholders; for interaction we do not attempt to provide a quantitative evaluation of the designers, this data can show aspects of behaviour that reveal effectiveness of the findings, but try to show, through cross- design problems, suggest design solutions and highlight di- checking of results in a post-hoc analysis of the Music Circle rections for future iterations. For researchers, this data can MOOC data, the potential strengths of our method for use in give us a broad understanding of trends in behaviour within future projects. the context of specific technological environments. Large datasets also present significant challenges in analysis, with 2 Related Work the scale of raw data often making direct human interpreta- Our approach is rooted in a network theory perspective. Ji- tion an intractable task. However, by bringing computational awei et. al [Han et al., 2012] review data mining in this analysis into the loop, we can attempt to sculpt the raw data context, proposing that we can extract much more valuable into new forms that, while not necessarily giving absolute an- swers, present data in a suitable format for further (human) 1 https://www.coursera.org/course/ interpretation and criticism. digitalmedia 47 information from a database by viewing it as a heteroge- computer. We built networks of data relating to singular con- neous information network rather than a homogenous data cepts, and observed several network metrics as they evolved repository. We draw on techniques highlighted by Holme over time. In this way, we could use network measurements and Saramäki [Holme and Saramäki, 2012]; they review the that put an individual’s actions into a global interconnected emerging field of temporal networks, looking at techniques context, rather than observing them in local scope. for analysing how network topology changes over time, and Two sets of networks were built, that separately repre- how temporal information flows. There have been varied ap- sented commenting and viewing activity. Each set consisted proaches to social network analysis, for example, Gottron of networks that evolved in two hour windows over the pe- and Pickhardt[Gottron et al., 2013] explore techniques for riod of the MOOC, giving 503 networks in each set. This two temporal analysis of social data, Gilbert et. al. use statisti- hour period was chosen as a compromise between time res- cal methods to analyse Pinterest [Gilbert et al., 2013], and olution and practical limitations in data processing capacity. Diya et. al. [Yang et al., 2013] look for causes of student The networks had directed and weighted connections; each dropouts in MOOCs using a network theory approach. Rowe node represented a student, with weighted links representing et. al. [Rowe et al., 2013] outline their technique for mod- the numbers of comments or views made from one student to elling and analysing the behaviour of users in online commu- another. nities. They focus on defining individual role categories, and In this analysis, a set of four metrics were chosen for obser- look at how the global composition of these roles changes vation from each network. The first two were simple metrics over time. Chao et. al [Chau et al., 2011] look at the intersec- which sum the (a) incoming and (b) outgoing weights of each tion between HCI and data mining, investigating interactive node. These could also be calculated without using a net- machine learning approaches, and sensemaking. work. The next was (c) betweenness centrality. This metric was chosen as it provides interesting representations of each 3 Music Circle user’s importance within the global context of the network, based on how much information flows through their node. It Our system (pictured in figure 1) allows students to share shows the extent that the actor is positioned on the shortest and discuss creative work, and acts as a research platform for path between other pairs of nodes in the network [Leydes- studying the role of social media in learning [Brenton et al., dorff, 2007]. Betweenness centrality is calculated based on 2014]. The key feature is the Social Timeline, an online en- link direction and weight, thereby using the full information vironment for annotating and discussing time-based media. available in the networks we constructed. The last metric was The Social Timeline allows students to highlight and com- a calculation of each node’s (d) HITS authority. This was ment on sections of time-based media, and to further discuss calculated with the HITS algorithm [Kleinberg et al., 1999], these comments. The website has been used in a variety of which gives a measurement of the importance of a node based scenarios to explore creative feedback [d’Inverno and Still, on link structure. More specifically the algorithm gives each 2014] between students, including jazz piano tuition and as a node two co-dependent scores; a hub score based on the au- rehearsal support tool for musical ensembles. thority of nodes that link to it, and an authority score based Over the course of a MOOC in the summer of 2014, the on how the degree to which the nodes that point to it are hubs. website was employed for students to discuss, share feedback These four metrics were observed for each two hour iter- on and peer assess videos of their coursework pieces. During ation of the networks, giving each student a set of timelines, the six week course, the students were required to submit a one for each metric. The analysis provided a rich data set for piece of coursework every two weeks. Each coursework brief further exploration. Network analysis was carried out using asked them to program a software application, and to submit iPython with the NetworkX library. a video demo of their application to the Music Circle website for peer assessment. Students were also asked to peer assess three other peoples’ work for each submission. As part of the 5 Discovering Themes peer assessment process, they could discuss other students’ work through the website. Having collected the sets of timelines for each user, the next To give an overview of the course statistics, during a six step was to cluster these timelines into similar groups to re- week period, 3716 users registered with the website. Of these, veal underlying patterns. An exploratory approach was taken, 3558 viewed one or more videos, 827 made one or more com- searching for interesting features in the data set by creating ments, and 258 made one or more replies to comments. 2898 both clusters of single features and clusters of compound fea- videos were submitted for three separate assessments, and tures in order to reveal correlations between groups of met- were viewed a total of 112,189 times. 7370 comments were rics. Useful clustering results were obtained using two meth- made, along with 978 replies. Detailed log data was collected, ods: k-means alone, and k-means with unsupervised pre- including timestamped records of all discussions and all me- training using Restricted Boltzmann Machines. In the lat- dia viewing activity. ter case, RBMs were used to find sparse, low dimensional representations of the salient features in the data, before k- 4 Encoding Stories means clustered these features. We used the Extended RBM from the Oger toolbox [Verstraeten et al., 2012], with gaus- Having collected the data, we needed to present it in a format sian visible units for our continuous valued data. The RBMs that was both interpretable by humans, and cluster-able by a were trained with guidance from [Hinton, 2010]. 48 Figure 1: A Screen Shot of the Music Circle Website 49 6 Visualising Crowd Behaviour Having calculated clusters, we visualised them in two ways. Simple graphs of means and variances for each cluster group (e.g. figure 2) gave an easily interpretable summary. A more complex view showing members of individual clusters was presented in sets of polar graphs, where each graph dis- played every individual timeline in a cluster, superimposed with semi-transparent colouring to show patterns of density (e.g. figure 3). Upon identifying a cluster of interest, a set of graphs was generated to show the cluster mean for each metric, compared to the global means (e.g. figure 4). 7 Results An exploratory approach was taken to analysing the data, vi- sualising features separately and in combination to find clus- ters of possible interest. The following examples describe salient outcomes of this process. 7.1 Example A: Betweenness Centrality Figure 3: A polar plot of superimposed timelines from a sin- gle cluster Figure 2: Mean View Betweenness Timelines, in 5 Clusters Figure 2 shows the means of 5 clusters of betweenness cen- trality timelines from the views network, generated by train- Figure 4: The cluster mean vs global mean for ‘comments ing an RBM with 503 visible units and 5 hidden units, the out- made’ put of which was clustered with k-means. In this context, we could consider betweenness centrality to indicate the extent to which a student is engaged in a community of other students who are active in viewing each others’ work. A large group of low activity users is shown in cluster 2, which is what would be expected in a social network dataset. A smaller group of high activity users is highlighted in cluster 4 (shown in more detail in figure 3). This timing of this higher viewing ac- tivity correlates with an incentive being offered to students for engaging in forum activity. Cluster 3’s value is dropping while other clusters are rising, indicating that this cluster may include students who need attention in some way. Further analysis shows that the number of comments received by this group is below the global average (see figure 4), strengthen- ing the case that this group may need some sort of help or motivation. 7.2 Example B: HITS Authority Figure 5: Means of HITS authority for the comments network Clusters of the HITS authority timelines for the last 40% of the course were clustered using k-means (shown in figure 5). The graph shows that the students in cluster 4 have a steadily 50 declining authority, which may indicate the students in this of interest, which are validated by patterns in other metrics group have low activity and are therefore becoming less im- on other domains. e.g. in example A, the betweenness cen- portant community members. The concern is verified when trality clusters for viewing behaviour highlight a group that looking at their viewing timelines; they are significantly be- may need attention. Further investigation of the comment- low the global mean for betweenness centrality of views. ing network reveals that this group is less active at making comments, compared to the global mean. Examples A and B 7.3 Example C: A Compound Feature also demonstrate the values of analysing our data from a net- work perspective, viewing user activity as a complex evolv- ing system of interconnected nodes. In example A, the clus- ters highlight a group whose betweenness centrality is drop- ping progressively. Observing non-network based metrics for this group, i.e. the number of views made and received, the timelines for this group do not differ greatly from the global average, so this group would not show up in any clusters. However, the highlighting of the group is validated by their inactivity in commenting. In example B, a group is revealed whose HITS authority for comments is dropping. Again, this would be difficult to spot from this group’s number of com- ments made and received, which are close to the global aver- age, but the choice is validated by revealing their low view- Figure 6: ing activity. Overall, network analysis algorithms such as betweenness centrality and HITS evaluate each node in the Figure 6 shows 10 clusters of a compound feature, made wider context of a complex system. This means these met- with k-means. The first half is a betweenness centrality from rics are much more sensitive to global events in the network, the views network, and the second half is the same metric and can reveal dynamics that locally scoped measurements from the comments network. By joining features in this way, may fail to. The results show the merits of temporal analysis the clustering process can pick out potentially interesting cor- of these network metrics; the clusters of interest were high- relations or contrasts between the sources. In this example, lighted by discovering anomalies in temporal dynamics, and we can see that the students in cluster 5 have a relatively reveal more detailed information compared to instantaneous high value for views but a very low value for comments. This analysis. could indicate students who are potentially active, but are part A challenge of using this system is in interpretation. To of low activity peer communities, and would perhaps benefit fully interpret a graph of clusters, it is necessary to under- from being introduced to new peers. stand the network analysis metric being presented, along with its meaning in the context of the network and in the wider 8 Discussion context of the source domain of the data. For example, to un- By exploring and clustering the sets of timelines describing derstand betweenness centrality of commenting activity, we behaviour on Music Circle, it was possible to reveal global need to understand the concept of this measurement along patterns of crowd behaviour in the specific context of the with the network theory that supports it, and we also need to network metrics we observed. The clusters also highlighted understand how people are connected by comments on Music groups of outliers; understanding these small groups can be Circle and the affordances of the interface that allow activity of great value in trying to understand a complex social sys- to propagate though the network of students. It’s also a chal- tem, both in terms of isolating problems, and understanding lenge to present the clusters in an optimal way. The means positive deviance [Ramalingam, 2013]. The technique can and variance give a good idea of general trends but miss some be used both post-hoc and for live analysis. Post-hoc anal- details. The graphs of superimposed timelines can become ysis can help us to understand crowd behaviour in order to dense and difficult to compare, but do give much more detail. improve the design of future iterations. Live analysis has a Conducting analysis with both of these perspectives seems a number of possible uses in the context of our platform; out- good compromise, but ideally an interactive tool would be lying groups may predict students who are likely to drop out very useful. or disengage, and need some contact, reward or support from peers and teachers. Outliers may also highlight successful 9 Conclusions groups who we may try to link with new peers to strengthen the overall community of learning. A further use is to give The motivation for this project was to reveal patterns of global students live feedback of their status in terms of these metrics, crowd behaviour based on a large scale database of social and in order to aid their learning or motivate them. For example, a educational activity. Our approach was to look at simple in- live timeline of centrality in the comments network, together formation through the perspective of network analysis. We with a summary based on cluster membership, could provide observed how a variety of network analysis measurements a good motivator to increase commenting activities. vary over time, and then undertook an exploratory analysis The first two examples in particular demonstrate the poten- of these timelines though clustering. The clusters highlighted tial strengths of our analysis technique. Both highlight groups interesting global behaviours of groups of users, and also re- 51 vealed smaller groups of outliers that may need some sort [Han et al., 2012] Jiawei Han, Yizhou Sun, Xifeng Yan, and of intervention or attention. The strength of this technique Philip S Yu. Mining knowledge from data: An information is demonstrated in examples where the highlighted clusters network analysis approach. In Data Engineering (ICDE), were shown to need attention though cross checking with 2012 IEEE 28th International Conference on, pages 1214– other data sources. The possibilities of our technique were 1217. IEEE, 2012. demonstrated through post-hoc analysis of forum data. The [Hinton, 2010] Geoffrey Hinton. A practical guide to train- next step would be to apply this technique on a live forum, ing restricted boltzmann machines. Technical Report 1, and observe the effects of any pedagogical interventions that University of Toronto, August 2010. are made based on the analysis of the resulting data. [Holme and Saramäki, 2012] Petter Holme and Jari Saramäki. Temporal networks. Physics reports, 10 Future Work 519(3):97–125, 2012. The analysis highlights two areas which could benefit from [Kleinberg et al., 1999] Jon M Kleinberg, Ravi Kumar, Prab- further development. Firstly, the development of presentation hakar Raghavan, Sridhar Rajagopalan, and Andrew S tools to aid human analysis of the clusters. Secondly, the net- Tomkins. The web as a graph: measurements, models, work analysis algorithms employed here have been successful and methods. In Computing and combinatorics, pages 1– in highlighting clusters but also add an extra layer of interpre- 17. Springer, 1999. tation. It would be interesting to investigate the development of domain specific networks measurements, whose output is [Leydesdorff, 2007] Loet Leydesdorff. Betweenness central- closely matched to the semantics of the forum behaviour be- ity as an indicator of the interdisciplinarity of scientific ing observed. journals. Journal of the American Society for Information Science and Technology, 58(9):1303–1319, 2007. Acknowledgments [Ramalingam, 2013] Ben Ramalingam. Aid on the edge of The work reported in this paper is part of the PRAISE chaos: rethinking international cooperation in a complex (Practice and Performance Analysis Inspiring Social Educa- world. Oxford University Press, 2013. tion) project which is funded under the EU FP7 Technol- [Rowe et al., 2013] Mattew Rowe, Miriam Fernandez, and ogy Enhanced Learning programme, grant agreement number Harith Alani. Modelling and analysis of user behaviour in 318770. online communities: Ieee computer society special techni- cal community on social networking e-letter. IEEE Com- References puter Society Special Technical Community on Social Net- working E-Letter, 1(2), May 2013. [Brenton et al., 2014] Harry Brenton, Matthew Yee-King, Andreu Grimalt-Reynes, Marco Gilles, Maria Krivenski, [Verstraeten et al., 2012] David Verstraeten, Benjamin and Mark d’Inverno. A social timeline for exchanging Schrauwen, Sander Dieleman, Philemon Brakel, Pieter feedback about musical performances. In Proceedings of Buteneers, and Dejan Pecevski. Oger: modular learning the 28th International BCS Human Computer Interaction architectures for large-scale sequential processing. The Conference on HCI 2014-Sand, Sea and Sky-Holiday HCI, Journal of Machine Learning Research, 13(1):2995–2998, pages 281–286. BCS, 2014. 2012. [Chau et al., 2011] Duen Horng Chau, Aniket Kittur, Jason I [Yang et al., 2013] Diyi Yang, Tanmay Sinha, David Adam- Hong, and Christos Faloutsos. Apolo: making sense of son, and Carolyn Penstein Rosé. Turn on, tune in, drop large network data by combining rich user interaction and out: Anticipating student dropouts in massive open online machine learning. In Proceedings of the SIGCHI Con- courses. In Proceedings of the 2013 NIPS Data-Driven ference on Human Factors in Computing Systems, pages Education Workshop, 2013. 167–176. ACM, 2011. [d’Inverno and Still, 2014] Mark d’Inverno and Arthur Still. Creative feedback: a manifesto for social learning. In Pro- ceedings of the Workshops held at Educational Data Min- ing 2014 conference, 2014. [Gilbert et al., 2013] Eric Gilbert, Saeideh Bakhshi, Shuo Chang, and Loren Terveen. ”i need to try this”?: A statistical overview of pinterest. In Proceedings of the SIGCHI Conference on Human Factors in Computing Sys- tems, CHI ’13, pages 2427–2436, New York, NY, USA, 2013. ACM. [Gottron et al., 2013] Thomas Gottron, Olaf Radcke, and Rene Pickhardt. On the temporal dynamics of influence on the social semantic web. In Semantic Web and Web Science, pages 75–87. Springer, 2013. 52