Introducing the Sky and the Social Eye Alan Woodley Shlomo Geva Queensland University of Technology Queensland University of Technology Brisbane, Australia Brisbane, Australia a.woodley@ qut.edu.au s.geva@qut.edu.au Richi Nayak Timothy Chappell Queensland University of Technology Queensland University of Technology Brisbane, Australia Brisbane, Australia r.nayak@qut.edu.au timothy.chappell@qut.edu.a ABSTRACT • Is relatively inexpensive compared with field visits; and We introduce the Sky and the Social Eye task which was run for • Allows for a large spatiotemporal area to be investigated. the first time as a Grand Challenge at the 2016 ACM Multimedia Conference and as a Task Force at the 2016 MediaEval Despite this, satellite data also has limitations, for example: Workshop. Participants combined satellite images with social media to create a richer user experience. For the first year, the task • Satellites tend to have a low temporal frequency (for was exploratory allowing participants to produce their own example fortnightly) and so may miss an important event system without traditional constraints such as shared datasets or such as a flood or forest fire; and metrics. Here, we describe the task, summarize the participants’ • Clouds are present in some events, such as floods and approaches and propose some future directions. snowstorms, obscuring the ability of satellites to capture images. 1. INTRODUCTION Remote sensing data, such as satellite images, have been used to Examples of a satellite image is presented in Figure 1 [13]. Here, explore environmental and social challenges for decades. For two images are provided. Both images were taken in Sri Lanka example, NASA’s Landsat program has been used to identify the and show the impact of floods. The first image was taken on extent of forest fires, map land use change, assess carbon stock March 21st 2016 was taken prior to the flooding, while the second and analyze reef water quality [5]. Increasingly, social media is image was taken on May 31st 2016 after flooding. The second also being used for similar purposes, for example, mapping the image shows enlarged waterways from the flooding as well as the extent of damage from natural disasters such as floods or presence of cloud cover, which obscures the land below. earthquakes [4; 10; 11]. Despite this, there is a lack of research In contrast, social media is continuously being generated and is that explores how remote sensing data and social media can be unencumbered by issues such as cloud cover. Based on this, social combined. media is increasingly being used to address problems that were The Sky and the Social Eye task was proposed to fill this gap by traditionally addressed by satellite images analysis [4; 10; 11]. using social media to enrich satellite images. It ran as a Grand The Sky and the Social Eye task extends this research by linking Challenge at the Association of Computing Machinery (ACM) satellite images and social media, such as text, images and videos Multimedia Conference and as a Task Force at the MediaEval Workshop for the first time in 2016. The aim of the task was for 3. PARTICPANTS participants to develop algorithms and systems that combined Three groups participated in the inaugural Sky and the Social Eye. satellite data with social media data. For the first year, the task Ahmad et al. [1] developed a system called JORD, which was exploratory and creativity was encouraged. This allowed retrieved the names of events from the EM-DAT database [8] participants to develop their own systems, using their own and searched Twitter, Flicker, YouTube and Google to retrieve datasets, case studies and metrics. and fuse information about the event. They tested JORD with 80 Here, we describe the inaugural Sky and the Social Eye task. We events including floods, landslides, cyclones and wildfires. begin by detailing the motivations for the task and then outline Bischke et al. [2] produced a system that fused satellite images the approaches undertaken by the participants. We conclude by with Twitter data. They used a case study of a wildfire in Fort discussing potential future directions for the task. McMurray, Canada. They collected Landsat 8 satellite images from Amazon Web Services. Tweets, including text and other 2. MOTIVATION images, were collected from Twitter’s Historical Powertrack There are strong advantages for using satellite images to explore API. Then the tweets’ text was analyzed to identify geolocation social and environmental events. For example, satellite data: and sentiment and the tweets’ images were analyzed with a • Provides strong and compelling evidence about events on the convolutional neural network to remove near-duplicates. Finally, ground; the data were fused and presented as a visualization. Copyright is held by the authors MediaEval 2016, October 20–21, 2016, Hilversum, Netherlands. Figure 1. Satellite images which highlight the impact of a flood in Sri Lanka. Crandall et al. [6] used satellite data as (noisy) ground truth to Participants in the task combined social media with satellite train two photo image classifiers: first, one that estimated if a images, based upon on particular events. Sky and the Social Eye photo contained evidence of an event and second, one that was run for the first time in 2016 and the organizers plan on aggregated the estimates to produce an observation for given transitioning it to a traditional MediaEval Task for future years. times and places. Satellite images were sourced from NASA’s Terra satellite while photos were sourced from Flickr’s public 6. ACKNOWLEDGMENT API. Satellite images where classified into bins according to The organizers of the Sky and the Social Eye task would like to their observed percentage of snow. These bins were then used to thank the task participants and reviewers as well as the organizers train the photo classifiers using both support vector machine on of both the 2016 ACM Multimedia Conference and the 2016 tagged text and a convolutional neural network on the photos MediaEval Workshop. themselves. Individual photos were then aggregated and a SVM 7. REFERENCES trained over the aggregated data to estimate the probability of [1] Ahmad, K., Riegler, M., Nguyen, D.T.D., Halvorsen, P., the actual environmental state. Finally, the results of the Lux, M., and De Natale, F., 2016. JORD - Linking sky and experiment were collected and presented as a visualization. social multimedia data to events In Proceedings of the 23rd 4. FUTURE DIRECTIONS International Conference on Multimedia Modeling (Reykjavik, Iceland, 4-6 January 2016). The major future direction of Sky and the Social Eye will be to transition it towards a traditional MediaEval Task. This will [2] Bischke, B., Borth, D., Schulze, C., and Dengel, A., 2016. require the use of a shared set of documents, topics (queries) and Contextual enrichment of remote-sensed events with social metrics, enabling participants to better compare their systems to media streams. In Proceedings of the ACM Multimedia others. Moreover, the choice of the metrics is in itself a research Conference (Amsterdam, Netherlands 15-19 October problem as the metrics need to consider both multiple data types 2016). (that is: images and text) and draw together evaluations from the [3] Campbell, J.B. and Wynne, R.H., 2011. Introduction to remote sensing community, whose metrics tend to be based on set Remote Sensing. Guilford Press. retrieval (such as accuracy) [3], and the information retrieval communities, whose metrics tend to be list based (such as Mean [4] Cervone, G., Sava, E., Huang, Q., Schnebele, E., Harrison, Average Precision) [7]. J., and Waters, N., 2016. Using Twitter for tasking remote- sensing data collection and damage assessment: 2013 In addition, the processing of remote sensing/social media on a Boulder flood case study. International Journal of Remote cloud computing environment poses its own challenges. Sensing 37, 1, 100-124. Traditionally, operational remote sensing analysis has been [5] Cohen, W.B. and Goward, S.N., 2004. Landsat's Role in processed on a local server environment, often requiring the use Ecological Applications of Remote Sensing. BioScience 54, of heavy sampling [14]. A cloud computer/supercomputer 6 (June 1, 2004), 535-545. DOI= environment offers stronger computational capacity but has its http://dx.doi.org/10.1641/0006- own challenges, particularly regarding transfer of large datasets 3568(2004)054[0535:lrieao]2.0.co;2. around the computing environment and the need to handle streaming data – challenges which may not be solved by current [6] Crandall, D., Wang, J., and Korayem, M., 2016. Tracking massive parallelization paradigm such as MapReduce or Mahoot natural events through social media and computer vision. In [9; 12]. There is potential for the computer science community Proceedings of the ACM Multimedia Conference to play a leading role in this area of research. (Amsterdam, Netherlands 15-19 October 2016). [7] Croft, W.B., Metzler, D., and Strohman, T., 2010. Search 5. SUMMARY Egnines: Information Retrieval in Practice. Pearson, Here, we have described the Sky and the Social Eye task which Washington D.C., United States. was run as a Grand Challenge at the ACM Multimedia Conference and as a Task Force at the MediaEval Workshop. [8] Guha-Sapir, D., Below, R., and Hoyois, P., 2015. EM-DAT: based big data platform for massive remote sensing data International disaster database. Catholic University of processing. In Proceedings of the Proceedings of the Louvain. Second International Conference on Data Science (Sydney, [9] Lv, Z., Hu, Y., Zhong, H., Wu, J., Li, B., and Zhao, H., Australia, 8-9 August 2015), Springer International 2010. Parallel K-means clustering of remote sensing Publishing, 120-126. DOI= http://dx.doi.org/10.1007/978- images based on mapreduce. In Proceedings of the 3-319-24474-7_17. Proceedings of the 2010 international conference on Web [13] United States Geological Survey, 2016. Landsat 8 Imagery information systems and mining (Sanya, China2010), Reveals Heavy Flooding in Sri Lanka. Springer-Verlag, 1927687, 162-170. http://remotesensing.usgs.gov/gallery/gallery.php?cat=2#52 [10] Middleton, S.E., Middleton, L., and Modafferi , S., 2014. 5. Real-time crisis mapping of natural disasters using social [14] Wedderburn-Bisshop, G., Walls, J., Senarath, U., and media. IEEE Intelligent Systems 29, 2, 9-17. Stewart, A., 2002. Methodology for mapping change in [11] Schnebele, E., Cervone, G., Kumar, S., and Waters, N., woody landcover over Queensland from 1999 to 2001 2014. Real time estimation of the calgary floods using using Landsat ETM+. In Proceedings of the The 11th limited remote sensing data. Water 6, 2, 381-398. Australasian Remote Sensing and Photogrammetry Association Conference (Brisbane, Australia, September 2- [12] Sun, Z., Chen, F., Chi, M., and Zhu, Y., 2015. A spark- 6 2002).