=Paper=
{{Paper
|id=Vol-1739/MediaEval_2016_paper_9
|storemode=property
|title=Introducing the Sky and the Social Eye
|pdfUrl=https://ceur-ws.org/Vol-1739/MediaEval_2016_paper_9.pdf
|volume=Vol-1739
|dblpUrl=https://dblp.org/rec/conf/mediaeval/WoodleyGNC16
}}
==Introducing the Sky and the Social Eye==
https://ceur-ws.org/Vol-1739/MediaEval_2016_paper_9.pdf
Introducing the Sky and the Social Eye
Alan Woodley Shlomo Geva
Queensland University of Technology Queensland University of Technology
Brisbane, Australia Brisbane, Australia
a.woodley@ qut.edu.au s.geva@qut.edu.au
Richi Nayak Timothy Chappell
Queensland University of Technology Queensland University of Technology
Brisbane, Australia Brisbane, Australia
r.nayak@qut.edu.au timothy.chappell@qut.edu.a
ABSTRACT • Is relatively inexpensive compared with field visits; and
We introduce the Sky and the Social Eye task which was run for • Allows for a large spatiotemporal area to be investigated.
the first time as a Grand Challenge at the 2016 ACM Multimedia
Conference and as a Task Force at the 2016 MediaEval Despite this, satellite data also has limitations, for example:
Workshop. Participants combined satellite images with social
media to create a richer user experience. For the first year, the task • Satellites tend to have a low temporal frequency (for
was exploratory allowing participants to produce their own example fortnightly) and so may miss an important event
system without traditional constraints such as shared datasets or such as a flood or forest fire; and
metrics. Here, we describe the task, summarize the participants’ • Clouds are present in some events, such as floods and
approaches and propose some future directions. snowstorms, obscuring the ability of satellites to capture
images.
1. INTRODUCTION
Remote sensing data, such as satellite images, have been used to Examples of a satellite image is presented in Figure 1 [13]. Here,
explore environmental and social challenges for decades. For two images are provided. Both images were taken in Sri Lanka
example, NASA’s Landsat program has been used to identify the and show the impact of floods. The first image was taken on
extent of forest fires, map land use change, assess carbon stock March 21st 2016 was taken prior to the flooding, while the second
and analyze reef water quality [5]. Increasingly, social media is image was taken on May 31st 2016 after flooding. The second
also being used for similar purposes, for example, mapping the image shows enlarged waterways from the flooding as well as the
extent of damage from natural disasters such as floods or presence of cloud cover, which obscures the land below.
earthquakes [4; 10; 11]. Despite this, there is a lack of research
In contrast, social media is continuously being generated and is
that explores how remote sensing data and social media can be
unencumbered by issues such as cloud cover. Based on this, social
combined.
media is increasingly being used to address problems that were
The Sky and the Social Eye task was proposed to fill this gap by traditionally addressed by satellite images analysis [4; 10; 11].
using social media to enrich satellite images. It ran as a Grand The Sky and the Social Eye task extends this research by linking
Challenge at the Association of Computing Machinery (ACM) satellite images and social media, such as text, images and videos
Multimedia Conference and as a Task Force at the MediaEval
Workshop for the first time in 2016. The aim of the task was for 3. PARTICPANTS
participants to develop algorithms and systems that combined Three groups participated in the inaugural Sky and the Social Eye.
satellite data with social media data. For the first year, the task Ahmad et al. [1] developed a system called JORD, which
was exploratory and creativity was encouraged. This allowed retrieved the names of events from the EM-DAT database [8]
participants to develop their own systems, using their own and searched Twitter, Flicker, YouTube and Google to retrieve
datasets, case studies and metrics. and fuse information about the event. They tested JORD with 80
Here, we describe the inaugural Sky and the Social Eye task. We events including floods, landslides, cyclones and wildfires.
begin by detailing the motivations for the task and then outline Bischke et al. [2] produced a system that fused satellite images
the approaches undertaken by the participants. We conclude by with Twitter data. They used a case study of a wildfire in Fort
discussing potential future directions for the task. McMurray, Canada. They collected Landsat 8 satellite images
from Amazon Web Services. Tweets, including text and other
2. MOTIVATION images, were collected from Twitter’s Historical Powertrack
There are strong advantages for using satellite images to explore
API. Then the tweets’ text was analyzed to identify geolocation
social and environmental events. For example, satellite data:
and sentiment and the tweets’ images were analyzed with a
• Provides strong and compelling evidence about events on the convolutional neural network to remove near-duplicates. Finally,
ground; the data were fused and presented as a visualization.
Copyright is held by the authors
MediaEval 2016, October 20–21, 2016, Hilversum, Netherlands.
Figure 1. Satellite images which highlight the impact of a flood in Sri Lanka.
Crandall et al. [6] used satellite data as (noisy) ground truth to Participants in the task combined social media with satellite
train two photo image classifiers: first, one that estimated if a images, based upon on particular events. Sky and the Social Eye
photo contained evidence of an event and second, one that was run for the first time in 2016 and the organizers plan on
aggregated the estimates to produce an observation for given transitioning it to a traditional MediaEval Task for future years.
times and places. Satellite images were sourced from NASA’s
Terra satellite while photos were sourced from Flickr’s public 6. ACKNOWLEDGMENT
API. Satellite images where classified into bins according to The organizers of the Sky and the Social Eye task would like to
their observed percentage of snow. These bins were then used to thank the task participants and reviewers as well as the organizers
train the photo classifiers using both support vector machine on of both the 2016 ACM Multimedia Conference and the 2016
tagged text and a convolutional neural network on the photos MediaEval Workshop.
themselves. Individual photos were then aggregated and a SVM
7. REFERENCES
trained over the aggregated data to estimate the probability of
[1] Ahmad, K., Riegler, M., Nguyen, D.T.D., Halvorsen, P.,
the actual environmental state. Finally, the results of the
Lux, M., and De Natale, F., 2016. JORD - Linking sky and
experiment were collected and presented as a visualization.
social multimedia data to events In Proceedings of the 23rd
4. FUTURE DIRECTIONS International Conference on Multimedia Modeling
(Reykjavik, Iceland, 4-6 January 2016).
The major future direction of Sky and the Social Eye will be to
transition it towards a traditional MediaEval Task. This will [2] Bischke, B., Borth, D., Schulze, C., and Dengel, A., 2016.
require the use of a shared set of documents, topics (queries) and Contextual enrichment of remote-sensed events with social
metrics, enabling participants to better compare their systems to media streams. In Proceedings of the ACM Multimedia
others. Moreover, the choice of the metrics is in itself a research Conference (Amsterdam, Netherlands 15-19 October
problem as the metrics need to consider both multiple data types 2016).
(that is: images and text) and draw together evaluations from the [3] Campbell, J.B. and Wynne, R.H., 2011. Introduction to
remote sensing community, whose metrics tend to be based on set Remote Sensing. Guilford Press.
retrieval (such as accuracy) [3], and the information retrieval
communities, whose metrics tend to be list based (such as Mean [4] Cervone, G., Sava, E., Huang, Q., Schnebele, E., Harrison,
Average Precision) [7]. J., and Waters, N., 2016. Using Twitter for tasking remote-
sensing data collection and damage assessment: 2013
In addition, the processing of remote sensing/social media on a Boulder flood case study. International Journal of Remote
cloud computing environment poses its own challenges. Sensing 37, 1, 100-124.
Traditionally, operational remote sensing analysis has been
[5] Cohen, W.B. and Goward, S.N., 2004. Landsat's Role in
processed on a local server environment, often requiring the use
Ecological Applications of Remote Sensing. BioScience 54,
of heavy sampling [14]. A cloud computer/supercomputer
6 (June 1, 2004), 535-545. DOI=
environment offers stronger computational capacity but has its
http://dx.doi.org/10.1641/0006-
own challenges, particularly regarding transfer of large datasets
3568(2004)054[0535:lrieao]2.0.co;2.
around the computing environment and the need to handle
streaming data – challenges which may not be solved by current [6] Crandall, D., Wang, J., and Korayem, M., 2016. Tracking
massive parallelization paradigm such as MapReduce or Mahoot natural events through social media and computer vision. In
[9; 12]. There is potential for the computer science community Proceedings of the ACM Multimedia Conference
to play a leading role in this area of research. (Amsterdam, Netherlands 15-19 October 2016).
[7] Croft, W.B., Metzler, D., and Strohman, T., 2010. Search
5. SUMMARY Egnines: Information Retrieval in Practice. Pearson,
Here, we have described the Sky and the Social Eye task which Washington D.C., United States.
was run as a Grand Challenge at the ACM Multimedia
Conference and as a Task Force at the MediaEval Workshop.
[8] Guha-Sapir, D., Below, R., and Hoyois, P., 2015. EM-DAT: based big data platform for massive remote sensing data
International disaster database. Catholic University of processing. In Proceedings of the Proceedings of the
Louvain. Second International Conference on Data Science (Sydney,
[9] Lv, Z., Hu, Y., Zhong, H., Wu, J., Li, B., and Zhao, H., Australia, 8-9 August 2015), Springer International
2010. Parallel K-means clustering of remote sensing Publishing, 120-126. DOI= http://dx.doi.org/10.1007/978-
images based on mapreduce. In Proceedings of the 3-319-24474-7_17.
Proceedings of the 2010 international conference on Web [13] United States Geological Survey, 2016. Landsat 8 Imagery
information systems and mining (Sanya, China2010), Reveals Heavy Flooding in Sri Lanka.
Springer-Verlag, 1927687, 162-170. http://remotesensing.usgs.gov/gallery/gallery.php?cat=2#52
[10] Middleton, S.E., Middleton, L., and Modafferi , S., 2014. 5.
Real-time crisis mapping of natural disasters using social [14] Wedderburn-Bisshop, G., Walls, J., Senarath, U., and
media. IEEE Intelligent Systems 29, 2, 9-17. Stewart, A., 2002. Methodology for mapping change in
[11] Schnebele, E., Cervone, G., Kumar, S., and Waters, N., woody landcover over Queensland from 1999 to 2001
2014. Real time estimation of the calgary floods using using Landsat ETM+. In Proceedings of the The 11th
limited remote sensing data. Water 6, 2, 381-398. Australasian Remote Sensing and Photogrammetry
Association Conference (Brisbane, Australia, September 2-
[12] Sun, Z., Chen, F., Chi, M., and Zhu, Y., 2015. A spark- 6 2002).