Whatever people produce on digital media can be a relevant source of knowledge and behavioural analysis. This is the subject of interest of a wide part of the new discipline known as Web Science. However, special care must be exercised when setting up studies on this kind of sources. Indeed, these studies rarely satisfy the established scienti c method guidelines, because of the nature and size of the data, as well as because of the bias and scarce generalizability of results. This paper identi es some of the most crucial challenges that need to be addressed when tackling knowledge extraction and data analysis out of observational studies on human-generated content.
The Web and the social media are today the environments where people post
their content, opinions, activities and resources. Therefore, a considerable amount
of user-generated content is generated every day for a wide variety of purposes
and related to diverse topics and contexts, with high frequency and speed [
Exploring and studying this content o ers a great opportunity to understand
the ever evolving modern society, in terms of topics of interest, events, and
people relations and behaviour. Several approaches based on the so called Big Data
paradigm have analyzed social media [
While the data available on this sources is typically created by individual entities, i.e. people or businesses, and pertain their own personal or social sphere, analyses are able to study individuals and also to capture the integrated, highlevel view of a more complex ecosystem or phenomenon, such as a city, a country, a community, or an event or social trend.
Therefore, we can say that the aim is to use appropriate tools for
understanding complex societal systems and dynamics. Such a tool was conceptualized back
in 1976 by De Rosnay [
In the remaining part of the paper I will present the ideal target of this kind of studies and some risks that are implied by the context and the data collected. 2
The Myth: Comprehensive, Coherent, and Cohesive Analyses The Big Data paradigm is pushing toward a scenario where people perceive the broad availability of large-scale storage, massive computational power, and exible and easy to use statistical and machine learning tools. This re ects in the expectation (both in individuals and businesses) to become able to obtain always and quickly any kind of insights they may need out of the existing data.
This implies that people expect that the executed analyses will always produce relevant insights for their needs. In particular, results are expected to be: { comprehensive, i.e., always providing a clear picture of the reality with all the relevant aspects, at all levels of details, from very ne-grained results to resilient and solid integrated results; { coherent, i.e., self-consistent in all their versions and dimensions, no matter what is the data source, the granularity of the analysis and the time/space considered; { cohesive, i.e., being able to convey a united and insightful understanding of reality, which can lead to informed and appropriate decision-making processes.
Is this the case? Or, better, is this an automatic outcome that we shall expect from any kind of analysis? Unfortunately, it's quite the opposite, as discussed in the next section. 3
The Challenges of Knowledge Extraction, Web Science, and Content Analytics In the majority of the analyses, if work is not conducted properly, risks and biases are not considered, and statistics is not applied correctly, results will su er of so many critical aspects that they will be hardly instrumental for any objective. This section has the purpose to introduce some awareness of the risk of big data and knowledge extraction. I don't aim at completeness, but just to report some of the most common challenges that a data scientist need to face when trying to extract knowledge and aggregated analytics out of usergenerated content. These challenges include the well known 4 V s of Big Data, i.e., Volume, Velocity, Variety, and Veracity, and some more. I report them in details below, together with some references to practical experiences. 3.1
Independently of the technique used for collecting and elaborating the data, and
actually also independently of the data itself, we need to keep into account that
reality is complex and varies in time, space and along many other dimensions,
including societal and economic variables. Aiming at capturing this complexity
in its entirety is a goal too ambitious for a single analysis. On the other side,
single aspects can be tackled, such as studying the continuously evolving nature
of knowledge [
On a more practical level, the data sources that are investigated are complex themselves, as they feature unstructured, multi-format, heterogeneous data. Tackling this variety is challenging per se, as each medium and type of content may need di erent collection, cleaning and analysis techniques. Furthermore, connecting together the extracted insights is challenging too. Finally, an additional complexity aspect is related to the structure of the data, which can feature record formats with data organized in deep hierarchies and relations.
In general, complexity may lead to harder understandability of the data. 3.3
As soon as the data is collected and analysed, an implicit level of bias is introduced. This is the bias of the observer, who is always in uencing the way the data is perceived, studied and processed, based on his preliminary knowledge, context, or visibility over the reality. This is also known as the street lamp e ect : if you look at a completely dark town, with only one lamp on, as an observer you tend to bias your understanding based only on what you see in the lit area. 3.4
The collected data is usually also biased with respect to the question that the analysis aims to respond. Established scienti c method de nes the way in which controlled experiments should be run and conclusions should be drawn. One crucial aspect that should be taken care of is to always apply an appropriate sampling method, granting that the sample is not biased with respect to the population the experiment want to assess. However, this is not so trivial to obtain. For instance, 1 shows two heatmaps representing the density of geolocated photos taken in Milano and shared on two di erent web sharing platforms, namely Instagram and Flickr: they both feature heavy bias with respect to the total population of the city, due to the mean used for capturing and sharing the photos (mobile app and web site respectively), the audience of the di erent platforms (casual smartphone users and photographers respectively): the two representations of the city are dramatically di erent, but there is no \correct" or \wrong" one: it depends on the kind of questions one wants to answer and on the kind of population one is interested in.
Furthermore, in data analysis and extraction from web sources, in many cases controlled experiments may not be implemented. Data scientists frequently need to resort to observational studies, which try to draw conclusions out of observations of the uncontrolled reality of facts and happenings. 3.5
A preliminary problem that many analyses may face is the di culty in nding and acquiring the data. In many cases, integrated data analysis requires availability of information that is now owned by the analyser himself. Therefore, the analyser need to rely on third parties to release and actually deliver the data. Such third parties may opt for releasing only part of the data, or only on a limited amount per time basis (throttling), or they can simply deliver limited or aggregated views on the data. Some strategies purposefully apply sampling or data limitations in a non-transparent way, so that the data consumer is not able to extract generalizable knowledge out of the analysis work. 3.6
Even when it is actually possible to collect the desired data, a further problem is about the quality of such data. An entire discipline is focused on capturing and solving data quality problems, which can span cases like dummy values, missing values, cryptic data, contradicting data, violation of business rules, data duplication, non-unique identi ers, and many more. 3.7
Granularity of data refers to the fact that the collected information may refer to di erent units of analysis on the time scale, space scale, and also along other dimensions. This is a critical issue, especially in the case of analysis on integrated data sources, where each source is based on its own granularity. In this case, techniques based on discretization and uniformation must be applied.
For instance, along the geographic positioning dimension di erent granularity
may mean that some data sources expose data about speci c venues/locations,
while others expose data about a geographical area de ned based on
administrative or political borders (e.g., cities, states, provinces, or countries), or on
xed geometrical shapes (squares or Voronoi tassellation); in this case, the
solution is usually to rely on a common grid structure. Analogously, along the time
dimensions some datasets may provide punctual information, while others may
describe di erent time periods, and thus the solution is to align the data on
common periods [
One (possibly obvious) concern regarding data is its volume: in web sources, the amount of data tend to grow enormously. This implies that any analysis needs to cope with the bandwidth, storage, and computation needs of big data. On the other hand, the challenge sometimes is just the opposite: the amount of collectable data is just not su cient to run sensible analyses. 3.9
One of the biggest issues with dealing with multiple data sources is that typically the collected data is not coherent: this implies that data cleaning and comparisons procedures must be run, to assess correlation and coherency of the sources. 3.10
The speed with which the content is produced may also be a concern, as it may become challenging to keep the pace with it in terms of data ingestion, storage and analysis speed. This problem is typically addressed by adopting data streaming solutions when possible. 3.11
Finally, the speed with which the output is produced is also relevant. Practical experiences show that in most analysis scenarios a strict real-time publishing is not requested. Careful curation of o -line, post-hoc analyses of the phenomenon at large is preferred instead. 4
In this paper I summarized the main expectations and actual challenges in modern data science approaches applied to web and user-generated content. The fact that only observational studies can be applied on web and social network contents is a further complexity factor in these studies. However, if proper data collection, aggregation, cleansing (wrangling) and analysis techniques are applied, these studies can extremely valuable results. Further examples, real cases and discussions can be found in the presentation of this speech, available online.1 1 https://marco-brambilla.com/2017/09/22/myths-and-challenges-in-knowledge -extraction-and-big-data-analysis/