The Web makes access to data of interest for disciplines such as geography, sociology, economics, or linguistics almost instantaneous, removing the barrier of lengthy and costly data collection. Using such data from the Web is problematic in terms of the validity, transparency, and reproducibility of the corresponding research, though, as little is known about the subject population and access to the data is often under control of private corporations. This paper discusses the implications of such research and points out potential solutions in the context of Linked Science.
Carsten Ke ler
Data collection has traditionally been the bottleneck of many research endeavors,
and gathering the data required to test a hypothesis may still take months, years,
or even decades. Physicists need to design and construct complex
instrumentation in order to prove the existence of subatomic particles, and astronomers may
even have to plan a mission into space to do their work. Other elds, however,
are bene ting from an abundance of data at their ngertips, often only a call
to a handful of API functions or web services away. Datasets collected through
social media platforms such as Facebook or Twitter and collaborative,
voluntary e orts such as Wikipedia or OpenStreetMap enable almost instantaneous
research across a number of elds. Likewise, the sensor web [
This discussion paper raises some of the issues that come with the access to those data sources and their use in scienti c research. It discusses the implications for validity of the corresponding studies and their reproducibility, and draws conclusions in the context of Linked Science. subjects can easily nd correlations that do not exist at a larger scale if the population investigated is not balanced with respect to the parameters under consideration. As an example, if the goal of a study is to see if there is a correlation between someone's age and the likeliness that they like red wine,1 the population under consideration must be balanced in terms of other personal and social attributes, such as gender, race, education, and income. If the sample population in our hypothetic study is not chosen appropriately, consisting largely of elder women and younger men, the results might indicate that there is a larger preference for red wine in elder people than in younger people, when in reality, it might be that women prefer red wine more often than men.
This ctional example shows that careful design of the sample population is crucial to come to valid conclusions.2 Research that draws heavily from social media data, however, is especially prone to this problem, because (a) too little is known about the participants in a study, and (b) the attributes known about the participants are very hard to verify. Moreover, di erent social media platforms are often used by user groups with di erent, but distinct, pro les. After all, social media is being used to socialize with peers, which will often have at least some demographic properties in common. It is hence di cult to use social media data for studies that are supposedly saying something about the general population. In reality, many of these studies are most likely really only saying something about the users of a speci c social media service; this is along the lines of the old joke that many psychology studies do not really say anything about the general public, but a lot about psychology students, as this is the main group taking part in their human participants tests. 3
Using social media data in research also entails problems for the reproducibility
of any studies. User pro les and the data available are in a constant ux, so
that it is virtually impossible to replicate a study with the same set of users.
While studies that test general statements about the user population of a service
(\Are Facebook users more inclined to conservative political positions than
Twitter users?") can be replicated in principle, this is only possible as long as the
service is available, makes the required information available through its API,
and maintains a large user base; all of these factors are outside of the control of
the investigator. Finally, data archiving is problematic because in their terms of
service, many social networks prohibit making local copies of their data obtained
through APIs. Even if such archiving is permitted, the large volume of the data
can make archiving di cult or at least expensive.
1 Clinite [
Using the Web as a Data Source: Challenges for Linked Science
Some recent studies have also raised transparency concerns. They have been
conducted by some social networks' in-house research teams who had access to
data that is not available to anybody outside of the company at that scale [
Openness, transparency, and reproducibility are core principles of Linked
Science [
A stricter enforcement of reproducibility and transparency principles for the acceptance of journal and conference submissions is required to solve this problem. The Linked Science principles of semantically annotating, interconnecting, and publishing scienti c resources show that the technologies for these processes are already there. E orts to develop executable papers that automatically perform the data analysis steps of a study show that the added value of providing these resources go beyond theoretical reproducibility|they actually reproduce the data analysis. Enforcing the publishing of these resources will also require legal certainty for researchers who work with data from private corporations. As their role as data providers for research is growing, legislation is needed that allows archiving of data obtained from their public APIs. The scienti c community hence needs to be more strict about its core principles, leveraging the opportunities o ered by technology-driven frameworks such as Linked Science, while the legal circumstances have to be adjusted to ensure transparency without risking lawsuits.