The work reported in this paper aims at describing a project that leverages the potential of Information Retrieval and Machine Learning towards novel techniques that unveil the latent states of expert users such as sociologists and economists by means of indicators when the user is accessing large collection of newspapers, blogs, etc. An indicator measures the degree to which a certain latent state is present during interaction when exploring and searching an information repository. In this paper the state of a user is the particular condition that s/he is in at a speci c context with reference to a problematic issue induced by the data s/he accessed to; risk is an example of state and a risk indicator aims at providing a measure of the degree to which articles examined by the user evoke risk in her/his mind. Observable attributes, e.g. keywords, clickthrough data or links, are the input data to model states and compute indicators. In this work, starting from some results of a software architecture designed to support sociologists in investigating Techno-scienti c Issues in the Public Sphere, we will discuss some challenges and we will present a formal framework to address them where informative objects, e.g. news articles, states and attributes are uniformly modelled as vectors as it is customary in Information Retrieval or Machine Learning. This is our rst step towards a long term objective, i.e. generalizing the well known Learning to Rank framework towards a Learning to Search framework which would encompass multiple and simultaneous states.
Although Social Computing (SC) has a long history and dates back to, for
example, Sadowski's work [
Not only the amount of data is rapidly increasing, the variety of types of data has also become larger and the degree of user interaction may be higher than in the past. The typology of data that are accessed by social scientists, students or causal users such as World Wide Web (WWW) surfers includes structured data extracted from relational databases, semi-structured data received from eXtended Markup Language (XML)-encoded streams such as news feed, or unstructured data collected, indexed and delivered to end users by search engines as inter-linked pages. The di erent data management systems mentioned above can provide a variety of services to the end users who may search, browse and annotate text, images, video and music.
Together with users' behaviour data (e.g. click-through data or search session), these systems can be utilised to organize and analyse the users' thoughts and feelings about di erent controversial topics. For example, the vast and complex arrays of data such as social media can thus be utilized to create representations of techno-scienti c controversies which are able to trigger intense public debates such as those concerning issues like climate change, Genetically Modi ed Organisms (GMOs), nuclear power.
When accessing to information by reading, browsing or annotating
multimedia documents, the end users form a view or judgement about something,
not necessarily based on fact or evidence, on the contrary, based on general
feeling or opinion. These situations not only clearly blur the traditional
boundaries among expertise, policy making, politics, and public opinion [
In this paper, the state of a user is therefore the particular condition that s/he
is in at a speci c context with reference to a problematic issue documented by the
data s/he accessed to. An example of user state is the condition that the user is
in with reference to themes related to the \risk society". The risk society refers to
the idea that within contemporary society Science and Technology (S&T) issues
are imbued with fears and preoccupations about unforeseen e ects, calling for a
precautionary approach on the side of policy making, society, and the public [
Note that understanding the user's state is di erent from the understanding the document's author opinion addressed by means of Sentiment and Opinion Mining and Analysis systems which aim to classify the opinion of a document's author who has implicitly expressed his opinion by means of a document. The user's state is not necessarily encoded in words, on the contrary, it might be understood by analysing di erent types of data ranging from click-through data to natural language queries in combination with document attributes. Moreover, our focus is not on users' mood that may be mined from short or tiny data such as tweets or \likes". Our interest is on readers who might not immediately comment and reveal their own feeling to the public, which is nevertheless of great interest to expert users such as sociologists or economists who are asked to be acutely aware of the issues of our world.
This paper brie y reports on our advances in using SC techniques based on Information Retrieval (IR) and Machine Learning (ML) to support sociologists in investigating Techno-scienti c Issues in the Public Sphere (TIPS). Starting from these initial results, it is our objective to further develop the framework and the TIPS software architecture to unveil the latent states that a user may experience when interacting with news articles that deal with techno-scienti c controversies. In this context, IR may play a crucial role because it is naturally devoted to meet user informative needs stemming from the problem of understanding social phenomena, when these phenomena are encoded in large collections of interlinked documents such as news, feeds, blogs, video and images.
Besides extending TIPS, we have got a long term vision. Quoting Liu [
For some years, a project called TIPS3 has been carried out by us to develop, experiment, and implement automatic procedures for collecting, classifying, and analysing digital contents { mainly online news and user generated contents in social media { in order to monitor S&T issues and their evolution. TIPS collects online articles from six newspapers and classi es them according to their pertinence to S&T topics; currently the corpus is constituted of more than a million documents, collected since 2010. Fig. 1 reports an overview of the architecture components.
TIPS collects articles from online news such as RSS feeds associated to speci c newspaper sections through a collector module. The articles are classi ed and a risk indicator is calculated to measure the degree to which the articles 3 http://hal.cloud.tilaa.com/tips/ informative objects sources collector module classifiers and indicators indexing module
IR technology
user interface S&T Classifier Risk Indicator Computation
S&T classifier score S&T classifier label
Risk indicator evoke risk in the users' mind. The articles are then enriched with the categories associated by the classi ers and the indicators. Furthermore, the articles are provided to an indexing module that stores the article attributes in a data repository that provides e cient access though meta-data and content-based search using IR technologies4. Additional layers can then be built on top of the IR module to provide meta-data and content-based access through a WWW user interface and to display dynamic charts representing indicator trends (Fig. 1).
The remainder of this section brie y describes how the risk indicator is
currently instantiated in the current release of TIPS. A computer scientist may
nd the procedures implemented by this system rather rudimentary, however,
the manual intervention is limited and it was required by the sociologists who
wished to closely monitor the procedures as possible. A risk indicator relies on
a set of keywords manually identi ed by the sociologists through the support
of an unsupervised ML algorithm for the extraction of themes from an
unstructured corpus. The keywords were selected by (i) retrieving documents through
the query \risk", (ii) extracting topics and subtopics through an implementation
of the HPAM topic modelling algorithm [
An indicator described by the set of keywords K for the document set D, will be computed as
IK(D) = 1 X jDj d2D
IK(d)
IK(d) =
1 X nL(w; d)=B
jKj w2K nL(w; d)=B + K
(1)
(2)
where
and nL(w; d) is the frequency of the term w in the document d; nL(w; d) is
normalized by B: (1 b) + b avdgld(dl()C) where dl(d) is the length of the document d and
avgdl(C) is the average document length in the corpus C; b 2 [0; 1] is a
parameter that controls the weight assigned to the document length normalization. The
nL(w; d)=B normalization has been introduced in [
In order to show a possible application of a risk indicator to actual news, in the remainder of this section we will report on the trends obtained for the risk indicator using an Support Vector Machine (SVM)-based classi er to identify S&T news articles. We considered a subset of articles collected by TIPS and published from January 1st, 2010 to December 31, 2015; the total number of articles constituting this subset is 630,549. A SVM-based classi er with linear kernel was trained on a sample of the news corpus that was manually labelled by a team of sociologists on the basis of their pertinence to S&T issues. The sample is constituted of 3817 documents; 1,393 were labelled as pertinent to S&T, while the remaining 2,424 as non pertinent. The e ectiveness of the classi er was tested using a 60/40 split for training/test with 10 fold cross-validation. We then classi ed all the 630,549 articles using the learned classi er. The risk indicator was then computed for all the articles, for the subset of articles classi ed as pertinent and for those classi ed as not-pertinent. The evolution of the risk indicator on a monthly basis and in the time frame 2013-2015 is reported in Fig. 2; since the time granularity is one month, the document set D in Equation 1 for the lth time interval til, denoted by Dtil , is constituted by all the articles published in the lth month of the time frame 2013-2015; in the event of relevant and not-relevant trends, the document set is respectively constituted of the relevant and the not relevant documents published in the time interval til.
One of the limitations due to using only content based descriptors is evident in Fig. 2: the trend of the risk indicator in the entire corpus di ers from that in the subset of relevant articles, thus suggesting that the distribution of terms in risky documents can di er from that in relevant documents; therefore, frequency might be no longer the best evidence to capture the degree of risk. Relevant
Not Relevant
All 0.06 0.05
Fig. 3 reports the variation of the risk indicator when computed for documents about \climate change" and documents about \nuclear power" for the subset of the news articles collected in TIPS and published from 2013 to 2015; also in this case the distribution of risky terms in the two subsets is di erent. However, even if the current indicator is able to capture a di erence in terms of perception of risk in \climate change" and \nuclear power" related articles, the indicator is unable to explain why such di erence exists. A more suitable representation of risk should be able to assists the users, e.g. specialists, also in this task. The framework introduced in the remainder of this paper aims to achieve such a goal through an explicit representation of states. 3
TIPS has been the starting point of the contribution described in this paper. Since the early phases of the TIPS project, we realized that, beyond the sociologists' requirements, there is a great potential and some challenges of SC { mentioned in Section 1 { can be met if some current limitations of TIPS can be overcome. The main limitations of TIPS that are considered in this paper are the following ones: Single state. Only one state (e.g. risk) can be at a time detected and measured from the information objects provided as input. The source software in principle is able to manage diverse states, but states are modelled as independent each Climate Change
Nuclear Power 0,06 0,05 0,04 R O T A ICD0,03 N I K S I R 0,02 0,01 0 other. Moreover, some manual intervention is required to tailor the system to manage a speci c state. The manual intervention to be made consists of compiling controlled keyword vocabularies such as the vocabulary of keywords evoking risk. Another state, e.g. \con ict" will require the compilation of another, distinct keyword list, and a group of experts in con icts would be in charge of this compilation.
Content-based state modelling. The current implementation of TIPS only
exploits content-based descriptors such as keywords, dates, and class labels.
Because of the type of descriptors selected during system design, we have regarded
risk as similar to relevance5 and we used a normalized term frequency that
worked well in general content-based IR [
However, in the light of detecting risk or other states, the current computation of indicators soon appeared rather simplistic. Normalized term frequency is not necessarily the best method to detect and measure risk and in general states other than relevance. For example, the distribution of terms in risky documents can di er from that in relevant documents and frequency might be no longer the best evidence to capture the degree of risk and actual interaction with the end user is ignored. 5 A document is relevant to an information need when it carries information that help the user to meet his/her need
In contrast, the users' perception of risk may be better captured by the user
behaviour during the interaction with the articles. Since an article might not
include \risk" although it could evoke risk, or may include the word while not
evoking the feeling, it is the complex of the users queries, click-through data,
and other interaction features that may suggest the \riskiness" of the articles.
Much information about the user's information need can be obtained through
interaction [
No simulation-based indicators. TIPS has been mainly designed for monitoring purposes by using IR and ML technologies. However, when studying the relationship between techno-scienti c issues and the public's opinion, some tasks could bene t from the possibility to simulate speci c scenarios, e.g. obtained by varying the degree to which news are \imbued" of risk.
In the current implementation, if the side e ects of new scenarios were to be investigated, we should either build suitable synthetic documents or collect actual documents, and provide those documents as input to the pipeline constituted of the modules depicted in Fig. 1. However, some steps require manual intervention, which might be cumbersome or even impossible for specialists. For example, if a sociologist wanted to investigate how perception of risk will evolve if news about nuclear accidents were broadcasted, s/he should wait for actual news or generate synthetic news lled with keywords about nuclear accidents. Actual news documents would be di cult to assess because of the assessors' e ort required to label a training set that is large enough to train TIPS. Although synthetic documents may in principle be generated about a topic that is traditionally perceived as source of risk, Natural Language Processing (NLP) technologies cannot be reliably utilized to generate these documents that a human expert can express a genuine perception of risk when s/he is asked to assess risk. If such technology existed, this challenge would be solved. 4
In this paper, we address the challenges illustrated above and emerged from the TIPS project, i.e. the limitations that only one state can be investigated at a time, only informative content-based evidence is utilised to implement a state, and the impossibility of making prediction and simulation of what would happen to states when evidence will change. To the aim of facing these challenges, we introduce a vector-based formalism describing the main concepts of TIPS. The formalism is given in terms of de nitions illustrated below.
De nition 1 (Information object). An information object is any data container provided with an identi er.
An information object is provided with an identi er to allow applications to connect the object, whereas search is based on the object's content. Examples of information objects are webpages, individual images, videos, music les, query sessions, click-through data, and other user behaviour data. In this paper, an information object is symbolized by lower-case y and de ned as a vector of the k-dimensional real space, since a component of y is a real number. De nition 2 (Attribute). An attribute is what we can directly observe from information objects such as documents or user interaction actions. Examples of attributes are informative content descriptors (e.g. keywords and terms), intra- and inter-object links, link anchors, meta-data, annotations, or tweets mentioning the information object (e.g. the article). The real components of a vector in Rd correspond to the attributes observed to measure an object. To obtain a complete formalism, in this paper, an attribute is symbolized by lower-case v and de ned as a vector of the k-dimensional real space, since a component of v is a real number. For example, an attribute component may refer to a term frequency, a click occurrence, a colour code or an encoded sound fragment frequency. A set of independent attribute vectors form a vector basis and therefore any linear combination thereof is a vector of the same space. For example, the j-th basis vector of the k-dimensional space has 1 at component j and 0 elsewhere. An attribute may occur in an information object to a certain degree. Therefore, we have to introduce the following de nition. De nition 3 (Weight). A weight is the degree to which an attribute is present in an informative object.
A weight is thus a real number. In particular, we formalize an attribute as a vector of the k-dimensional real space. It is assumed an independence relationship between the attribute vectors, so that it is possible to formalize an information object as a linear combination of attribute vectors. If v1; : : : ; vn are n attribute vectors, an information object vector can be written as x = a1v1 + + anvn (3) where n is the number of attributes used to represent objects (e.g. the number of keywords) and the a's are the attribute vector weights measuring the degree to which an attribute describes an information object. Questions about a v can be answered when the vectorial representation y of a user is matched against x, thus obtaining the corresponding a.
The main question is how can we model states? Contrary to attributes { which are manifest { states can only be indirectly { they are latent { observed from information objects.
De nition 4 (State). A state is a latent characteristic of a user when interacting with an information object.
The main thrust is that a state refers to both an information object and a user. As a state is a latent characteristic of an object-user pair, some attributes have to be observed to make the state explicit. However rich the description of information objects can be in terms of attributes, some hypotheses about the user's state when s/he is interacting with objects can be explained only if the latent, unobserved states can be explicitly modelled, thus allowing to predict and simulate how states can evolve when the information objects { user interaction included { are observed. In this paper, a state is symbolized by lower-case z and de ned as a vector of the k-dimensional real space, since a component of z is a real number. A state may occur in an information object to a certain degree. Therefore, we have to introduce the following de nition.
De nition 5 (Indicator). An indicator is the degree to which a state is present in an informative object.
An indicator is thus a real number too. As objects, states and attributes are both placed in the same vector space, it is possible to represent an information object as a linear combination of m state vectors as follows: x = b1z1 + + bmzm (4) where m is the number of states, the z's are state vectors and the b's are indicators. Eq. (4) enables to compute each indicator using linear algebra operations. Thus, questions about a z can be answered when the vectorial representation y of a user is matched against x, thus obtaining the corresponding b.
The formalism de ned above stems from the theory of abstract vector spaces
widely adopted in ML and IR and speci cally in Learning to Rank (LETOR),
which is based on vector-based attribute (also known as feature) spaces and
discriminative learning [
Instead of depicting vectors using the usual arrows in a three-dimensional plot, we exploit the visual paradigm adopted in the current implementation of TIPS. Consider Fig. 4 which gives a pictorial description of the formalism mentioned above. First of all, there is a temporal axis along which states evolve as curves. Each state corresponds to a curve; \risk" corresponds to the red curve and \con ict" corresponds to the blue curve. The y-axis refers the indicator values; for example, the value of the indicator of con ict is 0.32 when the information objects are those observed on June 2014. The indicator value is depicted as a bullet placed on the curve. It is the result of a computational process that takes attribute weights as input.
Consider a scenario where a specialist user, e.g. a sociologist, working on the e ect of S&T on the society and how the society a ects the progress in S&T. Suppose that the sociologist's task is to study and comprehend the public perception of \climate change" in the last two decades. A possible sociologist's research question is: how have the perception of risk related to \climate change" been changed in the last two years? In order to carry out this investigation, a 0.75 tr o ica 0.5 d n I 0.25
Conflict State
Risk State
Conflict Indicator (z1, b1=0.32) 0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 201 201 31 31 31 31 31 31 31 31 31 31 31 31 41 14 14 14 14 14 14 14 14 14 14 14 15 15 15 15 15 15 15 15 15 15 5 5 -01 -02 -03 -40 -50 -60 -70 -80 -90 -01 -11 -21 -10 -20 -30 -40 -50 -60 -70 -80 -90 -01 -11 -21 -10 -20 -30 -40 -50 -60 -70 -80 -90 -01 -11 -21
Time
states indicators
attribute weights
informative objects
possible source for information are articles published in newspapers; this is, for
instance, the approach carried out in [
A parallel can be drawn between the notion of state and that of category of a classi cation system. However, state (e.g. risk) and category (e.g. nuclear energy) are di erent each other. Although the pertinence of a news to a category may be viewed latent, once the classi cation have been performed the fact that an article is pertinent to the category can be used as additional attribute to characterize the articles.
In contrast, a state, e.g. risk is that it is not directly observable from a document. Indeed, we are not considering articles about risk or on the notion of risk, but articles that are imbued or evoke risk. It is a latent state that can be evoked because of some words or combination of words occurring in the article that can trigger other issues or images related to risk, or because of two di erent viewpoints are presented with the underlying purpose of discrediting one of them. The risk indicator is the degree to which the risk state is present in the userobject pair or in the interaction between the user and a set of documents.
Another di erence between category and state is that a state lies in user interaction and it is not a static feature of an information object { a category may be viewed a static feature indeed { and does not change without changing the object content. Instead, state may change. A user who is reading news about nuclear energy may be or not be in a risk state depending on the personal or social context in which s/he interacting with information objects. The same apply to states other than risk such as con ict or economic crisis.
A similarity between states and categories (or classes) is multiplicity. Similarly to the simultaneous membership of an object to di erent classes, multiple states such as risk, con ict or economic crisis can be latent in the same objectuser pair. Eq. (4) does indeed express the multiplicity and simultaneity of states in object-user pairs, where the z's can be adapted to the user's interaction. One of these z's may refer to relevance and the indicator b thereof may measure the degree to which x is relevant. Similarly, another z may refer to risk and the indicator b thereof may measure the degree to which x evokes risk. Thus, we have
x = brelevancezrelevance + briskzrisk when an object x is represented in terms of latent states or x = aa term frequencyva term frequency + aanother term frequencyvanother term frequency + aclick frequencyvclick frequency
The multiplicity of simultaneous states in an object requires a shift from the
current state-of-the-art IR technologies based on LETOR to a novel paradigm
called LETS. When only one state { relevance is the most important one in
IR { is considered, ranking is the natural task that has to be automatized and
LETOR is an appropriate approach to relevance-based document ranking,
especially if applied to the WWW. The application to domains other than the
WWW requires to model states other than and in parallel to relevance.
Therefore, approaches other than LETOR may be useful if not necessary, since the
users might no longer be casual users and the application domain might not be
or only be about webpages to be ranked against relevance. In these contexts,
ranked document lists may no longer be the most appropriate output as
witnessed by the experience learned from TIPS. When the ranked list is no longer
the desired output to answer one single state, LETOR needs to be re ned and
the change of the output space { multiple, simultaneous states { will naturally
lead to the change of the hypothesis space { the space of functions { and of
the loss function, as well as the change of the learning theory. The new search
scenario may be decomposed into several sub ranking tasks { corresponding to
the multiple simultaneous states although many key components in learning to
rank can still be used. [
true relevance true relevance
? - relevance predict-ed
state predictor relevance (a) LETOR framework attribut-es true states ? relevance predictor { there is one state, i.e. relevance { fueled by di erent attributes (or features). The LETS framework (Fig. 5(b)) instead includes more than one predictor, one predictor for each state. The prediction of these states may be simultaneous, thus requiring parallel optimizations and predictions. 5
As the proposed project is interdisciplinary across sociology and computer
science, contributions both in computational and social sciences are relevant to it.
In [
Relevant contributions in modelling informative content were proposed in IR.
In IR, Query Expansion (QE) techniques modify the initial query formulation
by extracting from documents relevant (or assumed to be relevant) to the
considered query additional descriptors to obtain a more e ective information need
representation. Many of these techniques are surveyed in [
Recent works focused on temporal Web dynamics and its application to IR, but they are mainly focused on Web user behaviour dynamics, on changing individual document content, or on the variation of the single term collection frequency over time. The TREC Knowledge-Based Acceleration track considers a time ordered corpus; however, the task is ltering documents that would change the pro le of people and organizations, and the list of entities is prede ned - this project is not restricted to entity types.
Topic Models (TM) aim at automatically discovering the main \themes" in
a document corpus. A well known TM is Latent Dirichlet Allocation (LDA)
where documents are modelled as a distribution over a shared set of topics,
which are themselves distributions over words generated by one of these topics.
LDA assumes a xed number of topics and the probability of seeing a topic is
independent over time - in contrast, we will also address time. These issues are
addressed in [
This project focuses on techno-scienti c controversies, i.e. issues able to
trigger intense public debates, even in the case they address techno-scienti c
discussions such as those on climate change, GMOs, cloning, nuclear power merge
and blur the traditional boundaries among expertise, policy making, politics,
and public opinion [
TIPS has been the starting point of an inter-disciplinary research project funded with the aim of providing expert users such as sociologists and economists with an e ective and e cient system to investigate techno-scienti c issues. Starting from this paper, we will pursue this objective and will also make a contribution at the level of methodology and system evaluation along two main directions.
We will address the problems related to the scenario simulation mentioned in Section 3 and will design, implement and evaluate methods for interacting with the representation of multiple and simultaneous states, e.g. risk and con ict. These methods will allow us to create scenarios by operating directly on state representations, thus avoiding the need to apply the entire pipeline for each scenario under investigation. To this end we will de ne a set of algebraic operators and implementation thereof within the functional scheme depicted in Fig. 5(b).
Besides explicitly considering interaction data when modelling states through ML algorithms, we will integrate interaction data to provide the user with information on the degree of a state present in the set of documents examined in the last sessions and how this degree di ers from the \state distribution" in the overall corpus. In other words, interaction data can signal the tendency of a particular user to explore, say risky documents when performing a task or accomplishing a speci c information goal. This signal can motivate the user to explore additional parts of the informative space in order to form his/her opinion on the issue and be \less subject" to the way the issue is presented in some venues | e.g. a particular set of newspapers or blogs.