Large parts of scientific work relies on seeking for information in very large datasets and respective metadata (e.g., document repositories on the web, databases, local image collections). Based on a search string or even sample data as a query, information retrieval systems (IR systems) return lists of ranked items that match the query, together with a short preview of the item. Using search strings or example data, it is not easy to express certain information needs, however. In this extended abstract we discuss in what way the interaction of a user with an information retrieval (IR) system can optimized with human-aware collaborative planning strategies.
Information seeking is a process at where humans search for information in, for instance, documents. As depicted in Figure 1, information seeking (IS) processes can be further diferentiated. In case of single-step ad-hoc retrieval, every query is executed independently, which is okay if information retrieval (IR) systems return relevant documents in a single step most of the time. If this is not the case, IR systems can improve IR performance when it is known whether the user, i.e., a human, has a purposeful (telic) goal. If no, the system might return documents the human might not expect to be retrieved in the first place. For instance, in searching for news articles a human might not have a particular goal. In case there is a telic goal, the problem is to appropriately specify the respective information need to ensure decent precision values for query answers, in particular when datasets are to be accessed (and not only document repositories).
As part of our research context we consider information needs of humanities scholars, seeking for specific information, e.g., to prepare expert testimonies about certain artifacts and their materials. Not only for materials data, researchers need comprehensive information about underlying datasets to express a one-step query that enables the system to return datasets and documents with appropriate information for the task under consideration. However, detailed knowledge about datasets (and respective documents) is usually not available, such that multiple queries will be required. An IR system could improve its performance by observing the changes
Yes
No Ad-hoc Retrieval
Information Seeking (IS)
One-Step?
Yes Optimized?
No Telic goals?
Yes Interactive IR (IIR)
Dynamic Search (DS)
No Exploratory Search (ES) of the issued queries over time and might be able to approximately represent and the scholar’s (telic) long-term goal, and consider it for IR query answering tasks.
Queries issued by a human as well as the respective sets of items returned as answers can
be combined to form a so-called session, ending when the information need of the human is
satisfied. The IR system could compare each query with its predecessor in a session. Depending
on which words are added and removed, the system could improve its performance (interactive
information retrieval (IIR), see Figure 1). This still might not lead to optimal results as the
system does not reflect that the human possibly has a long-term goal. Given a set of queries
part of one session, an IR system has to estimate what the long-term goal of the human might
be (dynamic search (DS), cf. Figure 1, see also [
In this extended abstract, we discuss the basic design of an IR agent that is equipped with an IR goal, perceives its environment through various metrics (sensors), builds a set of models (online/ofline) of itself as well as an approximate one of the human, and then uses these models to select (actions) sets of datasets and documents in order to fulfill the IR goals.
The agent can only satisfy the information need of the human if it can anticipate his IR goal. To anticipate his IR goal, it requires to collaboratively work with him. Collaboration starts at where the human expresses his information need as a query. The human sends the query to the agent and the agent then updates its model that contains an approximation of the IR goal of the human. The agent then sends a result to the human, depending on its updated model. Depending on the result, the human sends a reformulated query and optionally feedback to the agent. The feedback helps the agent to close the gap between its model and the IR goal of the human. This process repeats until the information need of the human is satisfied. We refer to the whole process to as a session that ends if the agent has anticipated the IR goal of the human and thus is able to satisfy the information need of the human.
Many diferent kind of problems can occur during the session, as the IR agent and the human have to collaborate with each other via queries, feedback, and results without being able to physically observe each other. If a human expresses a query, then he has an approximate mental model of the IR agent, which has an influence on of how the human expresses his query. For Human 1 2× 2 Agent
A0
A1
A2 ∙ ∙ ∙
A instance, the human expects that his query needs to be in a specific foreign language, the IR agent only compares the query with the titles of documents, or the IR agent weights some terms part of the query to high such that the human omits them. However, the human might be wrong with his expectations. The IR agent is aware of that the human has specific expectations of itself and aims to anticipate them. If the IR agent correctly anticipates them, then it might be able to adapt its behaviour. Adapting the behaviour needs to not only satisfy the expectations of the human. In addition, the behaviour of the IR agent needs to be explicable from the perspective of the human. Otherwise the human is not able to have an accurate approximate mental model of the IR agent and has dificulties in expressing a query. The IR agent cannot correctly anticipate, adapt, and act explicable for every human of who it collaboratively seeks for informations and is aware of that. In case the IR agent in uncertain whether it can collaborate with the human or the human explicitly requests an explanation it has to explain its behaviour.
More formally, an IR agent and a human interact with each other over a finite number of
time steps in a session, with ∈ N0. As proposed by Kambhampati et al. [
At each time step during a session, the IR agent’s A and human’s mental models H are in a specific state that change with each time step . Thus a session is a sequence of session states = ()=0, where each session state contains the current query , feedback the human expresses, given results − 1 the IR agent has send to the human in the previous session state, result that might satisfy the information need of the human, and updated mental models H of the human and A of the agent.
A session at time step is successful if in satisfies the information need of the human. Each session starts with initial, possibly non-empty, models H0 and A0 at time step = 0. Then, at the next time steps ≥ 1, the IR agent first updates its models A− 1, given query and optionally available feedback : and then selects result , depending on A: The human receives and updates his models:
A− 1 × × →−
A A →−
H− 1 × →−
H H →− and then expresses his information need as query +1 and feedback +1:
Session helps the agent to close the gap between the human models H and its own models
A over time and thus to anticipate the IR goal of the human.
3. Representation of Mental Models for Collaboration
As noted by Kambhampati et al., collaboration between a human and an agent works only, if
both have an approximate mental model of each other with a small gap in between [
The most fitting evaluation method, which we find in the literature, was part of the dynamic
domain TREC 2017 conference [
Our aim is to extend the human simulator, part of the evaluation at the dynamic domain TREC 2017 conference, with a human agent, such that the IR agent can improve its performance by approximating ℳhA as ℳ̃︂hA’ in addition to approximating ℳH as ℳ̃︂aH. The human simulator still has access to a ground truth dataset, however the dataset does only contain sets of relevant documents, each with respect to a session. Expressing queries is now the task of the human agent that does not know which documents are selected as relevant by the human simulator. From a subset of these as relevant selected documents, the human simulator generates a subjective representation in form of weighted words that simulate the information need of a human. Subjective as the human simulator generates the weighted words depending on the context of as relevant selected documents. The human agent does not know which documents are selected as being relevant by the human simulator and is not able to go through all documents in the corpus. However, the human agent can observe the weighted words that represent the information need. That is identical to a real human that is not able to compare all documents in a large corpus with a set of terms in his mind. From the set of weighted words, the human agent has to express its information need as a query. As it is for real humans the case, the human agent tries to anticipate of how the IR agent returns documents, given a query, as it only wants to retrieve documents, that are relevant. The human agent is allowed to reweight the weighted words before sending them as a query to the IR agent to retrieve a set of documents it could read. Only sending top weighted words to the IR agent, without adjusting them, is not a good strategy as the IR agent might not be able to always return relevant documents with respect to a subset of words that only subjectively represent the information need of a human. Therefore, the human agent knows that the IR agent might not be able to understand his information need and needs to find a good strategy for expressing a query. The IR agent needs to anticipate of how the human agent reweights words before sending them as a query, while it does not have access to the set of weighted words. However, it can observe the titles and contents of all documents in the corpus and the query, send by the human agent. We argue that the IR agent needs to be aware of that its actions influence the human agents actions. If the human agent reads a document, then the human simulator changes the weighted words, as the human agent has learned something new by reading a document. Collaboration between both agents only works if both aim to share and approximate the mental models of each other.
More concretely, the human agent does not compute +1 × +1 from H and the IR agent from A directly. Both H and A contain models and each model contains a function that maps, depending on the model, queries and feedback or weighted words to queries +1 and feedback +1 or weighted words, as depicted in Table 1. The human agent needs to express its information need as a query and to give optionally feedback for previously received results: H → +1 × +1, with H = (ℳH, ℳhA). Model ℳH contains a function H→A : ℳhA × H → +1 × +1 that maps weighted words H, generated by the human simulator to query +1 and feedback +1, and ℳhA an approximated function
hA→H : × → hA of the IR agent that maps and to a set of weighted words hA possibly part of model ℳ̃︂A from the perspective of the human. An approximation of hA→H is the humans understanding of how the IR agent interprets a query and feedback. In addition, it sends feedback to the agent, containing which documents the agent has returned in the previous iteration are relevant. Thus, the human agent is aware of that the set of weighted words H in his mental model ℳH only subjectively represent the relevant documents and the IR agent is not necessarily able to return relevant documents from these words. In addition it knows, that the IR agent is aware of that and aims to reweight the words in with hA→H : × → hA that objectively represent the information need of the human agent, from the perspective of the IR agent.
The IR agent needs to assign a score to all documents in the corpus and return top of them in descending order as : A → with A = ( ℳ̃︂A, ℳ̃︂aH, ℳ̃︂hA’). Model ℳ̃︂A contains the function
˜A→︂H : ℳ̃︂aH × × → A where the set of words A objectively represent the information need of the human agent, ℳ̃︂aH an approximation of ℳH containing the function and ℳ̃︂hA’ the function ˜aH→︂A : ℳ̃︂hA’ × aH → +1 × +1
˜hA’→︂H : × → hA’ The weighted words A objectively represent the information need of the human agent, from the perspective of the IR agent and it compares these with the documents in the corpus, by using for instance latent semantic indexing (LSI). Approximating hA→H : × → hA as ˜hA’→︂H : × → hA’ helps the agent to act according to the expectations of the human, by comparing hA’ with A. If the gab between hA’ and A is too large, the IR agent could either adapt its behaviour to act according to the expectations of the human, even if it is a loss in its retrieval performance or if the gab is even larger, then it should explain its behaviour. Note that hA’ and A are not necessarily identical.
Even if the human agent and the human simulator are not simulating a real human perfectly, we argue that it is a big step towards developing a human aware IR agent that collaboratively seeks for information together with a real human. The IR agent could be further improved in the future by evaluating it in the real world. If the IR agent performs bad in the real-world, then the human agent and human simulator need to be adapted accordingly. The efort of adapting the agents pays of, if the agents are realistic enough for the evaluation of an IR agent. Evaluating an IR agent, using a simulator is cheaper and orders of magnitudes faster, than evaluating it in the real-world.
During seeking, the human agent has to decide which queries +1 and feedback +1 it should send to the IR agent, while the agent has to decide which results it should send to the human agent, such that the human agent is able to find every relevant document in the corpus. Finding every relevant document in the corpus is the long term goal of the human agent during a session , however both agents need to be able to decide at each session state , which action to perform. Decision making could be made possible by modelling the information seeking process as a dynamic decision network (DDN), as depicted in Figure 3. A DDN is a dynamic Bayesian network (DBN) extended with decision and utility nodes. Decision nodes are identical to chance nodes, except that the value of a decision node is determined by an action. The IR agent gets a high reward if the human gives positive feedback for a document it read and the human agent if it sends a query at where the IR agent returns documents that are relevant. The nodes of the DDN could be separated mainly into three parts, namely the (i) human simulator, (ii) human agent, (iii) human aware IR agent, and are listed in Table 2. During a session, the set of relevant documents () does not change over time and is not observable by the human agent or IR agent. The human simulator generates a set of weighted words ℎ( ) from a subset of relevant documents , which subjectively represent the information need ℳH of the human. If the human agent reads a document with probability − 1(), then the human simulator adjusts the weights ℎ(). Adjusting the weights is similar to the − 2() − 1() () − 2() − 1(,)− 1() (, )() +1(, ) − 2() − 1(, )− 1() (, )() +1(, ) ℎ− 1( ) − 2( ) ℎ( ) − 1( ) ℎ+1( ) ( ) − 1()
Legend ()
+1() behaviour of a real human, as if a human reads a document, then he learns something new. If a human learns something new, then his information need changes too. From the set of weighted words ℎ( ), the human generates a query − 1( ) that the IR agent can observe, using H→A.
In addition to the query, the IR agent can observe the titles (, ) and (, ) of the documents and compare them with ℎ( ) it approximates, using ˜A→︂H for computing whether a document is relevant () or not. The human agent approximates A as hA, by approximating ˜A→︂H as hA→H and the IR agent is aware of that, by approximating hA→H as ˜hA’→︂H for acting explicable from the perspective of the human agent. In addition, the IR agent approximates H→A as ˜aH→︂A. Approximating H→A as ˜aH→︂A helps the IR agent to plan for future steps ahead. If the human agent reads a document, the human simulator reveals whether the document is relevant or not. The human agent forwards to the IR agent whether a document it read is relevant () = or not () = .
We model the process of a human and an IR agent that collaboratively search for information during a session, by sharing their mental models, as a DDN. The IR agent is modelled such that there is a diference between its mental model ℳ̃︂A and the approximation of the human mental model ℳH as ℳ̃︂aH. In a first sight, this might be counterintuitive as the goal of the IR agent should be identical to that of the human. However, the information need of a human, expressed as a query +1 is only a tiny representation of the subjective knowledge the human has about his information need, as diferent humans specify very diferent queries for very similar information needs. The agent is aware of that and should aim to translate the query into an objective representation A in ℳ̃︂A of the information need of the human it could compare with the documents in the corpus. Translating the information need of the human, expressed as , needs to be done by the agent with respect to the expectations of the human, as the human approximates that as hA in ℳhA. Therefore the agent approximates ℳhA as ℳ̃︂hA’ for estimating the subjective information need H of the human from query . The human either expects that the IR agent is not doing that, as he is highly certain, that his query is objective or he is uncertain and expects that the IR agent translates the query into an objective one before comparing it with each document in the corpus. Regardless of the humans expectations, the IR agent needs to act explicable, such that the human can anticipate, what the IR agent decides to do with a query . Thus, we diferentiate between ℳ̃︂A and ℳ̃︂aH.
A human aware IR agent needs to be able to act intelligent, during a session, depending on the actions of a real human. Thus, we aim to implement an IR agent, by evaluating it with a human agent in a simulated world controlled by the human simulator, that acts similar to a real human. The human agent itself could be evaluated by evaluating the IR agent that is implemented, trained, and evaluated only with the human agent, in the real world. That sufices, as this is the only and primary goal of the implementation of a human agent, even if the human agent might not act exactly as a real human. For achieving a very performant IR agent, it might be beneficial to train the IR agent ofline on that corpus, it later has to answer queries on, from real humans. And even further, the humans that later send queries to the IR agent could help to train it, as we refer to machine training instead of machine learning.
In the future, we aim to implement the human simulator and the human agent for the
implementation and evaluation of a human aware IR agent. The generation of a set of weighted
words, that subjectively represent the information need of a human, given a subset of relevant
documents, could be achieved by generating questions as proposed by Klein et al. in [