Commercial products labelled as smart devices usually recur to a centralised system that processes all the requests. A distributed model, where nodes independently interact with the environment, may provide a widespread support for both users and other devices. In the latter setup, each entity has a partial awareness about defines the requests accepted by the network, and this aspect complicates the task. This paper improves an existing distributed model, called PHASER, by proposing linguistic analysis techniques to manage non-matching requests. NLP methods produce a confidence; any PHASER node forwards non-matching requests to close peers. PHASER exploits the confidence to rank the adjacent peers and deliver the question to the best node. Partial Matching and a Bag-ofWords models will be compared with the currently adopted full matching. The Bagof-Words approach offered the best results in terms of both quality and required time.
I prodotti commerciali, etichettati come dispositivi intelligenti, di solito usano un sistema centralizzato per processare le richieste. Un modello distribuito, dove i nodi interagiscono indipendentemente con l’ambiente, puo` fornire un supporto piu` ampio per gli utenti. Nel secondo setup, ogni entita` e` parzialmente a conoscenza delle richieste accettate da ogni nodo del network. Questo lavoro si propone di migliorare un modello distribuito esistente, chiamato PHASER, ricorrendo a tecniche di analisi linguistiche per gestire le richieste non accettate localmente; ogni nodo PHASER inoltra queste richieste ai nodi adiacenti. Metodi di NPL producono una confidence; PHASER la sfrutta per ordinare i nodi vicini e inoltrare la richiesta al migliore. Modelli basati su partial matching e bag-of-words saranno confrontati con il sistema attualmente adottato, basato su full matching. Dal confronto, bag-of-words ha riportato i risultati migliori sia di qualita` che di tempo necessario per la risposta.
This paper introduces a new distributed approach to question answering and command execution in different Intelligent Environments (henceforth IE). This idea is rarely encountered in literature (with a few exceptions by Surdeanu et al. (2002)). IE is a new discipline including Domotic, Internet of Things, Cultural Heritage Technological Innovation and other similar issues. In our approach, devices in the environments constitute nodes of a network and this network provides services to an interacting user. Reasons of interests for NLP studies in this kind of application lie in the idea that user requests are delivered using Natural Language (mainly speech) in the simplest case, or multimodally by integrating speech with gestures and interaction with physical controls.
Nowadays, smart devices commonly propose
interaction through natural language to the users.
As the services offer becomes wider, a network
of specialised applications managing very specific
domains is masked behind a single named
character (Alexa, Siri, Cortana). While this is
common for single devices hosting multiple
applications, which inform the operating system about
their capabilities through dedicated languages
(e.g. SRGS1, Hunt
We will mainly describe the simplest case of
speech interaction and understanding; for a view
on the complex multimodal approach, please refer
to Valentino et al. (2017). Given these premises,
nodes in the given network are able to respond
to simple questions or commands uttered by the
user. In a first approach, utterances should
conserve a coherence with the “nature” of the node,
i.e. if I am “talking” to the kitchen or to the
microwave oven, I should make requests strictly
inherent with the device functions. On the contrary,
we wish to expand the “intelligence” of the
environment giving to the user the possibility to make
any kind of requirements to any node in the
network. In this view, each node is able to classify
the string deriving by the speech utterance
assigning it to one of the many classes of relevant action
the environment can realise, even if the node itself
is not able to execute that action. The
introduction of a distributed knowledge base and of
network information spreading techniques concur to
the realisation of an environment extremely
reactive, scalable and easily configurable for different
domains. The system is reactive as the network
connections are strongly optimised: redundant and
rarely used paths are pruned. Mechanisms for
knowledge distribution are optimised in order to
deliver the proper answer minimising network
reaction times. The system is easily configurable to
different domains as this kind of networks just
require a formal description of the semantics of each
node, of the action classes they are able to
process, and of the most probable connection among
nodes that a-priori the environment designer
implements. In order to realise this system, many
NLP software modules are needed, and among
these: an automatic spoken dialogue manager, a
Spoken Language Understanding system, an
ontology modelling the environment and the devices.
An extended description of each part can be foun
In Section 2 we present related works; Section 3 recalls the model of our system. In Section 4 we discuss a network of interactive entities, highlighting differences about the current version and the contribution of this paper. Experiments and Discussion are presented in Sections 5 and 6. Section 7 concludes the paper. 2
Our idea is to provide a distributed network of
entities, where each node interacts with the user
through multimodal interaction. Knowledge is
local to the node and limited to the provided
services. If the node is not able to produce the
expected output for a request, it sends the message
to others in the network, without a prior
determined target node. The intelligence perceived by
the users is built upon a collection of partial nodes’
intelligence. This system, called PHASER, has
been firstly intro
Distributed approaches for Human-Computer
Interaction have been widely discussed in
literature. Multi-Agent Systems have been applied
to smart environment
The goal of this work is to provide a strategy
to better rank close nodes according to the
exposed information about the accepted inputs. By
considering the Navigation problem from a
Question/Answering (Q/A) point of view, PHASER
could be theoretically compared with distributed
Q/A systems
Baeza-Yates et al. (1999) stated that the
Ranking problem is fundamental in Information
Retrieval. It can be solved with machine learning as
summarised by Liu and o
In IE the term Intelligent usually refers to
Artificial Intelligence (AI) applied to environments,
where technology offers more than static rooms as
introduced by Augu
In our concept, PHASER gives a role to each
entity who interacts with the others. Possible entities
are objects and people interacting with those
objects. We make use of an abstract concept of node
to include the needs of both the entities. In real
scenarios, objects are AI-powered devices,
considered an important step on an evolutionary process
that is affecting modern communication devices
Each object interacts with other connected entities, providing services and responding to questions. A graph results and objects individuate its nodes. For this reason, we will refer to objects as nodes as well. 3.2
A single node represents an entity in the environment. We formally define a node as a tuple:
N ( ; Cnf ; Close ; Discovered ; oBC ) where is a unique identifier of the node in the environment and Close is a set of related nodes in the environment. Discovered collects nodes connected after unforeseen interactions. Close and Discovered contain identifiers of the remote nodes. establishes connections and interacts with nodes in both the sets. Each node specifies a configuration Cnf , which determines ’s role in the environment. The configuration comprises inputs, outputs and how it reacts to network events. In details: Cnf = (name ; type ; class ; env ; I ; O ; P ) where type, class and env classify according to an ontology, while name labels it. I and O represent inputs and outputs respectively; they divide data into channels as in Equation 1 for multimodal interaction, where cx is a channel code and RGcx = ri1 ; ri2 ; : : : ; ricx is a set of regular expressions. If Ni and No are the number of input and output channels, we define I and O in Equation 2.
I =O =
Chj = cj ; RGcj
[ 1 x Ni =o fChxg (1) (2)
Input and Output compose the Business Card (BC) and it represents what a node may accept; each object exposes its own BC to the connected nodes; received BCs will be stored in oBC. The approach discussed in this work ranks Close and Discovered peers by obtaining confidences from their BC. PHASER nodes compose the network. There is not a hierarchic organisation, so all the nodes are at the same level. The network does not need a specific topology, but we assume that an expert of the considered domain designs it.
The presented formalism defines a PHASER
node which establishes a connection towards other
similar entities. This is the core part of our
system: a distributed model where single peers
interact with people - through I/O modules - and with
others. Input and Output modules are intentionally
generic because each node can have a customised
the interaction. This approach aims at
supporting Natural User Interfaces
The discussed formulae are the core part useful
to understand the introduced improvements. A
detailed description of the PHASER model has been
provi
In PHASER, each node is expected to have a knowledge, circumscribed to its own domain: a fridge should understand questions about food or ingredients; commands about lighting and heating are out-of-context. However, the environment could contain other devices to manage those requests. In such a configuration, a network of PHASER devices is populated of entities with partial knowledge, where nodes have a common strategy to propagate out-of-context requests.
Each node is able to interact with users. It means that each of those entities may manage commands for any other node in the network. They could handle non-matching requests in two ways: (i) broadcast them to all connected nodes or (ii) individuate the most probable nodes. The second approach is preferred in very active networks - i.e. smart museums - or where a large percentage of nodes processes unknown requests; moreover, the first approach easily overloads the network. This “Navigation problem” aims at forwarding the request to the best candidate; the model follows a depth-first-search by iterating on a sorted list of close nodes. The ranking is obtained with a greedy approach. The navigation continues until a node finds a response or all the sub-network is explored. Figure 1 reports a graphical representation of the sorting phase that is performed in the navigation presented in Figure 2. D1 chooses the next node in the interaction by sorting the adjacent vertices.
This work analyses how the discussed probability is obtained, presenting the current technique, and investigating improvements that support partial matching to tailor the propagation on the network. The outcome is a different rank of the list of close nodes explored in the navigation of the graph. 4.1
A network of PHASER nodes starts from a topology designed by experts. During the interaction, the network adapts connections to maximise the local utilities on each arc. As it is set up, each pair of connected nodes shares a business card. It comprises a set of active channels and, for each channel, a set of regular expressions (regex) for the accepted inputs.
The “Navigation problem” is solved by sorting the adjacent nodes according to their matching with the exposed regular expressions. Nodes with higher value of matching will be firstly called in forwarding. Inputs can be on multiple channels, so the matching complies with the structure. In this version, M calculates the value of matching as defined in Equation 3:
X 0 i<jRj M (R; n) =
m(Ri; n)=jRj m (Rx; n) = where n is the considered device, R is the request, divided into jRj channels. The expression “Rx is valid for n” of the Equation 4 means that exists a regular expression of n that matches with Rx. The higher M is, the higher is the probability that n accepts R. M (R; n) = 1 means a perfect match.
The currently adopted approach is based on full matching where the outcome of each m(x; n) is dichotomous. The calculated value M is then normalised to the size of R - involved channels in R -. This approach highly depends on the accuracy of the design of the set of regular expressions. Moreover, generic regexs - i.e. “.*” - accepts everything. This case undesired, as if a node accepts this input it will attract many requests with the consequence of not being able to process all of them; this would create a black hole, that uselessly overloads the network.
We considered a star-like network with TV in the middle. We tested two kinds of configurations for input representation. A request is delivered to the TV, which forwards the request to the node with the highest confidence; this is obtained with the different approaches. The network is design to let Kettle being the winner.
Table 1 collects data where inputs are
represented with a BoW style; in Table 2, instead,
inputs are represented as regular expressions. Each
node used OpenDial by Li
max n miR1x ; : : : ; miRnx o 0<x jRj (5) The function in Equation 5 supports multiple channels and a set of possible grammars for each of them, but mlvx is now the probability that the token v from the request is accepted on an input lx. This probability can be calculated with two strategies: regex-based and bag-of-words (BoW). The former approach calculates the longest substring that matches on each provided regex on the proper channels; this obtained length is then normalised on the total length of the request. The bag-of-words method, instead, splits both the request and the stored accepted inputs in two bags of words - Breq and Binput respectively - and calculates how many words of the request match on the total set. This value is then normalised on jBinputj. Both the strategies are locally performed by nodes on received questions that must be forwarded. No global dictionaries are saved in order to maintain a scalable distributed system where each node has partial knowledge about the others.
Since any NLP approach provides a confidence of the evaluated input request, other strategies have been considered. However, these approaches present drawbacks that will be discussed in Section 6. 5
This Section reports experiments conducted to compare the three discussed approaches in PHASER: perfect matching, partial matching, and BoW. Full and partial matching methods rely on regular expressions and assess how much the request matches the provided regexs. The system has been tested by simulating a smart house with 5 networked PHASER nodes. The considered nodes are TV, Microwave Oven (M), Fridge (F), Kettle (K), Alarm clock (A). The presented process operates in a context where the current node n is not able to understand the request r and it prefers to share it with the network, refraining from broadcasting. The node n gathers a confidence on r to sort the adjacent nodes, preferring nodes with higher values. A sequence results, where the first node is the best candidate to accept the request.
Results show that a full matching is not always a good choice. It requires a precise design of each regex, exposing the structure of accepted inputs; moreover, this strategy does not always discriminate different nodes and fails in many cases. Partial matching provides finer values and nodes are better sorted. However, this approach easily creates black holes, nodes that attract many inputs because of a wrong design. The BoW model gave command prepare a tea
warm warm water wake me command prepare a tea
warm warm water wake me perfect K (1.0) A (0.1) A (0.1) A (0.1) perfect K (1.0) A (0.1) A (0.1) A (0.1) partial K (1.0) M (0.44)
A (0.1) A (0.438)
BoW K (1.0) M (0.5) K (0.667)
A (0.5) partial K (1.0) M (0.44)
A (0.9) A (0.778)
BoW K (0.1) K (0.33) K (0.667)
A (0.4) the best results with two benefits: (i) the network is easier to design; (ii) each node could share unstructured data, improving local security.
Other strategies have been investigated. We considered more refined systems based on SRGS; however, this method has been excluded for many reasons: (i) SRGS requires a complete grammar from adjacent nodes and this may generate security issues because they expose a detailed structure of accepted inputs; (ii) grammar-based methods introduce overheads compared with the adopted approaches, due to the engine needed to recognise the request on the model represented by the grammar. 7
This paper presented PHASER, a distributed model for Human-Computer Interaction in Intelligent Environments. This work aims at improving the Navigation Problem, where a node forwards a received command if it is not able to understand or process it. Since the node operates with partial knowledge about both the request and the environment, it tries to analyse the input and choose the best adjacent node.
The most crucial part is not a refined linguistic analysis of each request, but a quick confidence on how much each adjacent node could be a good candidate to understand that request. This requirement is motivated by two reasons: (i) this process is part of a longer step where a user is waiting for a response; (ii) all the evaluations rely on information each node shares with others. In order to deeply understand the command, the node should expose sensible data and it is not always desired in a distributed context.
The work focused on three strategies: perfect and partial matches with regular expressions and a bag-of-words model. This last approach has given the best results with positive aspects mainly related to easy network design and security of each node. The investigated methods are just used to rank close peers on as an out-of-context request reaches the current node. It operates without understanding the request, so finer considerations are not possible. The considered approaches do not limit PHASER nodes in adopting more refined techniques in assessing and categorising an input request.