Nowadays, touch-based user interfaces are widely used in consumer electronics. Recent trends con rm the high potential of touchless interface technologies to manage also Human-Machine Interaction, in scenarios such as healthcare, surveillance, and outdoor activities. Moving from pointers, keyboards or joysticks to touch-screens represented a signi cant challenge. However, a touchless approach needs to ensure intuitiveness and robustness. This paper describes a framework enabling the wireless control of a mobile robot through a contactless controller. The data provided by a complex sensor, composed of two stereo cameras and three IR sensors, are processed with custom algorithms that recognize the movements of users' hands. The result is promptly translated into commands for the robot running a real-time operating system. Usability tests con rm the compelling employment of contactless controllers for mobile robots and drones both in open and closed environments.
Although there are di erent kinds of systems with di erent peculiar goals, even within the same domain, they all face a common challenge: interaction with the users [Suc87] guaranteeing the compliance with their needs [CCS+16]. Indeed, all systems involving human interaction provide a way to support a two-sided communication: human to machine (interfaces to manage inputs, command, etc.) and machine to human (interfaces to perceive the systems' feedback) [Min02, WG95]. To have robust, dependable and usable interfaces, they have to be accessible, as much intuitive as possible, and to respond in a time coherent with human requirements (if the interaction is too fast or too slow the user gets easily lost) [Fis01, Dix09].
Over the years, to produce more powerful interfaces, programmers tried to couple syntax and semantics, trying to associate the dynamics of a command to the actual output (gesture or movement) [Bro98]. Indeed, around the 40's, multi-purpose systems only employed panels with buttons (evolved into the modern keyboard), while systems dedicated to more speci c contexts required properly designed interfaces. For example, in the case of moving mobile objects, both directly or indirectly, the joystick (emulating the older cloche employed in the aviation) became in the 80s the default choice for electronic controllers [Wol88, Gra81]. Even tough it was widely declared a big failure when employed as a pointer [CEB78], it kept evolving (maintaining its original basic features) becoming a standard interface for systems moving/driving objects.
The advent of the new touchscreen devices radically changed the human experiences when approaching technological devices. However, to face speci c tasks like moving objects (e.g., gaming, etc.), even tough the touchscreen revolutionized the way of thinking about \how to perform an action", a simulated joystick still represents the more intuitive means to translate an intention to an action. In line with the current technological trends, this paper aims at moving a further step on behalf of the old joystick, bringing it in the touchless era [MBJS97].
The presented solution is a system, based on multiple devices, which detects hands movements, translating them into joystick-like commands, and nally communicating them to a mobile robot, that behaves accordingly to the received instruction. The rest of the paper is structured as follows: Section 2 presents an overview of the framework, its structure and its main components and functionalities. Section 3 addresses the system's requirements, functionalities and challenges. Section 4 presents the conducted tests summarizing their results. Finally, Section 5 concludes the paper, presenting the lesson lernt and the future works. 2
This section introduces the framework, presenting an overview of its structure, its main components and their functionalities.
The framework is mainly composed of three modules: the User Interface Module (UIM) , Figure 1(x); the Communication System (CS), Figure 1(y); the Mobile Robot (MR), Figure 1(z).
The interactions between the components of the framework are represented by the blue arrows in Figure 1.
Currently, the technological market o ers a broad range of solutions for gathering and processing hands' position in space/time. Vokorokos, et al. [VMC16] analyzed the e ciency of the three most relevant devices on the market: the Leap Motion Controller [Leab], the Microsoft Kinect [Zha12] and the Myo Armband [Nuw13]. Depending on scenarios and settings, those devices showed up di erent peculiarities. For example, the Microsoft Kinect enables to track the entire body motion within a detection range of 0:5m and 4:5m [FP11], the Myo Armband is a wearable bracelet reading the muscular' electric impulses, that guarantees a high level of comfort, however limiting the amount of information that can be acquired. Compared with the previous ones, the Leap Motion Controller represents a better trade-o between exibility, time-to-prototype, and precision [MDZ14], and thus it has been chosen for the presented project.
The applications in the virtual reality domain are increasing remarkably, both in presence [EFAA15] or in absence [PBAR15] of haptic feedback. Such solutions mainly aim at moving and virtualizing real objects or their models, and entire environments while enabling interactions with them. The Leap Motion Controller is massively employed in those systems, like the presented one, providing a visual rather than a haptic feedback [BGG+14, FSLS+16].
Brie y analyzing the Leap Motion Controller, it is a sensor composed of two stereo CMOS image sensors and three infrared LEDs. Such a sensor, coupled with a set of libraries, besides exporting the image of the area over the sensor, exposes APIs to get the kinematics of the hand and all its joints, thus enabling the hands' acquisition as in Figure 1(a). The Command Generator Module in Figure 1(b), running on a PC, operates over the obtained data to extract relevant information, and decide which commands to send to the mobile robot through the Communication Client in Figure 1(c). The communication system, running a custom Communication Protocol over an XBee connection [Fal10], Figure 1(d), transfers the commands detected by the User Interface Module to the Communication Server, Figure 1(e). The Decoding Module, Figure 1(f), elaborates the received information which is then transferred to a bu er shared by the two Motor systems, Figure 1(g), in charge of nally performing the robot's motion. 2.1
Robotic actuation and motion are widely known as safety-critical systems. Thereby, the guarantee that any task in the system respects its deadline, or at least that the system remains operative during possible deadline misses, is mandatory. To operate correctly, the system has to perform a series of periodic tasks. Those can be characterized as follows: tdet 50 ms is the interval of time between the instant at which the user performs a command and its detection.
tel 20 ms is the interval of time between the command detection and the message transmission to the robot.
tcom is the time required for the communication to take place. It is unpredictable, but it can be minimized accepting that some data can be lost.
tact 50 ms is the time between the reception of the message by the robot and the actual motion. To obtain a smoother speed pro le of the robot, a lter operation can be performed with an additional delay tsteady 150 ms. (X) User Interaction Module (a) (b) (c)
Data acquisition
Command
Generation Communication
Client (Y) Communication System
Dependable (d) Communication
Protocol (Z) Mobile Robot (e) (f) (g)
Communication
Server Decoding
Module Motor_Sx System
Motor_Dx System 15 ms: thread handling the serial communication (thread tick); 50 ms: thread detecting the hands' motion and the commands creating the messages to be sent
A Rate Monotonic scheduler is employed to schedule the threads composing the Robot Module, which is powered by the Embedded Real-Time OS ARTE [BBP+16]. The threads periods are: 40 ms: thread decoding new messages and managing the two servo motors. 3
ule and the Communication System can be characterized by di erent requirements, functionalities and challenges, that are presented in this section. 3.1
The User Interaction Module (UIM) is intentionally released without a GUI, since a more natural interaction is obtained by simply having the user actions re ected by the movements of the mobile robot. The UIM is implemented in the C++ language, and is platform independent. The only requirement is the support for the library of the Leap Motion Controller [Leaa]. Since the User Interaction Module is touchless, the user is supposed to be able to perform at least basic hands' movements. The interaction area is about one cubic meter centered above the sensor. The hands are continuously tracked within this area, and when speci c paths, positions or movements recalling the semantic of a joystick are identi ed, the related commands are communicated to the robot.
recognition enable three functional units:
The Controller detects the hands' position, movements, speed, and rotation radius expressed according to the Cartesian axis placed as shown in Figure 2. Custom messages' structures de ned by the Communication Protocol encapsulates such parameters.
tion 3.2, manage a circular bu er containing the aforementioned messages, discriminating which have to be sent or deleted because out of date.
Two modalities have been implemented to drive the robot:
Normal mode - the robot is driven like a normal 3-wheel car. The commands are: stop, go forward or backward, turn right or left.
Rotatory mode - the robot rotates around itself, and the only commands are: rotate or stop.
The stop, \rest" state of the joystick, is established with a minimum margin of tolerance of 1 cm, around (0; 20; 0) cm on the area above the Leap Motion Controller [WBRF13]. If the hand is perceived in such an area (no movements are required) a single message is sent requiring the robot to hold the position.
The go forward or backward is determined by the hand's position on the z axes: z move forward and +z move backward. The distance of the hand from the origin of the z axes de nes proportionally the speed of the robot.
backward but it is referred to the x axes: x turns left and +x turns right. The distance of the hand from the origin of the x axes de nes proportionally the rotation radius to be followed by the robot.
By rotating the hand upside down and then back again activates the rotating mode. The command \rotate" de nes the rotation speed by moving the user hand along the x axis. 3.1.1
The information acquired from sensors like the Leap Motion might introduce uncertainty or noise. Such a noise can generate incorrect or redundant information. Thus, undesired behavior might take place, generating feedback di erent from the expected. To avoid this possibility and ensuring a consistent reading, two techniques have been designed and implemented: the rst is sampling in frequency, and the second is sampling the interaction area. The sampling frequency is about 20Hz, and to be sure that the identi ed command is correct, the user's hands have to be detected in the same position for at least 3 consecutive frames provided by the Leap Motion sensor. Recalling that the interaction area above the Leap Motion sensor is about a cubic meter, it is linearly clustered. Such a choice ensures fewer variations of the hand's position, thus generating fewer messages. The size of the partitions of the area are chosen accordingly to the feedback provided by the motors: passing from one area to an adjacent one results in a change of speed by a factor of 5. Such solutions provide a higher reactivity and a smoother and more usable mobile robot. 3.2
The users' experience and the actual system usability gain signi cant bene ts by employing a wireless communication (by exchanging messages) between the UIM and the Robot. The proposed protocol aims at being robust.
Indeed, the TCP [KR07] inspired its design (in terms of ACKnowledgement mechanism and of timing constraints). This protocol aims at guaranteeing that if more than one message is queued in the bu er (more than one command is detected in a relatively short period, or possible communication delays happened) only the most recent is delivered to the robot. These behaviours are modeled by the state machine in Figure 4. 3.2.1
One of the requirements for the custom designed protocol were the ability to handle undesired retransmissions in order to avoid delays and to guarantee the highest possible responsiveness of the system. Thus, during its development we have taken some concepts from the TCP protocol. In fact, the implemented mechanism to reduce the communication overload and the number of unneeded computations performed by the robot is shown in Figure 3(b).
In particular, the developed protocol allows to use 16 di erent message formats. The formats employed in this system's version are two: one communicating the ACK, Figure 6(a), and one communicating speed and rotation angle, Figure 6(b). The messages' structures are detailed in Section 3.2.2 and represented in Figure 6.
If a message is available, it is sent from the User Interaction Module to the Robot. The Controller keeps generating new messages (related to the identi ed commands) while it is waiting for an ACK from the Robot.
If the Controller receives an ACK with the expected Id (equal to the sent one), it means that the sent command has been received by the robot, so the bu er is scanned looking for the next message to be sent, Figure 3(a). On the Robot side, the Communication Server receives the messages sharing the information with the Decoder Module through a shared bu er while
Circular Buffer
Msg [Id = 1]
If the Id contained in the ACK received by the Communication client does not match the expected one, or if no ACK is received before a predetermined period of time (named timer), the last message (the most recent) inserted in the controller bu er is sent removing all the older messages, Figure 3(b). Moreover, an internal timer is set once a message is sent. If such a timer expires before the UIM receives a proof of reception (ACK), the message is sent again.
Promoting the robustness analysis through formal veri cation, two virtual state machines are used to model the Communication Client (UIM), Figure 4, and the Communication Server (MR), Figure 5. sending an ACK message (containing the same Id) back to the User Interaction Module. (a) (b) 3.2.2
The structure of the messages has been inspired by the message structure of the MIDI protocol [Mid] (MSB to LSB). As aforementioned, the system uses two formats for the messaging, Figure 6.
Similarly to the MIDI protocol, the messages are composed of StatusByte (Figure 6(x)) and DataByte (Figure 6(y)). The StatusByte (8bits) contains:
The value identifying header is 1;
Used for feedback mechanism;
If the message is an ACK the value is 0b0000, if the message contains data the value is 0b0001. e yt B 1 e yt B 1 e yt B 1 1 bit - Header identifier 3 bit - Id 4 bit - Type of message 00000010 -- ADCatKa 1 bit - Data identifier 1 bit - Sign for direction 6 bit - Speed absolute value 1 bit - Data identifier 1 bit - Validity 1 bit - Sign for direction 5 bit - Rotation angle absolute value (b) (c)
knows that the message is complete (there are no more bytes/information to be read). An ACK message is generated by the Robot and it contains only the StatusByte. If its value is 0b0001, the Robot knows that the payload contains two more bytes of information. When a message is sent from the UIM to the Robot, 3 bytes are sent: one StatusByte and two DataByte as payload.
The Speed format, Figure 6(b), is composed of the following three parts: the Communication Server receives the messages as presented in Section 3.2; the Decoding Module elaborates information about speed and rotation radius encoding the actual commands for the Motor Modules ; the two Motor systems (one for each motor) manage the servo dynamics. 3.3.1
The Decoding Module elaborate the commands depending on the required behaviour: curving; straight motion; motion around itself.
Rotating the robot around the axes centered on itself requires di erent speeds on the wheels (Figure 7(a)). De ning speed (v) and rotation radius (r), the wheels' speed is calculated as shown in equation 1:
Vright = v
(v; r) Vleft = v + (v; r) (1)
Figure 7(b) identi es the di erential speed enabling the turning action (named (v; r)) which is obtained as shown in equation 2: v (v; r) = C (2) r where C is the distance between the wheel and the center of the robot wheelbase.
The speed of two motors is simply set equal to v. v+dx
v C
C v-dx r r (a) (b) The velocities of two motors are opposite and their absolute value is equal to v (the sign depends on the direction of rotation). 3.3.2
When the Motor System receives a command, it converts the speed or rotation radius into engines' instructions. Thus, developing a linear pro le for the wheels' velocities, it needs particular care to prevent sliding and overturns.
Figure 8 exempli es how a speed pro le can be realized when a new desired speed is requested before the previous one is reached: at t = 0 a speed request arrives, Figure 8(a). The system provides to the motors a linear speed prole to satisfy the request in a xed T ; at t = 3 (with t < T ) a slower speed request arrives, Figure 8(b). The system calculates a new pro le starting from the current robot's speed; nally, the stop command arrives at t = 6, Figure 8(c). Since no more new requests arrive in a period where t < 6 + T , the robot is stopped within the expected T , holding that speed (in this case 0) till a new command arrives. The presented system is composed of several elements. Considering that design and implementation errors can happen in several of those elements proper testing is needed. As manual testing is a cumbersome procedure, test automation is highly needed. Thus, we searched for existing test automation frameworks and decided to adopt the CUnit [Ham04] and the gcov systems [Rid04]. Furthermore, Usability Testing is a key aspect when dealing with HMI systems. In the rest of this Section we present what has been adopted in this work. Performance analysis and dynamics veri cation are two strategic tests. A useful framework to automatically test functions code is CUnit. Such a framework has been employed to check the coherence of the expected outputs of Command detection and messages generation, reception and transmission which are implemented as state machines. Code Coverage Analysis [CCA] is crucial to identify and purge the dead or obsolete code from the system. Moreover, the coverage analysis counts the number of times each program's statement is executed. Finally, the pro ling tool Gcov is used to increase the code optimization. The User Interaction Module aims at proposing a set of manageable and e ective commands.
To test their functionality several approaches are viable, and the most e ective are the usability test [DR99] and the usability inspection [Nie94]. The rst approach is a user-based method that involves the end users in the testing phase, while the second is an expert-based method that during the testing phase involves only experts and context aware users. Both of them aim at identifying possible issues related to the usability. They mainly di er in the categories of the identi ed problems. Indeed, although the usability test identi es fewer issue's occurrences, these are more general and frequent issues [JMWU91].
The rst usability test conducted concerns the types of commands: di erent semantics (joystick, steering wheel, etc.) were taken into account. This rst experiment, named A/B test, consisting in executing the basic commands as move forward or backward and turn left or right with di erent speed and rotation radius, was useful to gather testers' opinion about e cacy, accuracy, and personal perception. Indeed, 27 testers out of 30 expressed their preference for the joystick-like interface. The second test, named Hallway testing, involves di erent groups of testers randomly selected to test the whole system. The test consists of driving the Robot between a series of obstacles. After a few commands, the testers gained con dence with the system and easily accomplished the test. Imagining to use a Cartesian system to represent the lessons learned, putting on the vertical axes the number of attempts and on the vertical axes the coe cient of experience matured during tests, the output is a learning curve which can be approximated by an exponential function. Finally, even though the current system does not provide a GUI (the system's feedback is directly provided by the motion of the robot) all the testers appreciated the system's response, classifying the system as \user-friendly". 5
The presented project aimed at bringing a traditionally physical user interface for human-machine interaction like the joystick in the touchless era. In the proposed work, a touchless joystick has been realized and employed to drive a wireless mobile robot customized for that purpose. All the system's components have been tested and satis ed the conducted formal veri cations. Thanks to the adoption of real-time policies, the timing constraints have been respected throughout all tests. Moreover, according to the testers' feedback, the whole system was easy-to-use, responsive and effective. The current system's version perceives both the user hands, but it uses only one of them to get the commands to guide the robot. The next step is introducing the acquisition and elaboration of the second hand in charge of handling sensors equipping the robot. An initial idea we are working on is to equip the robot with a mobile camera. Such a camera could be handled (zoom, movements, taking pictures, etc.) by the user with one hand while driving the robot with the other. This new feature involves updates in the data acquisition, communication protocol, and the actuation module. While the data acquisition and the actuation modules have to be restructured, the communication protocol, thanks to its implementation, requires only to be extended with the de nition of the new types of messages. An alternative might be including voice commands to control the camera and empower the interaction with the robot [CSM+16]. Finally, the introduction of a GUI is under evaluation to enrich the user experience when operating on the mobile camera. [CCA] [CCS+16] [CEB78]
terface design guidelines. Intellect Books, 1998.
Springer Science & Business Media, 2011.
CCA. Code Coverage Analysis. http:// www.bullseye.com/coverage.html.
Davide Calvaresi, Daniel Cesarini, Paolo Sernani, Mauro Marinoni, Aldo Franco Dragoni, and Arnon Sturm. Exploring the ambient assisted living domain: a systematic review. Journal of Ambient Intelligence and Humanized Computing, pages 1{19, 2016.
Stuart K Card, William K English, and Betty J Burr. Evaluation of mouse, ratecontrolled isometric joystick, step keys, and text keys for text selection on a crt.
Ergonomics, 21(8):601{613, 1978. [Dix09] [DR99] [Fal10] [Fis01] [FP11] [EFAA15] Ru aldi Emanuele, Brizzi Filippo, Filippeschi Alessandro, and Carlo Alerto Avizzano. Co-located haptic interaction for virtual usg exploration. In 2015 37th
IEEE, 2015.
networks: with ZigBee, XBee, arduino, and processing. O'Reilly Media, Inc., 2010. Gerhard Fischer. User modeling in human{computer interaction. User modeling and user-adapted interaction, 11(12):65{86, 2001.
Valentino Frati and Domenico Prattichizzo. Using kinect for hand tracking and rendering in wearable haptics. In
IEEE, pages 317{321. IEEE, 2011. [FSLS+16] Ramon A Suyrez Fernyndez, Jose Luis Sanchez-Lopez, Carlos Sampedro, Hriday Bavle, Martin Molina, and Pascual Campoy. Natural user interfaces for humandrone multi-modal interaction. In Unmanned Aircraft Systems (ICUAS), 2016
1022.
J Martin Graetz. The origin of spacewar.
O'Reilly Media, Inc., 2004.
Michael Gonzalez Harbour. Programming real-time systems with c/c++ and posix. [KR07] [Leaa] [Leab] [MDZ14] [Mid] [Min02] [Nie94] [Nuw13]
Giulio Marin, Fabio Dominio, and Pietro Zanuttigh. Hand gesture recognition with leap motion and kinect devices. In 2014
IEEE, 2014.
Midi Association. Midi Association - The o cial midi speci cation. http://www. midi.org/specifications.
Jakob Nielsen. Usability inspection methods. In Conference companion on Human factors in computing systems, pages 413{ 414. ACM, 1994.
Rachel Nuwer. Armband adds a twitch to gesture control. New Scientist, 217(2906):21, 2013. [PBAR15] Lorenzo Peppoloni, Filippo Brizzi, Carlo Alberto Avizzano, and Emanuele Ru aldi. Immersive ros-integrated framework for robot teleoperation. In 3D
Symposium on, pages 177{178. IEEE, 2015.