Unconventional interfaces have been increasingly present in people's daily lives. Creating “invisible” interfaces has made interaction with computer systems more straightforward and intuitive. This work aims to present the entire process of developing the integration of an interaction device that allows mapping and collecting information from user interaction with the surface of a sheet of paper. To do so, it was identified the need to create a microcontroller system where sensors, placed under a sheet of paper, are responsible for mapping the user's interaction with the paper. The information collected by these sensors is sent to a software responsible for processing the information and reproducing all touches made by the user on the sheet of paper to a computer screen. All the development steps are presented in this article, as well as the achieved results.
The popularization of the use of computers both in the workplace and in the domestic environment,
which began in the 1970s, highlighted the importance of establishing a better way of interaction between
people and computer systems. Consequently, the area of Human-Computer Interaction (HCI) has
emerged as a relevant research field in developing interactive systems [
Since then, the user has come into contact with different devices and interaction paradigms, which have
evolved to facilitate user communication with interactive systems. Natural User Interfaces (NUI) are
beginning to gain space in the market through unconventional devices [
In this sense, several researchers have developed interaction devices using piezoelectric properties materials-based. Used to create pressure-sensitive devices, piezoelectric sensors in interaction devices allow mapping the user’s activities during the interaction and sending this information to be processed
2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). This article is organized as follows. Section 2, Theoretical Foundation, presents themes related to Unconventional Interaction and related works. Section 3, Work Methodology, discusses the necessary steps to achieve the research objectives. Finally, in section 4, we exhibit the conclusions of the work.
In the evolution of interfaces between users and computer systems, it is possible to highlight three
main paradigms: first came the Command Line Interfaces (CLI), then the Graphical User Interfaces
(GUI) and, more recently, Natural User Interfaces (NUI –) [
Command line interfaces, performed through a prompt, are found in older operating systems, such
as MS-DOS or early versions of Unix. They were - and still are - the interfaces most used by advanced
users, as they allow commands to be carried out concisely and powerfully [
On the other hand, graphical user interfaces have gained the preference of non-expert users, mainly those based on the WIMP paradigm (Window, Icon, Menu, Pointer). From a usability point of view, this interface brings benefits such as the ease of remembering actions and building models that remain consistent for several similar tasks.
In recent years, a series of devices began to appear on the market as new modes of interaction: touch
screens, voice command recognition, and gesture recognition, among others. NUI is classified into
haptic, organic, wearable, gestural, and voice interfaces. NUI is an acronym that designates
humancomputer interfaces using unconventional devices based on natural elements. In this case, the term
“natural” is used in opposition to most computer interfaces that use devices whose operation must be
learned. In this type of interface, it can be assumed that the interaction is based on the user’s previous
experience; therefore, it is easy to learn and become an expert [
The popularization of the use of computers both in the workplace and in the domestic environment,
which began in the 1970s, highlighted the importance of establishing a better way of interaction between
people and computer systems. Consequently, the area of Human-Computer Interaction (HCI) has
emerged as a relevant research field in developing interactive systems [
The work by Rendl [
Gong [
On the other hand, Kalantari [
The electrical characteristics of Linqstat were used in applications where it is necessary to map the
pressure exerted on a specific area [
Using the same idea of building pressure sensors presented in work by Suprapto [
About the methodology used in this project, the work had, as its starting point, reading about the themes involved in the research. The reading aimed to understand ways to create sensors sensitive to pressure and/or touch and the logic for possible software to interact with the hardware. After completing the hardware design, a software testing phase was implemented to validate the prototype. To achieve the objectives proposed in this work, the following steps were conducted:
1- Literature review of the topics involved (conductive materials, resistive materials, pressure sensors).
2 - Meetings with specialists in the area of technology, hardware and development. 3 - Research of materials for use in the hardware. 4 – Study of forms of integration between hardware and software. 5 - Hardware development with sensor tests. 6 - Initial software development for data collection (hardware interactions).
7 - Software development for data collection validation (turning manual interactions into responses within the software).
8 - Analysis of post-test results.
The objective of this project was to develop a proof of concept of an interaction device that allows mapping and collecting information from user interaction with the surface of a sheet of paper. For this, the need to create a microcontroller system was identified, where sensors placed under a sheet of paper are responsible for mapping the user’s interaction with the sheet. The information collected by these sensors is sent to a software responsible for processing the information and for reproducing all the gestures performed by the user on a sheet of paper on a desktop screen. Figure 1 presents a conceptual diagram of the prototype idealized in the project.
In Figure 1, the development of the prototype was divided into three modules: sensitive layer, microcontroller, and desktop. The sensitive layer is responsible for detecting the interaction between the user and the sheet of paper; the microcontroller collects data from sensors in the sensitive layer, processing and transmitting the data via serial communication to a desktop (where software receives the information). For the proof of concept, this software is a simple game, and the information sent by the hardware is responsible for modifying the graphical representation of the game. A detailed description of each proposed module is presented in the following items.
For the design of the sensitive layer of the system, a literature review was conducted to identify possible sensors and/or materials that could be used in the project. Besides the necessary properties to detect user interaction, the use of low-cost commercial sensors that offered precision and flexibility was imposed as a requirement for developing the sensitive layer.
The so-called piezoelectric devices are an attractive low-cost alternative to be adopted as a sensor in the sensitive layer. According to Gautschi [10], piezoelectric sensors are analog devices that produce a proportional electrical charge/current as an output when subjected to pressure or mechanical stress. The characteristics described by Gautschi [10] added to the low cost, and the ease of finding piezoelectric sensors in retail stores enable this category of sensors in this project.
After defining the use of piezoelectric sensors in the sensitive layer of the interaction device, it was defined that the mapping of the paper sheet would be performed through a matrix where each position of it has a piezoelectric sensor that will detect when the user presses the sheet. As the proposal of this project is the elaboration of a proof of concept, it was decided to build a 3x3 matrix that allows dividing the sheet of paper into nine quadrants, as shown in Figure 2.
According to Tressler [11], different materials and configurations can be used in the construction of piezoelectric sensors; therefore, to develop a sensor matrix, as indicated in Figure 2, as well as to meet the initial requirements of the project, sensors based on a metallic brass structure were tested. Solutions based on a flexible piezo-resistive of a polyethylene-carbon compound known as Linqstat/Velostat were also tested.
In tests using piezoelectric devices based on a metallic structure, two commercial sensors with different diameters (15 mm to 20 mm) were tested; both devices are built on a 1 mm thick brass disc covered by a ceramic film and silver electrodes.
In tests using the Velostat sheet, unlike the devices based on a metallic brass structure described
above and which can be purchased ready to use, for the use of Velostat as a piezoelectric sensor, it is
necessary to use additional materials. Due to this characteristic, the proposed solution using Velostat
for the implementation of the sensitive layer of this project was based on the solution presented by
Suprapto [
In the arrangement shown in Figure 3 (A), the Velostat works as a variable resistor and its resistivity changes when pressed by the user of the device. To determine the material that would be used to wrap the Velostat, low-cost commercial materials that can be purchased at retail stores were chosen. The materials tested in this project were: 8mm wide copper tape, 3cm wide copper foil and 0.9 mm aluminum wire. The results of the different materials used together with Velostat in the sensitive layer will be discussed in the Results section.
Integrating the sensitive layer and the desktop software will take place through the Arduino UNO open-source electronic prototyping platform. Implemented under an Atmel MegaAVR microcontroller model ATmega328P, the Arduino will be responsible for acquiring the analog information from the sensors of the sensitive layer and converting the analog signals into digital signals through an internal 10-bit AD (Analog-Digital) converter. After converting the information from the sensors into digital signals, the Arduino will also perform a pre-processing of the information and transmit the data referring to the user’s interaction with the sheet of paper to a microcomputer through a UART (Universal Asynchronous Receiver/Transmitter) serial connection/communication
As the project proposed by this work involves the integration of the Arduino platform with software created for desktop (which in this case is a game), to describe the development of the software used in the project, this section was divided into hardware programming (code created for the Arduino platform) and validation software (game created to validate the interaction device).
Hardware programming: Developed in the Arduino IDE and using the C/C++ language, the code developed for the Arduino is responsible for processing the information from the piezoelectric sensors present in the sensitive layer matrix. This code allows the microcontroller to identify in real-time the quadrants being pressed or not by the user during his interaction with the game (validation software).
To standardize the communication between the software developed for the Arduino and the validation software developed for the desktop, an application-level communication protocol was designed and implemented to transmit the information processed by the microcontroller to the validation software. Figure 4 presents the first communication/integration tests between the sensitive layer and the desktop, where the information transmitted by the Arduino was received and displayed on the serial terminal interface of the Arduino IDE itself.
Validation Software: The validation software developed for this project was a tic-tac-toe game. A 3x3 matrix is drawn in it, and two players will play a game by marking “X” or “O” in a specific quadrant of the matrix. The player who manages to make a sequence of three identical symbols wins the game, whether in a vertical, horizontal or diagonal line. For the development of the game, the Python programming language was used with the support of the Pygame module used for the development of the game’s graphics, as well as the serial module that was used to establish communication between the game and the interaction device through the USB port. Figure 5 shows the tic-tac-toe screen developed to validate the interaction device.
In this solution, marking the “X” or the “O” depends on the player’s interaction with the sheet of paper. To mark a symbol in the desired position, the player must press the paper in the desired quadrant with his finger so that the sensitive layer, together with the software developed for the hardware, can interpret and transmit the player’s information to the game’s interaction with the device. After receiving the information, the game updates its graphic representation and assesses whether or not the player has won the game.
The UCD concept was applied to this project in order to observe how people of the most diverse ages, levels of knowledge in technology, education level, gender, etc. can perform tasks (such as playing a game for example) without the use of a computing device. In this sense, the paper interface becomes a simple way to carry out tasks. Without the need to know how to use a digital input device, people simply play (or do some other action) in an analog form, in an interface that is “transparent” to the user.
In some situations, not yet addressed in this proof of concept, users could use the sheet of paper, for example, to draw images, in a much more intuitive way than using another digital interface.
Following the definition brought by Norman [12] that the UCD is composed of four cyclic phases: Observation, Ideation, Prototyping and Testing, this project brings more strongly the three phases, in which the tests (4th phase) were carried out only with people involved in the project. Thus, the usability testing phase was not carried out with the target audience, since the objective of this project was to present the conception of a digital “non-interface”, from the user's perspective.
In the first testing stage, a qualitative analysis was performed using Velostat as a pressure sensor. In this analysis, three devices based on the arrangement shown in Figure 3 were built. In each of the devices, one of the materials described in the methodology was used (copper tape, aluminum wire and copper foil). To perform the tests, the sensors were connected to the analog output of the Arduino, and the behavior of the sensors concerning the pressure exerted by the user was assessed according to the information processed by the microcontroller. Figure 6 shows a schematic diagram of the Velostat connection to the Arduino.
The qualitative analysis of Velostat’s behavior considered the cost and flexibility of the different materials used; the malleability, that is, the ability of the material to adapt to facilitate its use; the measurement stability and the sensitivity of the arrangement regards to the pressure exerted by the user. Each parameter described above was classified through three ranges (low, medium and high), and the main qualitative analysis results are presented in Table 1.
Low
stability
Low
Low
In table I, it is possible to observe that the cost of the materials used was generally low. Despite the cost of aluminum wire being lower than the others, the value difference does not represent a problem or makes the project’s use of copper foil or tape unfeasible. Regarding flexibility, the copper foil was inferior to the other two materials. However, the handling of copper foil during the tests (malleability) was superior to the other materials.
The stability of the measurement was also analyzed, that is, how much the value measured by the Arduino varied while the user was not interacting with the device, as well as the sensitivity of the device concerning the interaction with the user. In both cases, copper foil presented better results. Despite the lower flexibility than the other materials tested, the superiority in the other assessed characteristics and the greater precision in the measurements during the interaction with the user enable copper foil to be used in the project.
After defining the material used with the Velostat, a qualitative analysis was performed comparing the Velostat-based and brass-based sensors. In this assessment, conducted with the same arrangement described above, both sensors’ flexibility, malleability, precision and cost were evaluated. Table 2 shows the results.
In tests performed comparing Velostat and brass-based sensors, despite having a higher cost, Velostat showed greater flexibility and malleability than the brass-based sensor. Based on the tests performed, the adoption of Velostat as a sensor used in the sensitive layer of the interaction device was determined.
Having defined the materials used in the sensitive layer, nine “pads” were created using Velostat with 9cm x 3cm dimensions. These pads were placed on the back cover of a notebook, and each pad was used to map a quadrant of the 3x3 tic-tac-toe matrix. Figure 7 presents the prototype created to test the interaction device, as well as the tic-tac-toe screen running on the desktop.
In Figure 7 (A), it is possible to observe the user pressing the first quadrant of the tic-tac-toe game drawn on the notebook sheet. When the user interacts with the notebook sheet, the pads of the sensitive layer, which are positioned on the back cover of the notebook (under the sheet), detect the action performed by identifying the quadrant with which the user interacted. After the hardware interprets the user’s action, the microcontroller transmits information about the player’s interaction with the device to the game. Upon receiving the information sent by the Arduino (Figure 7(B)), the game updates its graphic representation by marking the symbol (in this case, the “X”) in the position indicated by the user and evaluates whether or not the player has won the game.
With the advancement and popularization of information technologies, as well as the growing digitization in the most diverse sectors of society, the development of technologies that enable communication between humans and computers has attracted the interest of several research groups and companies that aim to development of HCI (Human-Computer Interaction) technologies for both hardware and software that meet the most diverse needs of users.
Due to the growing demand for HCI devices, this work presented the study and development of a low-cost prototype capable of detecting user interaction with a sheet of paper. For this, a proof of concept was designed and developed, consisting of a microcontroller system and a sensitive layer formed by pressure sensors based on a compound known as Velostat that is responsible for mapping a sheet of paper and converting the user’s interaction with the sheet into information to the computer.
After carrying out the tests, it was concluded that it is possible to perform an unconventional interaction with the low-cost computer using a physical part - a sheet of paper - as a base. The pressure sensors’ cost is relatively low, using the Velostat and a pair of copper foils. Velostat is used as a resistive base material, and copper foils as a conductive material to form these sensors with specific qualities such as malleability and the ability to be molded into different shapes. So, it was possible to use them positioned under a sheet of paper. Building and using relatively simple and adaptable software, it is possible to say that the development of equipment capable of capturing the pressure generated by the user’s finger touch and transforming it into data or interactions with a computer can be applied in several areas: education, entertainment, health, etc. For the proof of concept, the tic-tac-toe game was developed and used.
In future work, several other interactions can be developed in the contexts mentioned above and conducted a usability test with potential end-users.
[9] R. Barba, Á. P. de Madrid and J. G. Boticario. Development of an inexpensive sensor network for recognition of sitting posture. International Journal of Distributed Sensor Networks, 11(8), (2015) 969237. [10] G. Gautschi. Piezoelectric sensors. In Piezoelectric Sensorics (2002) (pp. 73-91). Springer, Berlin,
Heidelberg. [11] J. F. Tressler, S. Alkoy and R. E. Newnham. Piezoelectric sensors and sensor materials. Journal of electroceramics, 2(4), (1998) 257-272 [12] Norman, Don. The Design of Everyday Things: Revised and Expanded Editionpdf by. (2013).