Millimeter-Wave (mm-wave) sensing provides an increasingly viable sensing solution for smart environments for its compact and solid-state form factor, non-intrusiveness, and low cost. While prior work in this domain has mostly focused on sensing humans - e.g., location, motion, and posture, we propose a new approach that leverage mm-wave sensing to enable tangible ubiquitous controllers such as buttons and switches. By encoding the controller state with the Radar Cross Section (RCS) of origami structures, our componentfree controllers cost less than 40 cents per unit and require virtually zero maintenance efort, while achieving long-range wireless sensing with suficient accuracies.
mm-wave sensor order controller kit fold origami calibrate controllers use controllers
• Human-centered computing → Ubiquitous and mobile computing systems and tools. Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1
Millimeter-Wave (mm-wave) sensing poses an inviting opportunity for ubiquitous sensing for being low-cost, compact, and privacysensitive – key properties that make commercial integrations possible. Recent years have seen an increasing trend of mm-wave sensing techniques featured on consumer products such as smartwatches, phones, autonomous vehicles, as well as home devices such as occupancy sensors, smart lightbulbs, and thermometers. In these systems, mm-wave sensors emit structured RF waves into user environments and decode the reflectance signals to infer user information such as presence, proximity, hand gestures, body postures, and beyond.
Meanwhile, conventional sensing techniques for ubiquitous
interactivity have several constraints that have prevented smart
environments from being widely adopted across society. Existing
controllers such as switches and buttons are often wired,
eliminating flexible deployments. Though, it is possible to enable
flexible deployments with batteries and wireless transceivers, these
components inevitably increase the maintenance and monetary
costs. In response, prior work proposed interactive sensing
techniques around computer vision [
In this work, we leverage the increasingly popular mm-wave sensing technique to build wireless interactive controllers that are designed around origami-inspired structures. While most prior work on mm-wave sensing focused on sensing direct signals from users, our system senses objects – controllers that encode user interactions into their radar cross section (RCS), which we can detect to infer user interactions wirelessly at a room-scale. Moreover, our controllers consist of only common everyday materials (e.g., cardboard papers, aluminum foil), which are low-cost and component-free, enabling users to deploy them across their environments with negligible cost or maintenance efort.
Specifically, we designed a series of origami-inspired controllers to implement conventional controls, including ones with discrete states such as buttons and switches, as well as ones with continuous states such as knobs and sliders. We built a signal processing and detection pipeline around a Frequency-Modulated Continuous Wave (FMCW) radar and conducted a series of evaluations to demonstrate our system’s feasibility. Overall, we believe our system proposes a promising approach to achieving ubiquitous interactivity. 2
This work took much inspiration from prior literature, including
recent efort in HCI that leveraged paper as interaction medium,
where researchers empowered paper with sensing [
To situate our work, we first review previous approaches proposed for room-scale interactions. We also review how RF sensing has been utilized to power interactions in HCI. 3.1
Many previous works have aimed to achieve room-scale
interactivity by deploying ubiquitous sensing modules based on various
principles. Vision-based sensing approaches have been widely
proposed [
Meanwhile, there are also wireless sensors which allow
flexible installation and portable use (e.g., TV remotes). Still, these
controllers are often battery-powered, which requires user
maintenance (e.g., battery replacement). In response, there have been
developed battery-free wireless controllers. For example, The
Peppermill [
In this paper, we propose an approach for achieving battery-free wireless controllers that consist of only ultra-low-cost everyday materials in origami forms, working in conjunction with mm-wave sensing. The proposed approach enables users to deploy controllers with negligible cost and efort to maintain. Furthermore, in the future, we expect our work to facilitate users making and replicating these paper-based controllers easily on their own, enabling DIY smart environment experience. 3.2
On the technology front, our approach is closely related to systems
that leverage RF sensing. RF sensing has been actively integrated
into applications in our lives. One of its representative uses is
in autonomous vehicles where mm-wave radar is used primarily
for detecting objects around the vehicle (e.g., other vehicles and
pedestrians) [
Closer to our work are prior systems that focus on interactivity
in HCI. For example, Soli [
Ultra-wideband (UWB) radar has been another common RF
sensing technique used for various purposes such as localizing
surrounding IoT devices and enhancing interactions with them [
As shown in these works, while most previous works have been focusing on sensing direct signals from users, we aim to sense the states of objects as controllers that encode user interaction. In detail, we propose an approach to utilizing origami-inspired structures as ultra-low-cost reflectors that can change their shapes upon user interaction. We expect that this shape change can result in unique RCS values which can be sensed remotely with a mm-wave radar. In Section 5, we explain how our origami-based controllers were designed. 4
We used the Infineon Position2Go 2, a 24GHz radar sensor development kit utilizing BGT24MTR12 RF transceiver and XMC4700 32-bit ARM® Cortex®-M4 MCU series, which costs approximately $300. The sensor has one transmitter (Tx) and two receivers (Rxs) 2https://www.infineon.com/cms/en/product/evaluation-boards/demo-position2go/ and its board size is 50 mm × 45 mm. The horizontal and vertical ifeld of view are 76 ° and 19°, and the minimum and maximum distance for sensing are 1 m and 25 m. The sensor streams raw data to a PC via USB, with 2.5 W power consumption. We utilized the oficially provided Matlab APIs to receive the streamed data and developed our detection algorithm, which will be described later in Section 7. Note in the equation below that the received power ( ), which is calculated through FFT computation on raw radar measurements, is linearly proportional to RCS of a target object () at a fixed distance to the radar ( ), with a constant scale factor of transmitted power level ( ), transmitter gain ( ), receiver gain ( ), and signal wavelength (): = 2
(4 )24
As a result, we treated the received power as an indicator of RCS in the rest of this paper. 5
Our design rationale is to create shape-changing reflector
structures that can 1) be actuated by force at a single point, and 2) result
in distinctive RCS at diferent shapes. To explore potential origami
models suitable for our purpose, we first looked into a variety of
existing works available on the Internet. We anticipated that
structures consisting of mutually perpendicular surfaces would have
high RCS values. This is inspired by the fact that corner reflectors,
which have three mutually perpendicular intersecting flat surfaces,
reflect waves directly towards the source, resulting in high RCS
values [
We found four designs that could be used for our controllers.
In general, all form-changing origami designs change their RCS
when morphing. However, as we anticipated, the four selected
origami designs feature mutually perpendicular surfaces, which
get distorted during the folding and unfolding process, resulting in
significant RCS changes. For example, we incorporated Miura-fold
[
• Button (umbrella-fold): This controller works as a push-andpull button and has discrete states (i.e., on and of). It consists of an umbrella-fold and a 3d-printed handle attached to the origami. The umbrella part opens when the handle is pushed and closes when it is pulled. • Toggle Switch (corner reflector): This controller works as a toggle switch and has discrete states (i.e., on and of). It is a corner reflector structure made of cardboard papers coated with silver origami. The three surfaces get mutually perpendicular in the on state. This structure is disrupted when the switch is turned of. Users interact with the attached 3d-printed handle to switch between the two states.
We measured RCS changes of each controller when they morphed. To do this, we first recorded signals without a controller as a base signal. Then, we placed a controller perpendicularly, 1 m away from the sensor. We recorded the signals with a controller in each state and calculated the ratio of their amplitude to that of the base signal, measured in dB. Overall, We found significant RCS changes between diferent states of the controllers. We elaborate an algorithm as to how these values are utilized for detection later in Section 7 and document the evaluation of our system in Section 8. 6
Before explaining the sensing algorithm, we describe our envisioned future scenario of how users will set up and use our proposed origami controllers (Figure 1). First, a user purchases controller tool kits of interest from a distributor. These tool kits allow the user to easily make controllers from pieces on their own. Then, the user opens a dedicated smartphone or smart speaker app to initiate the setup. The app will prompt the user to identify mm-wave sensors in their room and origami controllers within the sensing range, and start the calibration process. For calibration, he user collects a small amount of sample data corresponding to each state of the controller, and the our system is ready to use. 7
Our detection approach is based on FMCW sensing [
Our current implementation ignores frames that contain nonnegligible human body movements that interfere with our RCS sensing. To do this, we applied a simple threshold-based algorithm. In detail, we calculated the diference of two consecutive frames, say and +1, and classified the frame +1 as containing human body movements if the norm of the diference (i.e., |+1 − |) is larger than a predefined threshold. Note that the current detection algorithm requires users to exit the controller proximity after interaction. This means our system only works in an asynchronous manner, where there could be a lag between user interaction and its detection.
Once a frame is detected as not containing human body movements, the frame is processed for detecting the controller’s state. Here, we assume that the system knows the bins of the feature vector that correspond to the controller position in terms of its distance from the sensor. This information is provided during the user calibration process. By extracting the values in these bins of , we can focus on the data relevant to the controller’s state. Thus, we denote the values in these bins as a sub-vector and use it for the subsequent processing.
For detecting discrete states of the controllers, users first collect data for on and of states of the controllers in the calibration, as we described in Section 6. Then, we calculated the mean values of data collected from these two states as the thresholds to classify new frames of data after calibration.
Similarly, for detecting continuous states of the controllers, users provide sample data in the calibration for some of the data points in the detection range. We trained regression models (i.e., linear) with the data collected during the user calibration process. Note that we found the RCS change as the controllers morph to be monotonous
We examined the accuracy of the proposed algorithm. In this pilot test, we set the sensor parameters as followings: ℎ = 4 chirps per frame, = 256 samples per chirp, = 256, and = 200 MHz bandwidth. We averaged the measurements across the two Rxs for calculating and in this pilot test. We conducted the test in an open indoor space of approximately 10 2.
We first tested the accuracy of detecting discrete states of the controllers – button and toggle switch. 8.1.1 Seting. We examined the sensing accuracy in a variety of settings in terms of the controllers’ distance from the sensor, azimuth angle, and angle of incidence. Figure 4 (left) shows the locations we tested and Figure 4 (right) illustrates the angles of incidence we tested. For each controller, we first placed it in the mm-wave RF beam direction with 1 m interval up to 5 m. Then, we fixed the distance to 3 m and changed the azimuth angle with 15 degrees interval up to 45 degrees. Lastly, we fixed the distance to 3 m and the azimuth angle to 0 degrees, and changed the angle of incidence with 15 degrees interval up to 45 degrees.
For each placement pattern, we first placed our controller and conducted calibration, as described in Section 7. After the calibration, we changed the state of the controller to be on and of repeatedly, five times each, while we recording the output of the algorithm each time. 8.1.2 Result. Figure 5 shows the accuracy, each corresponding to when we changed distance (top), azimuth angle (center), and angle of incidence (bottom). The toggle switch showed a stable detection accuracy over the conditions. On the other hand, the accuracy for detecting the button’s state gradually decreased as it was placed far from the sensor or its perpendicularity lost (i.e., as we changed the azimuth angle or angle of incidence). Overall, the high detection accuracy confirmed the validity of using our origami-based reflectors as discrete controllers. 8.2
Next, we tested the performance of detecting continuous states of the controllers – knob and slider. 8.2.1 Seting. We first recorded signals without a controller placed in the environment and obtained . Then, we placed a controller 1 m away from the sensor board perpendicularly so that both the azimuth angle and angle of incidence were 0°. We then changed the expansion level of the controller from 0% to 100% with 25% interval, while we recording the corresponding signals . We calculated and plotted the ratio of the amplitude of to in dB. We also trained a linear regression model and calculated the mean absolute percentage error (MAPE). 8.2.2 Result. Figure 6 shows the measured ratio of the amplitude of to in each of the controller’s expansion level. As expected, the values gradually increased as the controllers were expanded. The MAPE for each of the controllers are 33.2% (knob) and 28.6% (slider), respectively. The results clearly showed the correlation between the expansion level and RCS, with which our regression models can be easily trained for using origami-based reflectors as continuous controllers. However, our proof-of-concept regressor implementation did not yield high accuracies due to large deviations when controllers were expanded to certain levels (e.g., 75%). We suspect this issue was caused by fabrication defects, which we plan to further investigate and make improvements in our future work. 9
As we discussed in Section 1, our low-cost controllers can be a promising approach for ubiquitous interactivity. For example, average homeowners can easily deploy the controllers into their environments and connect them with various IoT applications, such as light, music player, TV, air conditioner, etc. Moreover, the controllers can be installed in public places such as museums, hospitals, restaurants, and buses, replacing exiting controllers that are mostly wired and powered. Overall, we believe the advantages of our approach being ultra-low-cost, wireless, and battery-free facilitate
There are some directions to further refine our approach. First, the accuracy and robustness can be improved by adopting better fabrication process. For example, adding linings on top of basic origami structures could yield more programmable shape changing of the continuous controllers. We could also add protective coatings to mitigate degradation of origami structures over time.
Secondly, we will expand the origami design set. In this paper, we demonstrated four designs, but considering the abundant lfected signal to that of the base signal when controllers were expanded from 0% to 100% with 25% interval (top: knob, bottom: slider). Blue lines represent the fitted linear regression model while orange dots correspond to the measured data points. origami structures found, we believe there are many other possible designs. We would like to run simulations to estimate RCS changes of origami structures for comprehensive exploration and optimization.
Third, for achieving the ubiquitous interactivity through our controllers, it is demanded to enable the system to detect spatial information of controllers. We would like to utilize beamforming and Angle of Arrival estimation with radars that have multiple transmitter and receiver antennas to increase spatial resolution. This improvement on spatial resolution will allow a user to deploy multiple controllers in the environment. Additionally, improved spatial resolution will help locate users more precisely, which could mitigate the current limitation that the users must be out of the controller proximity for detection.
Lastly, current controllers are relatively large in comparison to conventional ones. In future work, we will miniaturize the controllers with better fabrication techniques based on human-machine ergonomics. Specifically, we will apply automated fabrication techniques such as laser cutting combined with vacuum forming, which will enable us to develop more complicated origami structures. At the same time, we would like to further investigate the interaction space where users can easily assemble controllers from basic material primitives, as we discussed in Section 2 and Section 6. 11
In this paper, we proposed a novel approach to achieving ubiquitous interactivity: ultra-low-cost (less than 40 cents), wireless, battery-free controllers made of origami in concert with mm-wave sensing. We demonstrated four controller designs (i.e., button, toggle switch, knob, and slider). These controllers change their RCS significantly upon user interaction, which can be detected remotely by mm-wave sensing (e.g., FMCW). Our pilot test demonstrated the feasibility of the proposed approach. We believe that our work demonstrates a novel technique for ubiquitous interactivity, and will greatly facilitate users’ DIY of future smart environments.