This paper is devoted to the development of the ontology-driven standalone scientific visualization station based on a single-board microcomputer with custom tangible user interface. Such a station can be used as a powerful demonstration tool in various scenarios including interactive museum exhibitions. According to the approach proposed, the particular instance of a software scientific visualization system is generated automatically by a high-level platform SciVi that was been developed earlier. Previously, ontology-driven software generation mechanisms within SciVi were tested on the firmware generation for the microcontroller units. Currently we present a generalization of this technique to the case of systems on chips like Raspberry Pi or Orange Pi. Data preprocessing and rendering capabilities of SciVi are reused without modifications from the previous stages of development, while the new mechanisms of taking into account the specifics of systems on chips software and hardware organization are introduced via extending the appropriate SciVi ontologies. The generalized technique is tested in practice by creating a set of interactive museum items for the “Transmutations” exhibition within Kidsmuseum, branch of Perm Regional Museum.
Particular tasks of presenting scientific data often require standalone scientific
visualization stations, which are equipped with the monitor, rendering system and custom
user interface. Normally, regular computers are sufficient for this, however sometimes
specific circumstances prevent this traditional scenario. One of these specific cases are
interactive mobile museum exhibitions within a Smart Museum concept [
Traditional computers do not meet at least the criterion (6). The criterion (3) is met only partially, since shutting the computer down just by cutting off the power can harm the data on the disks and cause severe system failure. But this criterion is indeed important, since preparing the exhibition in the mornings and shutting everything down in the evenings are the tasks of the museum hall keepers, who need to do it as fast as possible, and it would be problematic for them to perform special operations on individual exhibits.
To meet all the mentioned criteria at once, we propose using systems on chips (SoCs).
SoCs became a popular form of portable computers [
As seen in the literature, SoCs are widely used in the scientific and cultural projects.
For example, the popular scientific visualization library VTK is ported to Raspberry
Pi [
We propose using high-level scientific visualization platform SciVi [
Scientific Visualization System on a Chip with Tangible User Interface 3
The motivation for this work was a need to create several interactive museum items for the “Transmutations” exhibition within Kidsmuseum, branch of Perm Regional Museum. Since it was a real project involved in the Perm Regional Museum, within this work we were able to do both a theoretical research and the tests of the proposed approach on the practical use case. The interactive items were successfully created and integrated in the “Transmutations” exhibition available for visitors in Perm. 2
During the previous research we developed a multiplatform adaptive scientific
visualization platform SciVi based on the ontology engineering methods and means that was
used in various application domains to present different scientific data in a
comprehensive observable form [
The interactive museum exhibit consists of two main parts: museum item and visitor interface. Museum item is the most important part that has its own value and in practice can be exposed even without interactivity. But the visitor interface spices up the museum item with the new visitors’ experience beyond regular watching, allowing not just passively look at the item, but actively communicate with it. Smart Museum concept assumes turning the regular museum exhibits into the interactive ones to bring the museum visiting to the new qualitative level, not just exposing the items, but deeply immersing the visitors into the exhibition.
While creating the museum items (even the multimedia ones) is the task familiar to the museum research workers, assembling and tuning the visitor interface is hard due to the lack of the corresponding high-level tools for that. Therefore interactive exhibits are still an issue in many museums.
SciVi scientific visualization platform provides efficient tools to automate all the
main steps of creating the interactive exhibits [
All the above steps are driven by ontologies, which ensure flexibility and extensibility of SciVi. The configuration step (1) is supported by the built-in data flow diagram (DFD) editor that enables the user to declare particular data processing, visualization and TUI functioning algorithms in an intuitive visual form.
The previous work of creating IoT-based museum exhibits was been focused on
utilizing programmable MCUs [
In the present work we tackle the problem of creating low-budget standalone visualization system that does not require any additional digital infrastructure around it (no kiosks and no network). 3
The distinctive feature of the present work is that we propose to merge TUI and rendering hardware into a single device. In fact, we propose building scientific visualization system on a chip equipped with a sensor-based TUI. The main requirement to the chip is a support of graphics rendering hardware acceleration. The most popular single-board computers capable of data processing and rendering are Raspberry Pi and Oragne Pi. While they are very similar, each model has its own features and tuning nuances. But thanks to the ontology-driven architecture, there is no problem in extending SciVi in a way it takes into account platform-dependent particularities during the process of generating needed TUI- and rendering software modules.
On the MCUs, SciVi generates the firmware as a solid binary image comprising all
the functionality (and, if necessary, defines fuse-bits to fine-tune the MCU functioning).
The features of MCUs and related hardware modules are described in the ontology of
electronic components that is a part of SciVi knowledge base [
First, we decided to use Web-based applications as a target software format, because it would have level the differences between Raspberry Pi and Orange Pi. But the testing showed, that the rendering performance of WebGL is insufficient to deliver smooth animations in 3D scenes. The scene associated with the interactive exhibits we created (see the section 4) contains ca. 5 000 vertices connected in ca. 10 000 polygons and is rendered to the resolution of 1024 600 pixels. Google Chromium (the default browser on most OS for SoCs) was able to maintain the refresh rate of just 9 frames per second (FPS). In contrary, the native application written in C++ utilizing OpenGL 3.3 (that
Scientific Visualization System on a Chip with Tangible User Interface 5
is emulated on top of OpenGL ES 3 on SoC) ensures 16 FPS. Both refresh rates are
quite low, so further optimizations of rendering are required. However, in terms of user
experience, 16 FPS lay above the border, when the average human starts to recognize
fluent motion instead of individual images, while 9 FPS are below that line [
The proposed stack of technologies is shown in the fig. 1.
SciVi Renderer is a standalone native application, which source code is generated by SciVi in C++ according to the particular settings made by the user. It utilizes OpenGL graphics rendering API (available on most OS for SoCs via Mesa library) and various device access libraries to receive / transmit data from / to TUI hardware (e. g. sensors and actuators connected via to the SoC via I2C, SPI, UART, OneWire, etc.).
Significant difference of SoCs compared to MCUs is the presence of full-fledged OS (e. g. Raspbian or Armbian) that, in turn, brings concepts of users, console sessions and permissions. To meet the requirement of fast startup (mentioned in the Introduction), auto-login should be switched on (which is a default behavior of SoCs), as well as the autostart of target application and tuning of the device permissions (for example, to access SPI bus that by default requires root privilege level) should be set up properly.
Fast shutdown is still an issue, because cutting off the power is generally unsafe for the SoCs filesystem. So, ideally hardware shutdown button should be mounted that will cause the execution of poweroff command. However, even if the special button cannot be installed and the SoC goes down without involving proper cleanup mechanism, the probability of failure is fairy low in case of visualization system. This is because the visualization system does not perform a lot of disk write operations, so the filesystem normally does not run out of sync and sudden halt should not bring it into the inconsistent state. To further reduce the failure probability, swapping is disabled and root filesystem is mounted in the read-only mode. Should the failure ever happen, it is relatively easy to repair the SoC since all its data are on SD card that, in turn, is easy to back up and restore.
Currently, SciVi automatically generates the source code of the Renderer application along with a configuration shell script to make it easy to build and install the application on the target SoC (via the standard ./configure, make and make install command sequence). The SciVi infrastructure allows cross-compiling the software directly for the target platform. This feature is not yet set up for SoCs, but it is a matter of one-time extending the special compiler’s ontology and tuning the Dockercontainer with the needed compilers installed inside it. We plan to do it as an upcoming development step. It will then be possible to generate an installation package for the SoC’s OS, so the deploy will be even easier. 4
To test the proposed approach we created a set of museum items for the
“Transmutations” exhibition within Kidsmuseum, branch of Perm Regional Museum. The most
significant one demonstrates so-called Allen’s ecogeographical rule [
The TUI of the exhibit created is shown in the fig. 2. The visitor can choose the latitude with a special handler on the simple Earth model, while the built-in visualizer generates an image of average mammal leaving in the corresponding climate zone. The 3D avatars of “most-polar” and “most-equatorial” animals are shown in the fig. 3. The corresponding key 3D models are created from scratch by one of the paper’s authors using Blender 3D editor and stored in PLY format. Both models have the same topology (ca. 5 000 vertices connected into ca. 10 000 triangles) enabling trivial linear morphing between them. The intermediate forms are automatically calculated based on the key models and visualized by SciVi in real time.
We utilize Orange Pi PC2 as a SoC with Armbian GNU/Linux as its OS, 7 inch 1024 600 pixels touchscreen connected to the SoC via HDMI, external analog-todigital converter (ADC) MCP3201 connected via SPI (unfortunately, Orange Pi PC2 has no built-in ADC on board) and potentiometer connected to the ADC. The knob of potentiometer is attached to the handler on an Earth model. The total cost of hardware
Scientific Visualization System on a Chip with Tangible User Interface 7 is about $130 (including the delivery of the electronic components), which is fairly low compared to the other options of reaching similar functionality.
SciVi Renderer is configured to poll the ADC and use the obtained value as a transition quotient to morph the animal model from the polar form to the equatorial and back to polar. The morphing result is rendered using the fake global illumination based on the material capture technique. The result image is showed to the visitors via the connected touchscreen, wherein the touch moves are mapped to the orbit camera rotation, so the visitors can change not only the morphing phase, but also the point of view to the 3D model. 5
Previously we created the scientific visualization system SciVi that is able to leverage the creation of interactive museum exhibits based on MCUs and TUI. In the present work we generalized our technique from MCUs to SoCs, which can be used as standalone scientific visualization stations equipped with TUI. Utilizing this generalized technique, we created a set of interactive exhibits based on the Orange Pi PC2 singleboard microcomputers, which leverage both maintaining the TUI and rendering the result images. The created interactive museum items are presented in the “Transmutations” exhibition within Kidsmuseum, branch of Perm Regional Museum, supporting the Smart Museum concept within the cultural institutions of Perm Region. Thereby, SciVi system was applied in practice solving a real Smart Museum task.
The future work includes optimizing the rendering mechanism to increase FPS rate and setting up the cross-compilation of the rendering software modules for SoCs within SciVi.