In this paper BrailleIO, a small .NET framework for developing two-dimensional tactile applications, is presented. It offers general features for displaying tactile information and for interaction. BrailleIO includes hardware abstraction, window and visualization features as well as basic interaction functions, such as panning and zooming on different content types. Information visualization can be organized in several independent screens that can be divided into multiple areas, having a full box model. Interaction can be realized via hardware keys of the used device or by basic gestures if the device is touch-sensitive.
Information technology is omnipresent in our daily life and we accept that because of so many benefits. With the increasing possibilities of information technology and the devices available everywhere getting more powerful, presentation of data can get even more user friendly. In our visual world this means that rich applications and big data get Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s). more and more visual to expose their secrets. This way, in addition to purely text-based systems interactive visual and graphical user interfaces pose a big benefit for the most users but leave also some behind.
Visually impaired users rely on a substituting alternative way to receive the information that is presented in a visual way. This mostly includes spatial information, too. While the visual world is entering the third dimension in information presentation now, visually impaired people are getting access to the second digital dimension with the help of the arisen smart mobile devices with touch screens. These are well usable tools for easy and self-determined interaction. But that is only audible in most cases. The step to a real tactile and, therefore, graphical interaction is still underrepresented.
Because of so much graphical information, the growing request for two-dimensional dynamic tactile displays for blind computer users results in an increasing number of available ideas and products. Several different designs and techniques for refreshable tactile displays exist.
Many prototypes for building tactile displays were
developed, using different ideas to generate a dynamic tactile
display [
There are not only prototypes already available. Completely
functional systems built for solving application-specific
problems are available. The Mimizu system, using the
DotViewPin-Display [
Within the HyperBraille project the touch-sensitive device
called BrailleDis was developed. Two generations of
pinmatrix devices exist, varying in body dimensions, amount
of function keys and touch sensor resolution [
As mentioned before, those products are often delivered with their own proprietary software to visualize or retrieve information. Developing a new software product, addressing those output devices, is a time-consuming and expensive task. It could be hard for an application developer to convey information through the original software on the output device that is not related to the original purpose the system was built for.
In the following the framework BrailleIO is presented. Its goal is to lower the obstacles and simplify the start for developing applications for two-dimensional tactile displays by giving basic tools for tactile user interface design and hardware modeling.
To reduce time and costs for developing a software application that uses a tactile display as output device, the framework BrailleIO is developed. It is a .NET 4 based software framework written in C# and, therefore, only usable for Windows operating systems.
A main goal of BrailleIO is to give basic implementations
for common needed functions and structures when building
a tactile application. The barrier to enter should be as low
as possible. The framework enables developers to achieve
quick success without starting to solve fundamental
problems in the first place. The framework should give
possibilities for structuring information, presenting graphics and
text - preferably as Braille - and enable interaction.
Successfully tested concepts, proven in previous projects [
Therefore, it seems to be necessary to idealize and abstract the used hardware for input and output as well. This means, <<singelton>>
BrailleIOMediator - views + AdapterManager
1 <<interface>> IBrailleIOAdapterManager # Adapters * * *
- view_ranges *
<<interface>> IBrailleIOAdapter
<<abstract class>> AbstractViewBoxModelBase + ViewBox + ContentBox - content - content renderer 1 real hardware implementation application developers can use basic features without any knowledge of the used device or get access to required information through the framework.
The framework is divided into two main parts. The first part handles basic graphical elements and the generating of output. The second is responsible for managing, modeling and implementing the real used hardware devices. As shown in Figure 1, the two parts are linked together by a so-called mediator (BrailleIOMediator). The mediator handles the visibility and the rendering of the tactile user interface by managing different views which are building together the resulting tactile output. Every important part of the framework should be accessible through this component. All modeled hardware devices - called adapters - have to be registered to the mediator. It distributes the generated tactile output to all registered and activated devices. The access to interaction events caused by user interaction with a device is not handled by the mediator. An interaction handling process has to be connected directly to every single registered adapter, which can be accessed through the mediator.
The framework offers elements to build and structure tactile graphical user interfaces. The basic object for that is the so-called screen (BrailleIOScreen). As shown in Figure 1, a BrailleIOMediator can hold an unlimited set of screens. Normally, only one screen should be active at once and, therefore, will be rendered and sent to the displaying device. With this multiple screen metaphor it is possible to implement and provide several different views or applications simultaneously. The user can then switch easily between the different screens. A screen itself cannot have any content directly. view-range screens ViewBox content
As the arrangement of a two-dimensional tactile output into
several regions can support the bi-manual exploration of
information [
The view-ranges are stored at the corresponding screen in an internal list in the order they were added. This list represents a hierarchical, linear structure that has direct influence on the rendering process. The position inside the list corresponds to a common z-index mechanism. This means an overlapping view-range in a further position inside the list overwrites an overlapped view of an earlier position. In addition, the definition of a layer order, not corresponding to the list order, is also possible by setting the z-index property for the view-ranges. View-ranges are always opaque. That means the underlying content will be erased by an overlapping container, even if it is empty.
View-ranges have a full box model corresponding to the CSS
box model for web sites [
With these tools a screen can be organized in regions with
different content which can be separated by significant
tactile features, such as dividing space, separating lines or
marking frames. The division of the available display space seems
not always to be reasonable, especially on small displays
such as the GWP [
= ContentBox ContentWidth t h g i e H t n e t n o C be necessary to mark them as divided. Therefore, the box model can be used even if only one of its parameters is applied to create a border or a gap.
A view-range is mainly characterized by the size and the position on a screen. These parameters are stored in the ViewBox member variable. In the ViewBox, the outer dimension of the view-range is defined as well as the x and y position in relation to the top left corner of the corresponding screen. The remaining visible area of a view-range is called ContentBox, which has the dimension of the ViewBox subtracting the several box model sections (see Figure 4). All dimensions and positions used as parameters in this framework are defined in pins. Therefore, the definition of sizes and points is depending on the resolution of the used output device.
For handling oversized content and enabling the access to content that doesn’t fit in a view-range, a panning concept is realized. With the parameter OffsetPosition the content can be moved under the view range frame. The offset position defines the start position of the content in relation to the left corner of the ContentBox. If necessary, simple tactile scrollbars are added and rendered to give feedback to the user about the position inside the content. The scrollbars consist of a continuous line with an adjacent indicator which is three points long and one point thick. To keep the scrollbar recognizable a one pixel space in the direction of the content is set up. Therefore, the scrollbars only reduce the available content space by three pins.
A view-range can get several types of content to display. As shown in Figure 5 the basic data structure, to which every other content type will be transformed, is a two-dimensional Boolean matrix (bool[,]). A true-value represents a raised pin on the tactile display, a false-value a lowered one. Other content types are text, which is rendered in Braille or as an image, as well as pictures, which are rendered as binary images. With a free definable threshold for the lightness of a pixel, darker pixels will be set to raised pins and lighter ones to lowered pins.
In the end the framework allows for setting any other type of content. However, this requires the definition of a specialized renderer for the given content type, implementing the IBrailleIOContentRenderer interface, converting the content to a usable Boolean matrix.
BOOL Nearly all renderer are hookable. This means, an extension or an user of the framework can get access to the renderer functionality by registering a hook. This hook will be called before the renderer starts his work and after the renderer has rendered the result. A hook has the opportunity to manipulate all function parameters in the beginning of the rendering. Furthermore, the redering result can be modified before it will be returned and sent to the output devices. This gives programmers the power to use already implemented standard renderer and adapt or extent them to their needs, without implementing an own renderer.
As mentioned before, the abstraction and modeling of hardware is an essential part of the framework, too. Therefore, the first idea was to identify basic properties and features. A construct for modeling and mapping a real hardware interface was developed. In this, special properties are defined, for instance, a proposed image refresh rate, the number of pin-rows and columns or the availability of buttons or touchsensitivity of the hardware device.
A specific hardware device still needs to be mapped to the proposed framework. For this purpose, an adapter has to be provided that implements the IBrailleIOAdapter interface and generates a unique device representation as an object of type BrailleIODevice (see Figure 6). This has to be done once for every new hardware device that should be used with the BrailleIO framework.
The implemented adapter is responsible to achieve proper
access to the hardware device. This means the adapter has
the task to create, open and hold a channel to the
hardware device as long as needed and to bring the
standardized output-matrix on the device. At the same time, the
adapter implementation has to map the proprietary and
device-specific interactions and events to the idealized data
models which are expected by the framework. If a device
has hardware keys, they have to be modeled, too. Nine
basic function keys are defined as a minimal set for a sufficient
interaction on a touch-sensitive tactile graphic device (see
Figure 7). The key set consists of four navigation keys and
two zoom buttons to interact with oversized, graphical or
zoomable content, such as images. Furthermore, two
interaction buttons for approval and refusal are defined. Finally,
a special key for touch-sensitive devices seems to be
necessary to start and stop a gesture input to avoid midas touch
effects [
BrailleIODevice + string AdapterType + int DeviceSizeX + int DeviceSizeY + bool HasKeys + bool HasTouch + String Name + int RefreshRate 1
<<interface>>
IBrailleIOAdapter + bool Connected + BrailleIODevice Device + int DpiX + int DpiY ... + void Synchronize Events: + errorOccured + initialized + inputChanged + keyPressed + keyStateChanged + pinStateChanged + touchValuesChanged ... + bool Synch ...
Events: ...
BrailleIOAdapter_
ShowOff Some general device events are proposed as well (see Figure 6). These events should give feedback about availability and changes in key-, touch- or pin-states as well as the occurred errors. In these events the sending device and the original raw device data of the event are enclosed and sent to all registered listeners. The listener can decide if he wants to handle the generalized data, such as the general buttons, or the original device data, e.g. additional keys that are not mapped to one of the general keys.
Two real hardware-specific adapter implementations were
built for different types of the BrailleDis series, named the
BrailleDis 9000 [
A set of basic functions for interaction are implemented. As mentioned before, a first assignment of keys with functions is proposed by the adapter implementation (see section 2.3), but not realized by the framework. The linking between buttons and functions has to be done by the application developer. Nevertheless, all necessary functions are available in a basic implementation.
device IBrailleIOAdapter
OK ESC GST + +- GST BrailleDis 9000 ESC
Zooming can be realized by setting the zoom property of the corresponding view-range. The framework will handle the zoomed rendering if possible. Panning operations can be connected to the OffsetPosition property of a viewrange (see section 2.2). By setting the offsets to negative values the content can be moved below the visible area of the view-range’s ViewBox (see Figure 4). Changing the yoffset realizes a vertical and changing the x-value a horizontal scrolling. The offset can be changed freely. Several functions of the abstract base class implementation of the view-ranges (see Figure 1) offer offset manipulation in an easy manner.
A basic gesture recognizer is included. It recognizes a
number of basic gestures, such as pointing gestures (tab), swipes
(line), pinch, circle (half and full) and three finger drag
operations (compare [
Inversion and threshold adaption are also part of the framework for image handling. For instance, these features are useful for exploring images. Small sinks - regions of lowered pins inside an area of raised pins - can be transformed to a raised pin area for a better detection by inverting the presentation. The adaptation of the threshold level for the binary image conversion allows for adjusting the presentation to the given context. Especially if very light or very dark images are presented, the adjustment of the threshold is necessary to make structures visible at all.
In the following a small guidance about how to set up a project with the BrailleIO framework in a few steps is given (compare Figure 9). register gesture recognizer adapt content or views on user input define size, position, style and visibility Adapter real or virtual set up devices register to device events build screens add
add
register devices set the active device
choose active screen
set content After setting up the device that should be used for in- and output, it has to be registered to the AdapterManager related to the BrailleIOMediator. A registration to the events thrown by the device has to be done. If the device is touch sensitive, the optional basic gesture recognizer can be connected to interpret touch inputs. An unlimited number of devices can be registered but only one device can be marked as active. This device is used as main output. Other devices can be used as output by setting the Synch property of the AbstractBrailleIOAdapterBase. This leads the mediator to mirror the tactile result to these devices. With this mechanism the ShowOff adapter can be used, for example, as debug monitor beside a real tactile matrix device. At least one view-range has to be defined for displaying content. It has to be configured with a position, size and the optional box model. The view-range can be added to the mediator directly or it can be combined with other viewranges in a screen which has to be added to the mediator. An unlimited amount of screens are allowed. After setting a screen as active or visible it will be displayed on the output device. Every view-range has to get its own content which will be rendered and presented. The views or the contents can be changed, for example, on user interactions reported from the device events.
The framework BrailleIO was presented. It enables a fast
and easy entry into building applications on two-dimensional
tactile displays for visually impaired users. With the screen
and view-range constructs a proper information
organization and simultaneous reception is possible. The framework
also proposes a hardware abstraction for pin-matrix devices
including general hardware keys and function binding.
The framework is used, for example, as groundwork for a
tactile graphic production workstation called Tangram
workstation [
Several open issues have to be solved and innumerable improvements are conceivable. At this point, there is no way to map infinite further hardware keys to the generic key construct. A continuous numbering of further keys could be a solution to overcome the fall back to the proprietary naming of these buttons. In this context, the implementation of more concrete hardware adapters would proof the concepts and the portability beyond the BrailleDis devices. It is also unclear how to handle output devices with a significantly higher resolution than 10 dpi, which will lead to problems on rendering Braille. The resolution of an used device is available in his specifications and therefore has to be checked and used while rendering resolution dependent content.
The next big and challenging step is the implementation
of a powerful Braille renderer. Currently, strings are
rendered with an equidistant Braille font as an image and sent
to the output devices. There is no way back from the
rendered image to the original given text. This is necessary, for
example, for a controlled audio output of touched Braille
elements. For the near future, it is planned to build a
renderer based on the free transcoding project liblouis [
BrailleIO is realized as open source project. One big advantage is, if some functions or concepts are missing or are not optimal, the open source approach enables anybody to take part and improve, change or complete the project with further and better ideas. The framework is freely available through https://github.com/TUD-INF-IAI-MCI/BrailleIO.
I thank Heinrich Kuhlmann for his work on BrailleIO, Dr. rer. nat. Michael Schmidt for his implementation of a basic gesture recognizer and Denise Prescher for her work on the interaction design. The Tangram project, in which BrailleIO is developed, is sponsored by the Federal Ministry of Labour and Social Affairs (BMAS) under the grant number R/FO125423.