=Paper=
{{Paper
|id=Vol-1324/paper_4
|storemode=property
|title=BrailleIO - A Tactile Display Abstraction Framework
|pdfUrl=https://ceur-ws.org/Vol-1324/paper_4.pdf
|volume=Vol-1324
|dblpUrl=https://dblp.org/rec/conf/tabletop/Bornschein14
}}
==BrailleIO - A Tactile Display Abstraction Framework==
BrailleIO - a Tactile Display Abstraction Framework Jens Bornschein Human-Computer Interaction Research Group Technische Universität Dresden Nöthnitzer Str. 46, 01187 Dresden, Germany jens.bornschein@tu-dresden.de ABSTRACT more and more visual to expose their secrets. This way, in In this paper BrailleIO, a small .NET framework for de- addition to purely text-based systems interactive visual and veloping two-dimensional tactile applications, is presented. graphical user interfaces pose a big benefit for the most users It offers general features for displaying tactile information but leave also some behind. and for interaction. BrailleIO includes hardware abstrac- tion, window and visualization features as well as basic inter- Visually impaired users rely on a substituting alternative action functions, such as panning and zooming on different way to receive the information that is presented in a visual content types. Information visualization can be organized in way. This mostly includes spatial information, too. While several independent screens that can be divided into multiple the visual world is entering the third dimension in informa- areas, having a full box model. Interaction can be realized tion presentation now, visually impaired people are getting via hardware keys of the used device or by basic gestures if access to the second digital dimension with the help of the the device is touch-sensitive. arisen smart mobile devices with touch screens. These are well usable tools for easy and self-determined interaction. Categories and Subject Descriptors But that is only audible in most cases. The step to a real D.3.3 [Programming Languages]: Language Constructs tactile and, therefore, graphical interaction is still underrep- and Features—classes and objects, data types and struc- resented. tures, frameworks; H.5.2 [Information Interfaces and Presentation]: User Interfaces—haptic I/O, input devices Because of so much graphical information, the growing re- and strategies, Screen design, User interface management quest for two-dimensional dynamic tactile displays for blind systems (UIMS); K.4.2 [Computers and Society]: Social computer users results in an increasing number of available Issues—assistive technologies for persons with disabilities ideas and products. Several different designs and techniques for refreshable tactile displays exist. General Terms Many prototypes for building tactile displays were devel- Documentation, Design oped, using different ideas to generate a dynamic tactile dis- play [11, 12, 15]. Different approaches were used to gener- Keywords ate tactile stimuli and arrange them in matrixes with more Framework, tactile display, two-dimensional Braille display, or less resolution. The techniques range from mechanical, pin-matrix device, hardware abstraction, standardization, electromagnetic, piezoelectric, or pneumatics, hydraulic or tactile interaction, software development, tactile user inter- shape memory alloy based actuators up to surface changing face polymers [8]. 1. INTRODUCTION There are not only prototypes already available. Completely Information technology is omnipresent in our daily life and functional systems built for solving application-specific prob- we accept that because of so many benefits. With the in- lems are available. The Mimizu system, using the DotView- creasing possibilities of information technology and the de- Pin-Display [5], is a stylus-pen based drawing workstation. vices available everywhere getting more powerful, presenta- The pin display consists of 1,536 pins in a 48 x 32 pin-matrix tion of data can get even more user friendly. In our visual with inter-pin space of about 3 mm and a display size of 144 world this means that rich applications and big data get by 96 mm. The application allows for painting and erasing freehand structures or the presentation of small images. Permission to make digital or hard copies of part or all of this work for With the GWP system [1], a small pin-matrix display of 16 x personal or classroom use is granted without fee provided that copies 24 dots is used. The pin-matrix has a 3 mm pin raster, too. are not made or distributed for profit or commercial advantage and that The device is portable and has 15 function keys, allowing copies bear this notice and the full citation on the first page. Copyrights for for zooming, navigation and function calling. The GWP third-party components of this work must be honored. For all other uses, device is used to give access to mathematics, especially to contact the Owner/Author. Copyright is held by the owner/author(s). the graphical part. TacTT ’14 , Nov 16 2014, Dresden, Germany. Within the HyperBraille project the touch-sensitive device <> * BrailleIOScreen < > BrailleIOMediator AbstractViewBoxModelBase called BrailleDis was developed. Two generations of pin- * - view_ranges - views + ViewBox matrix devices exist, varying in body dimensions, amount + AdapterManager * + ContentBox - content of function keys and touch sensor resolution [13, 6]. The BrailleIOViewRange - content renderer BrailleDis devices have a ten dpi dot matrix of 60 x 120 1 * pins, resulting in a tactile area of 150 by 300 mm with 7,200 < > < > pins. The HyperReader software [10], using the BrailleDis IBrailleIOAdapterManager IBrailleIOAdapter real hardware implementation * * 1 devices for in- and output, was built to get access to desktop # Adapters applications, such as office or web browsers on Windows sys- tems. Thereby it offers different views on the content while Figure 1. Basic structure of BrailleIO. keeping spatial layout information [9]. Within the Hyper- Reader a region concept for tactile user interfaces is used and evaluated as a well usable tool for structuring informa- application developers can use basic features without any tion [7]. The architecture of the HyperReader software is knowledge of the used device or get access to required infor- built to enable extensions and adaptations for different ap- mation through the framework. plications. The software framework is big and hard to learn. Furthermore, it is not free to use and cannot address other tactile displays for in- or output. 2.1 General Overview - Structure The framework is divided into two main parts. The first part handles basic graphical elements and the generating of As mentioned before, those products are often delivered with output. The second is responsible for managing, modeling their own proprietary software to visualize or retrieve in- and implementing the real used hardware devices. As shown formation. Developing a new software product, addressing in Figure 1, the two parts are linked together by a so-called those output devices, is a time-consuming and expensive mediator (BrailleIOMediator). The mediator handles the task. It could be hard for an application developer to con- visibility and the rendering of the tactile user interface by vey information through the original software on the output managing different views which are building together the device that is not related to the original purpose the system resulting tactile output. Every important part of the frame- was built for. work should be accessible through this component. In the following the framework BrailleIO is presented. Its All modeled hardware devices - called adapters - have to goal is to lower the obstacles and simplify the start for de- be registered to the mediator. It distributes the generated veloping applications for two-dimensional tactile displays by tactile output to all registered and activated devices. The giving basic tools for tactile user interface design and hard- access to interaction events caused by user interaction with ware modeling. a device is not handled by the mediator. An interaction handling process has to be connected directly to every sin- 2. THE FRAMEWORK gle registered adapter, which can be accessed through the To reduce time and costs for developing a software applica- mediator. tion that uses a tactile display as output device, the frame- work BrailleIO is developed. It is a .NET 4 based software 2.2 Tactile User Interface framework written in C# and, therefore, only usable for Win- The framework offers elements to build and structure tac- dows operating systems. tile graphical user interfaces. The basic object for that is the so-called screen (BrailleIOScreen). As shown in Fig- A main goal of BrailleIO is to give basic implementations ure 1, a BrailleIOMediator can hold an unlimited set of for common needed functions and structures when building screens. Normally, only one screen should be active at once a tactile application. The barrier to enter should be as low and, therefore, will be rendered and sent to the displaying as possible. The framework enables developers to achieve device. With this multiple screen metaphor it is possible quick success without starting to solve fundamental prob- to implement and provide several different views or appli- lems in the first place. The framework should give possi- cations simultaneously. The user can then switch easily be- bilities for structuring information, presenting graphics and tween the different screens. A screen itself cannot have any text - preferably as Braille - and enable interaction. content directly. Successfully tested concepts, proven in previous projects [9, view-range screens 7], are used to build tactile user interfaces and bring them to a wider field of users. The reuse of those concepts can help to improve quality of tactile applications and can start to bring consistency in structure as well as in look and feel of such applications. Consistency is important, especially for visually impaired users, because the user can rely on known structures and interaction paradigms [2]. This can improve learnability as well as efficiency and can reduce errors at the ViewBox same time. content Therefore, it seems to be necessary to idealize and abstract the used hardware for input and output as well. This means, Figure 2. Screens and view-ranges. width (ViewBox.Width) OffsetPosition.X height (ViewBox.Height) ContentHeight OffsetPosition.Y visible area visible content of ViewBox = ContentBox Padding Border Margin ContentWidth Figure 3. Box model of view-range and relation to ViewBox. Figure 4. Relation between ViewBox, ContentBox and content of a view-range. As the arrangement of a two-dimensional tactile output into several regions can support the bi-manual exploration of information [9], the framework should allow this feature, be necessary to mark them as divided. Therefore, the box too. Therefore, screens can be divided into several content model can be used even if only one of its parameters is ap- areas (see Figure 2). Those areas are called view-ranges plied to create a border or a gap. (BrailleIOViewRange). A screen consists of an unlimited number of view-ranges (see Figure 1). Every view-range can A view-range is mainly characterized by the size and the be independently filled with content, placed on the screen, position on a screen. These parameters are stored in the sized, activated or deactivated. ViewBox member variable. In the ViewBox, the outer di- mension of the view-range is defined as well as the x and y The view-ranges are stored at the corresponding screen in position in relation to the top left corner of the correspond- an internal list in the order they were added. This list repre- ing screen. The remaining visible area of a view-range is sents a hierarchical, linear structure that has direct influence called ContentBox, which has the dimension of the View- on the rendering process. The position inside the list cor- Box subtracting the several box model sections (see Figure responds to a common z-index mechanism. This means an 4). All dimensions and positions used as parameters in this overlapping view-range in a further position inside the list framework are defined in pins. Therefore, the definition of overwrites an overlapped view of an earlier position. In ad- sizes and points is depending on the resolution of the used dition, the definition of a layer order, not corresponding to output device. the list order, is also possible by setting the z-index property for the view-ranges. View-ranges are always opaque. That For handling oversized content and enabling the access to means the underlying content will be erased by an overlap- content that doesn’t fit in a view-range, a panning concept ping container, even if it is empty. is realized. With the parameter OffsetPosition the con- tent can be moved under the view range frame. The offset View-ranges have a full box model corresponding to the CSS position defines the start position of the content in relation box model for web sites [14] (see Figure 3). The box model to the left corner of the ContentBox. If necessary, simple consists of padding (space between content and border), a tactile scrollbars are added and rendered to give feedback to border and a margin (free space around the border to other the user about the position inside the content. The scroll- elements). Free space is defined as lowered pins which results bars consist of a continuous line with an adjacent indicator in a recognizable gap between elements. A part of a border which is three points long and one point thick. To keep the is rendered as a continuous straight vertical or horizontal scrollbar recognizable a one pixel space in the direction of line of raised pins. Other border styles are not available the content is set up. Therefore, the scrollbars only reduce until now. A border can be used to separate elements with the available content space by three pins. a clear tactile stimuli. If a border is used, it is recommended to set a margin and a padding as well to mark the separating A view-range can get several types of content to display. As line as not related to the content. A border of one pin width shown in Figure 5 the basic data structure, to which every seems to be sufficient in most cases. This also saves rarely other content type will be transformed, is a two-dimensional available display space. All properties of the box model are Boolean matrix (bool[,]). A true-value represents a raised independently definable in all four directions. pin on the tactile display, a false-value a lowered one. Other content types are text, which is rendered in Braille or as With these tools a screen can be organized in regions with an image, as well as pictures, which are rendered as binary different content which can be separated by significant tac- images. With a free definable threshold for the lightness of tile features, such as dividing space, separating lines or mark- a pixel, darker pixels will be set to raised pins and lighter ing frames. The division of the available display space seems ones to lowered pins. not always to be reasonable, especially on small displays such as the GWP [1]. However, for displaying special, tem- In the end the framework allows for setting any other type porary or non-persistent information the method of show- of content. However, this requires the definition of a spe- ing brief overlapping view-ranges could be useful. To en- cialized renderer for the given content type, implementing sure that contents of different areas are not mixed together the IBrailleIOContentRenderer interface, converting the by recognizing them as one whole content area it seems to content to a usable Boolean matrix. Text Image Other < > < > BrailleIODevice IBrailleIOContentRenderer IBrailleIOAdapter AbstractBrailleIO BOOL BOOL + string AdapterType + int DeviceSizeX + bool Connected AdapterBase + int DeviceSizeY + BrailleIODevice Device ... + bool HasKeys 1 + int DpiX + bool Synch + bool HasTouch + int DpiY ... + String Name ... Events: Matrix bool[y,x] + int RefreshRate + void Synchronize ... Events: + errorOccured + initialized + inputChanged + keyPressed BrailleIOAdapter_ + keyStateChanged ShowOff + pinStateChanged + touchValuesChanged Figure 5. Content types for view-ranges. Figure 6. Adapter and device class structure. Nearly all renderer are hookable. This means, an exten- Some general device events are proposed as well (see Figure sion or an user of the framework can get access to the ren- 6). These events should give feedback about availability and derer functionality by registering a hook. This hook will changes in key-, touch- or pin-states as well as the occurred be called before the renderer starts his work and after the errors. In these events the sending device and the original renderer has rendered the result. A hook has the opportu- raw device data of the event are enclosed and sent to all nity to manipulate all function parameters in the beginning registered listeners. The listener can decide if he wants to of the rendering. Furthermore, the redering result can be handle the generalized data, such as the general buttons, or modified before it will be returned and sent to the output the original device data, e.g. additional keys that are not devices. This gives programmers the power to use already mapped to one of the general keys. implemented standard renderer and adapt or extent them to their needs, without implementing an own renderer. Two real hardware-specific adapter implementations were built for different types of the BrailleDis series, named the 2.3 Device Abstraction BrailleDis 9000 [13] and the BrailleDis 7200 [6] (see Fig- As mentioned before, the abstraction and modeling of hard- ure 8). In addition to the real adapter implementations, ware is an essential part of the framework, too. Therefore, a software adapter was developed. The so-called ShowOff the first idea was to identify basic properties and features. A adapter can be used for debugging if no real hardware de- construct for modeling and mapping a real hardware inter- vice is available or for monitoring a connected BrailleDis face was developed. In this, special properties are defined, device. It implements the IBrailleIOAdapter interface and for instance, a proposed image refresh rate, the number of can be used as a standalone input or output device for ap- pin-rows and columns or the availability of buttons or touch- plications based on the framework. It is inspired by the sensitivity of the hardware device. BrailleDis 7200 device and can be used as emulator. With the simulator it is also possible to enter single touch inputs A specific hardware device still needs to be mapped to the by mouse. proposed framework. For this purpose, an adapter has to be provided that implements the IBrailleIOAdapter interface and generates a unique device representation as an object of 2.4 Interaction type BrailleIODevice (see Figure 6). This has to be done A set of basic functions for interaction are implemented. As once for every new hardware device that should be used with mentioned before, a first assignment of keys with functions is the BrailleIO framework. proposed by the adapter implementation (see section 2.3), but not realized by the framework. The linking between The implemented adapter is responsible to achieve proper buttons and functions has to be done by the application access to the hardware device. This means the adapter has developer. Nevertheless, all necessary functions are available the task to create, open and hold a channel to the hard- in a basic implementation. ware device as long as needed and to bring the standard- ized output-matrix on the device. At the same time, the adapter implementation has to map the proprietary and device IBrailleIOAdapter device-specific interactions and events to the idealized data models which are expected by the framework. If a device OK + has hardware keys, they have to be modeled, too. Nine ba- sic function keys are defined as a minimal set for a sufficient ESC - interaction on a touch-sensitive tactile graphic device (see Figure 7). The key set consists of four navigation keys and GST two zoom buttons to interact with oversized, graphical or zoomable content, such as images. Furthermore, two inter- action buttons for approval and refusal are defined. Finally, a special key for touch-sensitive devices seems to be neces- sary to start and stop a gesture input to avoid midas touch effects [10]. Figure 7. General device model with nine keys. BrailleDis 7200 set up devices + register to register GST ESC - Adapter device gesture events recognizer register OK devices set the active real or virtual device AdapterManager adapt content or views on BrailleDis 9000 BrailleIOMediator user input build screens choose active add BrailleIOScreen add BrailleIOViewRange screen define size, position, style OK set content ESC + and visibility - GST Figure 8. General key mapping for BrailleDis 7200 Figure 9. Steps to set up a project with BrailleIO. and BrailleDis 9000. After setting up the device that should be used for in- and Zooming can be realized by setting the zoom property of output, it has to be registered to the AdapterManager related the corresponding view-range. The framework will handle to the BrailleIOMediator. A registration to the events the zoomed rendering if possible. Panning operations can thrown by the device has to be done. If the device is touch be connected to the OffsetPosition property of a view- sensitive, the optional basic gesture recognizer can be con- range (see section 2.2). By setting the offsets to negative nected to interpret touch inputs. An unlimited number of values the content can be moved below the visible area of devices can be registered but only one device can be marked the view-range’s ViewBox (see Figure 4). Changing the y- as active. This device is used as main output. Other de- offset realizes a vertical and changing the x-value a hori- vices can be used as output by setting the Synch property of zontal scrolling. The offset can be changed freely. Several the AbstractBrailleIOAdapterBase. This leads the medi- functions of the abstract base class implementation of the ator to mirror the tactile result to these devices. With this view-ranges (see Figure 1) offer offset manipulation in an mechanism the ShowOff adapter can be used, for example, easy manner. as debug monitor beside a real tactile matrix device. A basic gesture recognizer is included. It recognizes a num- At least one view-range has to be defined for displaying con- ber of basic gestures, such as pointing gestures (tab), swipes tent. It has to be configured with a position, size and the (line), pinch, circle (half and full) and three finger drag op- optional box model. The view-range can be added to the erations (compare [7]). All gestures are interpreted and mediator directly or it can be combined with other view- returned with further information, such as start and end ranges in a screen which has to be added to the mediator. point, direction or orientation. In addition, a timestamp An unlimited amount of screens are allowed. After setting a fingerprint allows for the inference on the duration, speed or screen as active or visible it will be displayed on the output temporary order of interactions. device. Every view-range has to get its own content which will be rendered and presented. The views or the contents Inversion and threshold adaption are also part of the frame- can be changed, for example, on user interactions reported work for image handling. For instance, these features are from the device events. useful for exploring images. Small sinks - regions of low- ered pins inside an area of raised pins - can be transformed 3. CONCLUSION AND OUTLOOK to a raised pin area for a better detection by inverting the The framework BrailleIO was presented. It enables a fast presentation. The adaptation of the threshold level for the and easy entry into building applications on two-dimensional binary image conversion allows for adjusting the presenta- tactile displays for visually impaired users. With the screen tion to the given context. Especially if very light or very and view-range constructs a proper information organiza- dark images are presented, the adjustment of the threshold tion and simultaneous reception is possible. The framework is necessary to make structures visible at all. also proposes a hardware abstraction for pin-matrix devices including general hardware keys and function binding. 2.5 Usage The framework is used, for example, as groundwork for a tac- In the following a small guidance about how to set up a tile graphic production workstation called Tangram worksta- project with the BrailleIO framework in a few steps is given tion [3]. This project enables collaborative work of a sighted (compare Figure 9). and a blind user on one graphic. Several open issues have to be solved and innumerable im- system for the blind using tactile displays and provements are conceivable. At this point, there is no way ultrasonic pens - mimizu. In K. Miesenberger, to map infinite further hardware keys to the generic key con- J. Klaus, W. Zagler, and D. Burger, editors, struct. A continuous numbering of further keys could be a Computers Helping People with Special Needs, volume solution to overcome the fall back to the proprietary nam- 3118 of Lecture Notes in Computer Science, pages ing of these buttons. In this context, the implementation of 731–738. Springer Berlin Heidelberg, 2004. more concrete hardware adapters would proof the concepts [6] D. Prescher. Redesigning input controls of a and the portability beyond the BrailleDis devices. touch-sensitive pin-matrix device. In This Proceedings, pages 19 – 24, 2014. It is also unclear how to handle output devices with a sig- [7] D. Prescher, G. Weber, and M. Spindler. A tactile nificantly higher resolution than 10 dpi, which will lead to windowing system for blind users. In Proceedings of problems on rendering Braille. The resolution of an used the 12th International ACM SIGACCESS Conference device is available in his specifications and therefore has to on Computers and Accessibility, ASSETS ’10, pages be checked and used while rendering resolution dependent 91–98, New York, NY, USA, 2010. ACM. content. [8] A. Richter and G. Paschew. Optoelectrothermic control of highly integrated polymer-based mems The next big and challenging step is the implementation applied in an artificial skin. Advanced Materials, of a powerful Braille renderer. Currently, strings are ren- 21(9):979–983, 2009. dered with an equidistant Braille font as an image and sent [9] M. Schiewe, W. Köhlmann, O. Nadig, and G. Weber. to the output devices. There is no way back from the ren- What you feel is what you get: Mapping guis on dered image to the original given text. This is necessary, for planar tactile displays. In C. Stephanidis, editor, example, for a controlled audio output of touched Braille Universal Access in Human-Computer Interaction. elements. For the near future, it is planned to build a ren- Intelligent and Ubiquitous Interaction Environments, derer based on the free transcoding project liblouis [4]. This volume 5615 of Lecture Notes in Computer Science, should allow to take HTML strings as input which can be pages 564–573. Springer Berlin Heidelberg, 2009. adapted with cascading style sheets. [10] M. Spindler, M. Kraus, and G. Weber. A graphical tactile screen-explorer. In K. Miesenberger, J. Klaus, BrailleIO is realized as open source project. One big ad- W. Zagler, and A. Karshmer, editors, Computers vantage is, if some functions or concepts are missing or are Helping People with Special Needs, volume 6180 of not optimal, the open source approach enables anybody to Lecture Notes in Computer Science, pages 474–481. take part and improve, change or complete the project with Springer Berlin Heidelberg, 2010. further and better ideas. The framework is freely available [11] R. Velazquez, E. Pissaloux, M. Hafez, and through https://github.com/TUD-INF-IAI-MCI/BrailleIO. J. Szewczyk. A low-cost highly-portable tactile display based on shape memory alloy micro-actuators. In 4. ACKNOWLEDGMENTS Virtual Environments, Human-Computer Interfaces I thank Heinrich Kuhlmann for his work on BrailleIO, Dr. and Measurement Systems, 2005. VECIMS 2005. rer. nat. Michael Schmidt for his implementation of a Proceedings of the 2005 IEEE International basic gesture recognizer and Denise Prescher for her work Conference on, pages 6 pp.–, July 2005. on the interaction design. The Tangram project, in which [12] F. Vidal-Verdu and M. Hafez. Graphical tactile BrailleIO is developed, is sponsored by the Federal Ministry displays for visually-impaired people. Neural Systems of Labour and Social Affairs (BMAS) under the grant num- and Rehabilitation Engineering, IEEE Transactions ber R/FO125423. on, 15(1):119–130, March 2007. [13] T. Völkel, G. Weber, and U. Baumann. Tactile 5. REFERENCES graphics revised: The novel brailledis 9000 pin-matrix [1] P. Albert. Math class: An application for dynamic device with multitouch input. In K. Miesenberger, tactile graphics. In K. Miesenberger, J. Klaus, J. Klaus, W. Zagler, and A. Karshmer, editors, W. Zagler, and A. Karshmer, editors, Computers Computers Helping People with Special Needs, volume Helping People with Special Needs, volume 4061 of 5105 of Lecture Notes in Computer Science, pages Lecture Notes in Computer Science, pages 1118–1121. 835–842. Springer Berlin Heidelberg, 2008. Springer Berlin Heidelberg, 2006. [14] W3C. World wide web consortium - cascading style [2] R. Babu, R. Singh, and J. Ganesh. Understanding sheets level 2 revision 1 (css 2.1) specification - 8 box blind users’ web accessibility and usability problems. model. http://www.w3.org/TR/CSS2/box.html, 2011. AIS Transactions on Human-Computer Interaction, [15] E. Wilhelm, T. Schwarz, G. Jaworek, A. Voigt, and 2(3):73–94, 2010. B. Rapp. Towards displaying graphics on a cheap, [3] J. Bornschein and D. Prescher. Collaborative tactile large-scale braille display. In K. Miesenberger, D. Fels, graphic workstation for touch-sensitive pin-matrix D. Archambault, P. Penaz, and W. Zagler, editors, devices. In This Proceedings, pages 42 – 47, 2014. Computers Helping People with Special Needs, volume [4] C. Egli. Liblouis-a universal solution for Braille 8547 of Lecture Notes in Computer Science, pages transcription services. Deutsche Zentralbibliothek für 662–669. Springer International Publishing, 2014. Blinde Leipzig (DZB), 2011. [5] M. Kobayashi and T. Watanabe. Communication