Software quality of apps often is low, which at least partly results from problems with testing them. A main problem are frequent context changes that have to be dealt with. Network parameters such as latency and usable bandwidth change while moving; usage patterns vary. To address context changes in testing, we propose a novel concept. It is based on identifying blocks of code between which context changes are possible. It helps to greatly reduce complexity.
Mobile applications { apps { draw greatly from the possibilities o ered by devices
such as smartphones and tablets, e.g. by making use of localization via GPS.
Software testing is a main challenge in app development. Testing software is
a cumbersome task [
Testing is greatly in uenced by context. Mobile devices are subject to many di erent contexts; the simplest one is location, since mobility typically means frequent changes of position. There are many further context changes such as network condition, availability of data from sensors, and even social issues such as sharing devices. Thus, devices need to be tested taking context into considerations.
We propose a novel approach for software testing of apps that takes into account context changes. Our contributions are a sketch of context as an in uencing factor of apps, the introduction of our unique approach for handling context in testing, and the demonstration of a real-life scenario to underline feasibility.
This paper is structured as follows. Section 2 highlights the relevance of context changes. Section 3 explains our approach. Application, limitations and challenges are discussed in Section 4.
The usage paradigm of apps contrasts that of applications on PCs and that of Web sites (Webapps). The main di erence is mobility. Even if an app not explicitly battery status dynamic hardware context currently available memory variation of available space memory
disk space static screen size and resolution component configuration processing power availability
accuracy geolocation technologies physical context sensor data devices / objects
around partly measured
by motion position environment business private in meeting etc.
Bluetooth
USB
NFC etc.
has effects on level of attention situation
Influences … of week connectivity direct communication social context time
data cellular bandwidth network com. context network
status status … of day latency available roaming unavailable application lifecycle application system 3rd party system apps software context operating system
Android version vendor specific adjustments takes notice of mobility it is in uenced by it due to changing conditions e.g. regarding connectivity. Apps might be halted on external events such as incoming phone calls. Moreover, they are built to be used in a changing environment.
Consequently, apps are heavily in uenced by context but are also capable of making use of it. To give an example: when you drive out of town your smartphone might need to switch to a cellular system providing less bandwidth (adaption to context) but help you nd a nearby swimming lake by matching geolocation information with map data (utilization of context). 2.1
In the following, we propose a categorization scheme for app context that distinguishes ve core contexts. It is sketched in Figure 1.
Firstly, there is the hardware context. Apps run on a multitude of devices, ranging from smartwatches to TVs. Components of devices greatly di er, leading to various screen sizes, resolutions, sensors, and input means besides touch to name just a few. Apps might not nd the kind of hardware they require for proper operation and might need to adapt to hardware of di erent quality. Additionally, available memory and battery capacity have to be taken into consideration. The situation is worsened by the rapid progress in hardware development, making requirements very hard to forecast. The problem can be handled by seeking for a kind of \greatest common divisor" among devices. Context testing should include various possibilities of encountered hardware, e.g. with a sensor being present or not, and considering di erent levels of quality (such as precision). A testing strategy similar to equivalence partitioning [13, p. 28] might be chosen.
Fragmentation in terms of hardware goes along with software fragmentation,
forming the software context. With at least four popular platforms [
Mobility means that the physical context is continuously changing [
Depending on the location, connection parameters such as availability, bandwidth and latency vary. This forms the communication context. it is in uenced by static (e.g. the carrier of a user's choice) and dynamic factors. Location typically determines which mobile services are available and which bandwidth can be used. Apps ought to be robust and maintain functionality in o ine scenarios. Testing includes simulations of small bandwidths, high latencies, changes in connection quality, and abrupt unavailability of service as well as resuming service.
The social context is harder to grasp than the other context categories. It
comprises of user-speci c ways of using an app. Firstly, more and more mobile
devices are used both for work and for private purposes as part of Bring your
own device (BYOD) [
In consequence, app testing has to be adjusted. Frequent changes of context have to be expected and patterns of change not necessarily are predictable. The multiplicativity of contexts makes testing more complex and time-consuming. 2.2
The relevance of context in mobile computing has been discussed as early as in
2001 [
Standard testing literature is valuable since Web-based applications are
typically covered (e.g. [17, Chap. 22]). Techniques for testing of graphical user
interfaces (GUI) can be applied to apps. Mobility is sometimes covered. Factors
such as di erent connection speeds of mobile services [15, pp. 166] might be
addressed despite not explicitly discussing context. However, textbooks on app
development often do not address testing but for some exceptions such as [
There is a variety of { typically research-in-progress { papers on app testing.
Directions of research are automation [
To combine context changes with app testing, it has to be possible to automatically change context parameters whilst test cases are executed. The naive approach would be to specify the context for each test case a priori. Asserting that a single code unit produces an expected output in a certain context would not be helpful for testing business processes, though. As many in uencing contextual factors are not static but change constantly during usage, a dynamic approach is needed.
Being able to specify context changes still is static if they have to be stated within test cases. For scenarios including di erent variations of context many test cases are required, each containing the same test code. This multiplication is not desirable. Test cases would be costly to develop and hard to maintain. 3.1
To provide a solution that fosters dynamic changes of context, we use modularization. Test cases are split into blocks; between each block a change of context is possible. Blocks are reused and combined with context changes. Therefore, testing di erent scenarios is possible without duplication of test code. As blocks are the foundation of our concept, we call it block-based context-sensitive testing.
A given test case may result in a number of blocks, each containing operations and assertions. Similar to unit testing frameworks, operations are needed to simulate user interaction; assertions are used to verify expected behavior. Our idea is to derive blocks from existing test cases. A single case may be transformed into a structure of blocks that can be used to generate context-dependent ones. To preserve the test case's intention, blocks are ordered and executed consecutively.
Listing 1.1 illustrates in a schematic way how a test case looks like. If the test code itself would be divided into blocks, this schematic test case contains two of them: one from lines 3 to 7 (block A) and one from lines 10 to 12 (block B ).
The scope of each block has to be atomic w.r.t. changing contexts. In other words, during execution of test operations belonging to one single block, the ∅
Block 1 seg n a tC Block 2 h x e t n o C
Block 3 Block 4
A d.1 A d.2 A d.3 A d.4 Context Change A 1.1 A 1.2
A 2.2
A 2.4 Context 1
Context 2 context remains stable. Only between blocks changes of context are possible (lines 2 and 9). To realize di erent scenarios, only the context changes in between have to be altered. The solution is expected to be most bene cial utilizing a generator that automatically creates test cases from blocks. Manual e ort for writing context-sensitive test cases is reduced and redundant test code minimized. 1 public void 2 contextCha ntegset(EcxoanmtepxlteA( )) f; 3 clickSomewhere ( ) ; 4 enterText ( ) ; 5 c l i c k B u t t o n ( ) ; 6 . . . 7 asse rtThat ( ) ; 8 9 10 11 12 contextChange ( contextB ) ; doSomeOtherThings ( ) ; . . . asse rtThat ( ) ; g
Listing 1.1. Schematic Test Case with Context Changes
Assertions have to be extracted from blocks due to their potential dependency on a context. For each block at least one default assertion has to be assigned, which veri es the app's standard behavior. For each known context, a speci c assertion may be de ned to assess context-dependent behavior. The blocks shown in the schematic test case in Listing 1.1 therefore have to be tailored smaller. Strictly spoken, assertions in lines 7 (Block A) and 12 (Block B) are not part of the blocks but have to be treated as another building block of our concept. Depending on the app's expected behavior, assertions may be default or context-sensitive.
The building blocks of block-based context-sensitive testing are shown in Figure 2. The structure of a test case is depicted by an ordered list of blocks. 3.2
A possible test execution is illustrated in Figure 3. The beginning of the test case is denoted as the empty set. Before the rst block is executed, Context 1 is established. Next, an assertion tting to the current context is searched: there is one for Context 1. As the assertion holds, execution continues. Between Block 1 and Block 2 the context is changed again. This time, Context 2 is established and kept for the remaining execution. The second block is executed similarly to the rst one. This changes when reaching Block 3, which does not have any speci c assertions. Consequently, the expected behavior is invariant between various contexts and the default assertion is evaluated. Finally, Block 4 is executed together with assertion A 2.4, which is the assigned assertion for Context 2.
As shown in the matrix in Figure 2, there are numerous execution paths as prior to each block the context is changeable and alternative assertions can be de ned. The matrix grows with the number of contexts but typically is sparse.
To illustrate practical bene ts, we implemented a proof-of-concept tool and evaluated the approach in cooperation with an industry partner. We used a simple app to explain how it works: a client for the micro blogging service Twitter, which allows logging in to the service and posting messages. To prepare the setting, a jar archive containing our proof-of-concept has to be added to the classpath of the app's Android testing project. Moreover, as the Internet connectivity has to be changed to test the app in various contexts, ACCESS NETWORK STATE and CHANGE NETWORK STATE permissions have to be added to the AndroidManifest.xml of the app if not already contained.
The process steps for testing are logging in with invalid credentials, logging in with correct credentials, and posting a message. Each step can be conducted in a di erent context. Listing 1.2 shows the rst block implemented with our solution. The test operations are implemented in the operation part of the block (lines 4 to 7). The credentials are invalid: the app is expected to show a corresponding message. This is checked by the default assertion (lines 8 to 10). However, if the app is in disconnected context, it is expected to show an error message. This context-dependent behavior is veri ed by an alternative assertion (lines 14 to 16). l o g i n I n c o r r e c t B l o c k . a d d A l t e r n a t i v e A s s e r t i o n ( discContext , new
A s s e r t i o n ( ) f public void a s s e r t T r u e (a wssaeitr FtioornT(e)xt f("Could not l o g i n : No connect ion . " ) ) ;
Listing 1.2. Sample Block implementation
For each of the above stated process steps a block with context-speci c assertions is implemented and added to a list of blocks. A generator creates test cases as di erent mutations in terms of context changes from that list. In one possible test case, the rst two blocks are executed in connected context and their default assertions are used for verifying. Before executing the third block, the context is changed. The context dependent assertion for that block checks whether in the disconnected context the message \No connection" is shown. Even in a small scenario as shown here a lot of di erent execution paths are possible: testing without our approach would be burdensome.
We used two sample generators to create test cases from the blocks explained above. In each test case the generator changes the context at di erent steps in the process. Finally, test cases are executed by means of JUnit. Like any other test for the Android platform, they are running on the device or emulator itself. 4
In this paper we presented block-based context-sensitive testing. Our concept extends the literature of context-sensitivity in mobile computing. We have shown the viability of our approach in a case study and by presenting a prototype.
Out proof-of-concept Android tool is written in Java and can be checked
out from GitHub [
We found that by using our proof-of-concept it is possible to test apps in various contexts e ectively. Blocks of test code are reusable and thus the e ort in writing tests is reduced. Using generators the concept even gets more e ective. Testers only have to implement blocks and de ne context-speci c assertions once. Our approach also helps to nd errors in code units where they are not expected.
However, our concept cannot (yet) be a panacea. We deem it a supportive mean for testing parts of apps which are heavily in uenced by context. Due to the novelty of our approach, several limitations exist. Firstly, having a ordered linear list of blocks allows for only one way of execution, but the order of blocks and the decision if a block is executed may depend on the context used. We introduced a method for aborting test case execution to cope with this. Anyhow, our concept is not able to deal with process steps that only have to be executed in certain contexts. Secondly, the case study is an example of feasibility, yet not exhausting. E ectiveness has to be proven both qualitatively and quantitatively. Thirdly, our prototype is scarcely commented and not yet very user friendly.
These limitations have to be kept in mind yet do not question the general feasibility of our approach. In fact, they lead to future work. The most important task is an extension of our tool. Better support for test case generation w.r.t. context selection is desirable. Compiling best practices for context-sensitive testing would complement our work. A general challenge is to become able to cope with ample contexts. While our approach theoretically is capable of dealing with arbitrary context changes, actually simulating them for testing is very hard.
Future work needs to re ne our concept and investigate into the consequences
of context changes. Moreover, we will scrutinize the relationship of our work to
other methods, e.g. data-driven testing [
We would like to thank viadee Unternehmensberatung GmbH for their support.