The importance of software testing has been discussed and proven in many articles and in existing research. Software testing plays a key role in providing quality software solutions and services. Standard testing approaches and methodologies were adequate until the arrival of mobile technologies. With mobile technologies, the testing process was forced to change in the face of significant challenges, the most important one being mobility. Mobility provides a pallet of challenges that are unique and demand new testing approaches and methodologies in software testing. The identification of challenges and issues has helped the development of the mobile software testing process and tools. With a wide range of new testing tools, testers face a new challenge in selecting the right tool and methodology for testing mobile applications. In this paper, we will present criteria for selecting mobile application testing tools based on identified challenges and issues, testing approaches and strategies. We will provide a proposal for a simpler and quicker way of selecting the appropriate tool for testing mobile applications. Additional Key Words and Phrases: testing, mobile applications, mobile technologies The increased use and the rapid development of mobile devices and technology is a clear sign of future trends in ICT. Not only do the number of mobile users increase daily, the same thing is occurring in the market for tablet PCs. According to Gartner, the number of purchased tablets is going to surpass the number of desktop-based computers in the beginning of the year 2015 [Rivera and Van der Meulen, 2013]. More and more people are using mobile technologies in their everyday lives for interaction, entertainment, business and more. With mobile applications we can extend the usability of mobile devices even further. A mobile application is an application that runs on a mobile device and is context aware. This means that the application is aware of the computing environment in which it runs and can adapt/react according to its current context [Muccini, 2012]. Context-awareness is just one of the many challenges in mobile application testing, which demands new methods and testing approaches. There are many challenges in mobile software testing, which by definition [Gao et al. 2013] means: “testing activities for mobile-based applications and mobile web applications on mobile devices using well-defined software test methods and tools to ensure the quality in mobile service functions, behaviors, performance and QoS, as well as mobile features, such as mobility, usability, inter-operability, mobile connectivity, security and privacy.” Mobile applications that are free of faults and errors provide a better user experience, which has a direct impact on the business success of the application. Users grade the quality of the mobile application based on their user experience. Unfortunately, many new users choose applications based on previous reviews and grades. Therefore, old errors and faults, or a poor user experience in an otherwise working application can lead to the business failure of the application. In this paper we will present the criteria for selecting mobile application tools based on the identified challenges in mobile testing, testing approaches and strategies. First, in Section 2, we will present the main challenges and issues in mobile software testing. In Section 3, we will present the four testing approaches that have been identified by [Gao et al. 2013]. In Section 4, we will present and analyze related work on mobile testing tools, which provides the basis for our criteria definition. Our main focus is to present the criteria definition process and to extend the criteria for selecting mobile testing tools that have been identified by [Gao et al. 2013]. The idea is to have a selection of criteria that can be used for the characterization of a variety of mobile testing tools. Once we have the mobile testing tools characterized
properly we can make a quicker and more effective selection. In the Discussion, we will comment on our findings.
There are many challenges in mobile software testing due to the nature of the environment in which the
mobile applications are running. Based on the definition by [
One of the more important challenges in mobile software testing is the connectivity of mobile devices with
various mobile networks and devices. Unlike desktop applications, which use fixed network connections,
mobile applications connect to mobile networks, which can vary in speed, security and reliability
[
Based on the difficulties and requirements of the testing procedures, different testing approaches are
recommended [
Mobile applications use the resources of mobile devices, which are very limited. Despite the rapid
development of mobile devices, it is important that the consumption of resources is monitored and
controlled at all times [
The excessive use of resources can reduce the performance of mobile devices and can cause malfunctions in the mobile application. During the testing process the consumption of resources must be constantly monitored. 2.3
Mobile devices need energy to run. The use of mobile devices depends on battery capacity and the way the
device is used. All the device’s resources and activities use energy but not equally. GPS sensors, data
transfer and video editing are activities that use more energy than others [
Mobile operating systems have different user interfaces, which are defined by rules and guidelines. The
use and layout of elements is checked in the verification process when publishing the mobile applications
on the markets. Non-compliance with rules and guidelines can delay the publishing process, increase the
cost of development and testing. Different screen sizes can also have an impact on the look and usability
of the mobile application. Different mobile devices can react differently to the same application code,
which must be tested with GUI testing [
Context is by definition [
Context can limit or extend the operation of mobile applications or its functionalities with data from
the environment in which it is in. Mobile applications can be in different contexts with different data. This
creates a unique challenges in the testing process [
Context aware mobile applications adapt and evolve based on the data obtained from the environment.
This evolution can happen in real time without interrupting or stopping the operation of mobile
applications. Unfortunately, this can lead to unexpected and unplanned changes in the mobile
application’s operations. The reliability of a mobile application depends on the management of context
adaption. To insure the correctness of applications operation, context-specific test selection techniques
and coverage criteria have to be produced [
There are many different mobile devices, made by different vendors, which have different hardware and
software settings. The number of variations is even larger if we add all the devices that have a modified
mobile operating system. The vendors modify the operating system to create a better user experience for
the user, or increase the functionalities of a device. Due to these variations, mobile applications can run
and behave differently [
The user experience includes the user's perceptions and feelings before, during and after the interaction
with the mobile application. Often, users assess the application based on their user experience, therefore
the appropriate user experience is critical for the success of the application. The adequacy of the user
experience cannot be directly tested because of the subjective nature of the entire process. But we could
check the correctness of individual segments and determine compliance with good practices [
The adequacy of the user experience includes verification of elements and activities in the areas of design graphical user interfaces, interaction and usability of the application itself.
The design of the graphical interface is evaluated based on the proper and logical use of layouts,
navigation between screens, layout of graphical elements, fonts and text [
Mobile devices use touch screens as the primary means of interacting with the user. Touch screens enable
the display and input of data as individual values or as a group of data. The user activates the interaction
with a touch of the screen, which can be a single touch or a multi touch interaction. There are also
gestures that include different sequences and combinations of touches. Gestures allow additional
functionalities in the mobile application’s operation, which creates new challenges in the testing process.
Touch screens are tested based on the correctness of the displayed data and the responsiveness
[
Programming languages for mobile applications have been designed to support mobility, resource
management and new graphical user interfaces. Traditional testing techniques do not take into account
the operation of programming languages in mobile operating systems, so they need to be adjusted
accordingly. To analyze the code it is necessary to be aware of the specifics of the programming languages
and how they operate [
Mobile operating systems are new and still only partially reliable. Various reliability issues and
defaults are corrected in new versions of the operating system which are frequent but not always
backward compatible [
Based on [
Each of them is designed to handle challenges in mobile testing, identified in the previous section, but none of them can handle them all. It is important to select the correct approach based on the functionalities of the mobile application and the challenges that they provide. Each approach has its features and limitations, which have to be identified before the selection is made.
The emulation-based testing approach involves using a mobile device emulator, which creates a virtual
machine version of a mobile device for inspection on a personal computer. It is often included with a
mobile platform’s software development kit. It is relatively inexpensive because no testing laboratory is
needed and no physical devices have to be purchased or rented, but it can only be used to assess system
functionality within a very limited context. Although this approach is low-cost, it has several limitations –
for example, it has difficulty validating a full set of gestures because most emulators support very limited
gestures and device-specific functions. Another challenge is its limited scale for testing QoS. To overcome
these problems, a simulation-based approach can create a mobile test simulator to mimic various mobile
client operations and support more than one mobile client. However, even this workaround has its
limitations in validating device-specific mobile service functions. In addition, it is impossible to deal with
diverse devices and mobile platforms because emulators are usually based on a specific device platform
[
The device-based testing approach requires setting up a testing laboratory and purchasing real mobile
devices, which is more costly than emulation-based approaches but can verify device based functions,
behaviors, and QoS parameters that other approaches cannot. In addition, it also has the advantage of
being able to validate its underlying mobile networks via reconfiguration and selections in a testing
environment. One of the major challenges with this approach is the problem it has in coping with rapid
changes in mobile devices and platforms. Another challenge is its limitations related to system QoS
because large-scale testing require many mobile devices, which is usually impossible for enterprises [
This approach, based on testing through the cloud, is typically supported by testing vendors. The basic idea is to build a mobile device cloud that can support testing services on a large scale. This approach addresses the significant increase in demand for mobile testing services by using the pay-as-you-go business model. It also allows different mobile users to provision their required testing environment via a rental service model. Compared with other approaches, this can be more cost-effective than device-based testing for large-scale applications, and is much more effective for supporting diverse testing activities on mobile devices.
The crowd-based testing approach involves using freelance or contracted testing engineers or a community of end users such as uTest (www.utest.com), along with a crowd-based testing infrastructure and service management server to support diverse users. Currently, a service vendor supports primitive test management, a testing service, and bug reporting. Most mobile test operations are managed in an ad hoc way with very limited mobile test automation tools. This approach offers the benefit of in-the-wild testing without the need to invest in a laboratory or purchase or rent devices, but at the risk of low testing quality and an uncertain validation schedule.
Recent studies on mobile testing tools primarily focus on GUI-based testing, test automation and
whitebox testing. There are different studies that present solutions for different testing approaches and
strategies, which has created numerous tools for mobile testing [
Although the provided criteria allow a basic characterization of the testing tools, we believe that the criteria should be extended and created based on challenges in mobile testing. With a more extended definition of criteria, the selection process could be faster and more efficient.
Based on related work described in the previous section and the identified challenges, we propose a definition of criteria that is based on these challenges and the testing strategies. Our goal is to create more detailed criteria for selecting the mobile testing tool.
The definition process consists of three phases and is illustrated in Figure 1.
challenge
The process begins with the selection of the mobile testing challenge, which we have identified in
Section 2. The relevant challenge is analyzed from the perspective of the mobile application and based on
this result, the values or the range are set. Values and range can differ depending on the type of the
challenge. For example, the values for screen dimensions are small, normal, large and extra large, where
all the sizes are based on [
Based on the process of creating criteria we propose a list of criteria that are defined based on the challenges described in section 2. The proposed criteria are shown in Table II.
Functional testing Functional testing Resource constraints Autonomy 1:7 Wi-Fi Direct CPU CPU Speed RAM Memory Consumption Duration Guideline checker GPS Neighbor devices Altitude Brightness Temperature Context awareness Operating system Operating system versions Screen dimensions Layout checker Text visibility Text grammar Notifications Interruptions Context adaption
Enabled, not enabled Vendor and model
Enabled, not enabled
Supports Wi-Fi Direct connectivity Supports changing or limiting the operation of CPU cores Supports changing or limiting the operation of CPU speed Supports changing or limiting the amount of RAM Supports changing or limiting the amount of memory Supports monitoring the battery consumption Supports monitoring the battery duration GUI guideline checker Simulate data from the GPS Simulate data from the neighbor device Simulate data from the barometer Simulate data from the light sensor Simulate data from the temperature sensor Simulate data from the environment and the user Enables data input from the context in real time Simulation of a specific mobile device Supports changing mobile device platform Supports changing mobile device platform to different versions Supports changing screen size Checks the use of layouts for specific mobile operating systems Checks text visibility Touch screens
Gestured
The proposed criteria and its values were set based on the defined challenges in mobile testing. The challenges in mobile software testing vary from traditional ones because of mobility. Mobility has changed the operation of applications, devices and our interaction with these devices. Consequently, mobility has changed the software testing process, which is nevertheless still evolving. Because of this, there is a possibility that the proposed criteria can change, adapt or possibly be removed from the list. Future research will confirm or refute this claim. Also, in the future we expect the development of new tools and techniques that will enable more effective mobile testing. New tools will enable more detail testing, simulation and better approaches for testers. This also presents a potential business opportunity in the field of mobile testing tools, test automation and in the increasingly popular cloud-based testing environment.
As previously mentioned, the future trend in ICT is mobility, which also applies for mobile testing. The world of mobile devices is rapidly developing, which requires rapid development in mobile testing.