Mobile apps are now created by everybody: shops, banks, small businesses and large corporations. There are so many different applications available that users tend to uninstall many of the applications they previously installed. One thing is for sure, customers will opt out of the application if it does not work on their device. In order to assure the application runs fine on the device, we should perform some tests. But how to test applications for mobile devices, since there are so many of them?

Platforms

In this post I’d like to focus on smartphones, since they are the most popular from mobile devices. Smartphones can operate on different systems (OS – operating system), but the leading two are Android and iOS.

Android vs iOS – statistics

First thing to take under consideration is whether you want to develop your app on Android or iOS, or both. Android seems to be more popular in most countries in Europe (besides countries such as Switzerland, Denmark or Sweden), Asia, Africa and South America. Countries where majority is iOS are for example: USA, Canada, Australia. You can find the stats here. If you want to download an application on your Android device, you use Google Play Store while downloading an application on iOS is possible by Apple App Store. One can find the statistics about the stores on 42matters.com. For instance, for today, there are 2,962,686 apps on Google Play and 1,812,847 apps on App Store. In addition to that, you can find out that for Google Play average number of new apps per day is 4,121 and for App Store the number is 1,193.

Android

Most prominent feature of Android is that it’s available for everyone, people from all over the world can use it. Smartphones are cheaper than these running on iOS, so they are more affordable, accessible. Not only Android has more applications, it also has enormous number of devices. On Android.com is written that ‘Over 24,000 different devices run on Android’. Android is available on many phones, major producers are Samsung, Sony, Motorola, Huawei, HTC and LG. As a result, there are also many versions. This is how Android breaks down according to the official Android Developer website (as for state on May 2019):

• Android 9 Pie: 10.4 percent
• Android 8.1 Oreo: 15.4 percent
• Android 8.0 Oreo: 12.9 percent
• Android 7.1 Nougat: 7.8 percent
• Android 7.0 Nougat: 11.4 percent
• Android 6.0 Marshmallow: 16.9 percent
• Android 5.1 Lollipop: 11.5 percent
• Android 5.0 Lollipop: 3.0 percent
• Android 4.4 KitKat: 6.9 percent
• Older: 3.8 percent

iOS

Firstly, the Apple App Store offers fewer apps than Google Play. The reason for that could be that Apple is strict about what apps it allows, while Google’s standards for Android are loose. Second thing about iOS is that Apple devices has always had high prices. Furthermore, iOS runs on much fewer devices, enough to say that for smartphones it’s only iPhone. For this reason, there are just a couple versions that Apple iPhone is running. This is how iOS version shares break down according to the official Apple Developer website (as for state on February 2019):

• iOS 12: 80 percent
• iOS 11: 12 percent
• Earlier: 8 percent

80 percent (or even more now) of all iOS devices are running the latest version, so you can test one OS version and you cover the majority of cases.

Mobile app testing

Testing application on mobile devices is more challenging than testing web app on the desktop due to:

• different range of mobile devices (screens variety: extensions and colors)
• wide varieties of mobile devices (HTC, Samsung, Apple, etc.)
• different mobile operating systems (Android and iOS)
• making and receiving calls is the main task of the phone, that is why the application should not interfere with this major function
• frequent updates – with each update a new testing cycle is recommended to make sure no application functionality is impacted
• tools, which are good for the desktop apps testing, are not fully suitable for the mobile application testing

How to deal with problem of many devices

If you consider yourself as good mobile application provider, you should have a large number of devices intended for testing. A set of the most popular devices is the basis during the process of creating applications, enabling finding errors on different devices. How to choose device to ensure that tested application will run on thousands of devices all over the world? It surely is not an easy task. You can find some advices while selecting device for the mobile app testing:

• try to define the most popular and used devices in the market
• choose devices with different OS
• choose devices with different screen resolutions

Device farm

If you don’t feel like you need or afford so many devices, you can try so-called “device farms”. These are remote services that enable companies to run tests on a large number of devices in a short time, remotely through the Internet. Device farms can be relatively cheap, because you can only pay for the time spent on a specific device. Below you can find some examples of device farms:

• AWS Device Farm
• Xamarin Test Cloud
• Sauce Labs
• Firebase Test Lab for Android
• BrowserStack

Emulators and simulators

While real device mobile testing requires the use of the actual smartphone, emulators and simulators are based on the concept of virtual testing. An emulator is an application that mimics the hardware and OS. On the other hand, a simulator doesn’t replicate device’s hardware, but you have an ability to set up the similar environment as the original device’s OS. Emulators are considered as suitable for some stages of application development. For example, they are good for debugging. You can use Google’s Android SDK or Apple’s iOS Simulator to easily find the reason of the bug. However, the major part of mobile testing should be done on real devices, due to most important disadvantage of simulators/emulators: it can’t simulate all types of user interactions; hence it may lead to false results sometimes. That is why, when it comes to reliability, testing on a real device always gives you the highest accuracy of results.

Summing up

Emulators and simulators are covering a wide range of devices so it looks like they are a great alternative for testing mobile apps. Sure, we can use them while testing, but before finalizing the product, we have to run tests on real devices. Since there are so many devices available, testing mobile applications using device farm seems to be the optimal choice. It seems that it could help you to overcome the problem with high prices of real devices.

‍