Selecting target devices - learnings from Kin

Arriving at a decision on the target devices and OS to be included in testing is a difficult task. Many times people resort to industry data on phone sales as a reference for selecting the devices that they need to test. While it is a good source of information, one cannot rely entirely on such information nor can they select the latest devices or devices from the largest players alone.

I do not know how many of us know what a Kin  is – it ‘was’ a smart phone that Microsoft invented. Reportedly, it was in the market for a little over a month only. Like one reporter observed, many fruit juices have more shelf life than that. When Microsoft builds a mobile, will any one ignore it? Surely will not. If it lived a little longer, it would have inflated the testing budget by adding one more device and resources to test it, before it dies its natural death.
Deciding on the device combination is one of the key success factors for a good testing strategy for mobile devices. With 75% of the market share, Android has a place in all test combinations, followed by iOS with 15% share. However, a deep dive into the target market and its behaviour is required before the selections can be made.
The best way to short list the devices is to segment the target audience, and then depending on the risk appetite, eliminate the devices from the segments to arrive at a combination of Must, Good and Nice to have. This combination will include devices operating systems and their combination (there can be devices that need to be tested on more than one operating system).
Another challenge with the test combination is that it is dynamic. Every release, the test team will have to look at the combinations and revise the matrix to reflect the latest changes to the combination.

I test. Therefore I am.

Testing and Mobility

Last year, about the same time, I wrote about the probable impacts of tablet devices on testing. When I look back, I see the predictions were true, but reality has far exceeded my predictions. The tablets have become an integral part of the application space. It is reported that the sale of tablets have exceeded the sale of PCs and by 2015, tablets will exceed the PC population.

It is not very unusual for my blog to hibernate for long periods. But after reading several reports and stats on the growth of mobile devices and the predictions for the next few years, I thought it is time to update the old posts on testing tablet devices.

The northbound trend in the population of mobile devices including smartphones and tablets is is a very significant development that will have far reaching impact on application development and testing. Not only will more and more applications become mobile enabled, they will become an integral component in completing transactions. This will mean much more tan testing how the web pages will behave in different mobile browsers, but will mean testing complex hybrid applications.

Testing native or hybrid application needs a different strategy and approach than testing browser based applications on mobile devices. Right from the code base to the capabilities of the native hardware will have an influence on the testing outcome. Testers need to understand and develop suitable strategies to focus on functionalities as well as device and OS dependant features that need to be tested on each target device.

While the number of devices is showing a uniform growth trend, it is also increasing the diversity of the target devices that need to be included in testing. With several device manufacturers designing devices with all possible variations in form factor, processing power, interface standards and network capabilities, the hardware choices are numerous. So are the Operating System choices. With many different Android versions, and device manufacturers and telecom carriers modifying or adapting the operating systems to optimise in their networks increases the test combination many fold. Android currently has around 75% of the market followed by Apple with a share of 15% covering 90% of the market. However, the market is very dynamic and user base is not uniform across geographies.

In theory we have three broad categories of software for mobile; the native apps, mobile web and hybrid. The native apps reside in a hand held device whereas mobile web is accessing web content using a mobile browser. Hybrid applications on the other hand utilises a combination of both hybrid and web features. While there are several things that are common in them, there are many unique factors also.

When testing mobile web, the focus is primarily on the mobile browsers capability to interpret the code and deliver the content. Different browsers use different engines internally (Eg. Chrome uses WebKit while Microsoft uses Trident), and so, the way they render content is not universal, especially while handling evolving standards like HTML5. Native apps on the other hand, reside in the device and interact more closely with the host operating system and hardware and software resources of the device. This also means that devices running different operating systems will use different code base, the testing becomes extremely critical on different operating systems. Hybrid apps extends the challenges by combining both native and web characteristics into one.

These are in addition to the capabilities of the devices, capabilities dependent on carriers, usability, performance, security and interoperability that are other critical characteristics. In the next few posts we will discuss a few of them in details and also discuss about tools that help in accelerating/automating the tests on mobile.

I test. Therefore I am.

New era software tool licencing

There not a single day that we do not hear something new about the 'cloud'. HP has now announced that its testing software, especially the load and performance testing software will be available on the cloud.

Traditionally, the cost of licences and the ability of the projects to invest in licenses were the two biggest deterrents when it comes to introduction of new testing tools in an organisation. The new model will drastically change the game and will push the service providers and IT departments to work out better models of using tools and delivering services based on tools. I hope to see some significant changes this new year.

I test. Therefore I am.