Automating Quality Assurance

 

In a perfect world an application would be used on one device, on one browser, on the same operating system. Unfortunately the days of monolithic systems where an application need only be tested on the one device that it was intended for are gone. Today, hundreds of new and old mobile devices are in circulation running different  versions of software using different browsers rendering application in a myriad of different ways. This has severely complicated the testing process and forced testing professionals to radically change their approach. 

Emulators, simulators and real devices help combat the proliferation of devices and browsers by using software that allows for the test on several versions of Windows, Mac OS X, Android, iOS and Windows Phones.  This allows for a focus on quality no matter the platform or device and ensures  the software works properly in any scenario.

Emulators vs Simulators

Emulators are hosted through browsers or in a developer environment through the desktop. They run copies of the operating system and browser, just in a virtual environment. Their main benefit is scalability and reliability. Besides having the real device this is the closest thing you will find.    Access to emulators is purchased through  service providers. This allows near infinite scalability at a cost that is fractions of purchasing each new device and allows access to multiples devices to test the newest versions of software.

Simulators are hardware built with a similar set of controls to simulate realistic operations. They are useful in allowing code to be very quickly tested.  The trade-off is they are unable to show real world scenarios, such as a slow 3G network or background software causing issues that could break the applications. Simulators are great for quick development debugging and remediation but do not offer the rigorous testing required for edge cases and environmental issues that an emulator does.

Testing Triangle

 Device Testing Methodology:

 

Testing Triangle

The core of a quality application stems from using emulators and simulators in development life cycles, primarily for automation and unit testing. Using automation, these tools can be quickly created and maintained at a large scale to provide maximum testing coverage, cheaply and efficiently.  By using a cloud-based emulator a developer can quickly spin up a device, write a few lines, and test it on the fly.

 

by browser

Finding A Balance

Recently virtual "real" devices have become widely accessible through providers hosting  physical devices at different software levels in the cloud. Testers leverage these virtual devices through integrated automation tools and manual testing by offering on demand access to "real" devices through their cloud service. This combination of cloud based virtual devices with testing automation, can greatly increase the number of tests, types of tests and device coverage while lowering overall cost per test and increasing the value of test results by automating data capture of failed test cases (including automation of screen captures, for failed tests). 

By using data and analytics, we can understand the types of users that comprise the market for a particular application. This then allows us to focus our tests on the user's platforms with the highest likelihood of use in the real world. We look at the following data elements when devising a testing matrix:

  • Browser
  • Browser Version
  • Screen Resolution
  • Operating System
  • Device Vendor

Automating Across Devices

Thankfully, testing automation code can make the complexity of testing across numerous devices simpler by being static across all devices and sizes.  Testing elements will  maintain the same ID or CSS Selector allowing automation scripts to target and interact with the test application object, no matter the device. Below is a sample snippet of test automation code that shows testing of application elements that can be leveraged across many devices:

@Test

Public void confirmLink(){

Driver.findElement(By.ID("LINK WE NEED TO CLICK");

Driver.assertURL("http://thisisourexpectedURL.com");

Driver.quit();

}

This code snippet is an extremely simple test that can run as fast as the network allows and can be run across every device and browser currently in market.

By building out these small tests across the entire application and across devices it allows our team to improve quality across many devices while maintaining efficiency. We can quickly deploy code and each time, these test scripts will run across the application- confirming a fix didn’t impact other functionality and consistently works across all devices. This is done faster and with higher accuracy than any human and ensures clients a quality deliverable each and every time.