Blog

Making test automation a reality: part 1

We talk about our experience in test automation moving from the discussion of its goals to advantages and disadvantages keeping in mind manual checks and automated testing debate.

5 July 2019

Test automation

Test automation basics
Striking a balance
What to automate and where to go the “manual” route?
Clear and visible intent of test automation

Article by a1qa

a1qa

Within the last 15 years, the role of test automation and its purposes in IT have significantly extended. From focusing on shortening testing times, now it strives to reach better coverage and more effective usage of test cases.

This can be narrowed to one concept – the so-called “quality at speed.”

Although, the level of adopting test automation is extremely low: in conformity with the World Quality Report, 61% of the interviewed IT specialists emphasized they have faced difficulties automating their QA activities.

The reason is clear – the applications modify too often and too much with every release. Notwithstanding, this proves their endeavor to build a robust and smoothly working software test automation product.

This is basically pure truth. In this article, we will define whether test automation is a “silver bullet” for IT business or not.

Test automation basics

As time goes by, test automation is focused on becoming smarter and more effective, as the confluence of AI, ML, analytics provides the systematic change in the benefits it can deliver.

To achieve valuable results of the projects, companies should consider that automated testing is not about replacing manual testing or getting a supplementary reduction in costs. In the most general sense, it is aimed to deliver quality at speed, take QA to the upper level, and make DevOps and Agile provide greater results.

Broadly speaking, we can divide test automation goals into 3 items:

Process-specific goals (faster time to market)
High-level goals (LTV – lifetime value)
Revenue-based goals (costs).

When considering the targets more thoroughly, we’ll say it helps achieve the following local SDLC values:

The problem to get the utmost of test automation lies not only in implementing the service but also in adopting it successfully. To build the process properly, one has to delineate when to automate and when it won’t work out.

To automate might be a good decision when	The software testing should not be automated if
Saving time is your top priority	Tests require human intelligence and intuition
Your project is a long-lasting one or consists of similar small operations	The process is limited to ad-hoc or exploratory testing
Tests should be run for every build of the application	Customer’s requirements (related to existing features) change oftentimes
Test cases are tedious and take a lot of time and resources to be executed	It’s one-time testing
Load or stress testing is carried out

Striking a balance

Manual testing is still meaningful in the QA process. That said, in many cases, it is no longer enough.

To speak generally, conducting manual testing takes physical time and effort, that’s why it can be extremely wasteful for continuous testing.

Meanwhile, test automation increases the speed of testing and saves resources. It reduces the number of tests that have to be run manually (for example, checks of multilingual sites) but does not eliminate it altogether. Though, while increasing overall test coverage, more time will be needed for maintaining test scripts.

A flawless test automation solution should be easily maintained. Even if you have about 5,000 automated tests, there is no need to spend roughly 5,000 hours to make it work.

Certain cases simply don’t mesh themselves well to automation.

UI testing. When conducting the display checks, manual testing will show how well your UI is in accordance with end-user preference, will provide a close connection with consumers, and help understand the problem on an emotional level.
UX usability tests. They provide the answer from the consumer perspective if the application is easy to use or not.
Exploratory testing. It is defined as a type of checking where QA engineers test the application, discover, investigate it on the wing, but test cases are not created beforehand.

It’s perfect if you can build an optimal duet of manual testing and test automation: when they do not diminish the value of each other – instead, perform testing in close cooperation.

Let us focus on some cases of such a successful “teamwork”:

You have an aim to accelerate the test results and release the new version of the product on the market.
Manual tests start where test automation ends.
There’s a large volume of regression testing, and the software product is launching frequently.
You involve both automation and manual checks to test different aspects of the same functionality.
Tests are “semi-automatic” and involve some part of manual intervention in order to proceed to the next set of automated tests (e.g., it’s an online store, and the purchase can be made by certain customers, which can be proved only with the help of manual intervention).

According to DZone’s 2019 Guide to automated testing, the shift left model is still growing in strength if to speak about automated testing.

Comparing with the results achieved in 2018, the number of respondents who begin automatically testing software at the development stage has grown by 12%. The number of the interviewed who begin automating at the QA stage has decreased by 9%.

What concerns manual testing, this is a little shift left: contrasting this year with the previous one, the percentage of IT professionals surveyed who begin manually testing software at the development stage has decreased by 2 points. The number of respondents who do it at the testing stage has increased by 4%.

At which stage does your organization usually begin automatically testing software — According to DZone’s 2019 Guide to automated testing

What to automate and where to go the “manual” route?

Let us define three main factors that should be considered in this case.

Factor	Answer the question
ROI (return on investment)	Do the resources spent on automating a specific scenario worth the profit derived from it?
Test complexity	Can it be automated? Does it make more sense to leave the test as a manual one?
System stability	At what level are the application and its parts stable? Can it be covered by automated tests? Where manual testers should intervene due to the frequently changing functionality?

Clear and visible intent of test automation

If conducted accurately, test automation can be quite lucrative. We will discuss the advantages that can be reached from automated testing a little after, but now we don’t want to miss the other side of the coin when test automation just won’t fit well into your project.

What are these traps one needs to consider?

The amount of new regression defects is slightly reduced comparing with those detected in the new functionality.
Implementation of automation is not a one-off effort, as you need to spend time to keep the automated checks relevant and get the most leverage from them.
Sometimes the results of testing are needed in a matter of hours, especially speaking about the checks of the newly introduced features. In this case, it can be more convenient for the project to move to manual testing and get the results ready ASAP.

However, there is no secret that the benefits of automated testing delivered to the project are clear and substantive. Let’s name some of them.

Test automation advantages	Description
Huge ROI	Despite the fact it takes some resources to write test cases, the ROI of this approach for wide-ranging and long-lasting projects can be enormous. This can be attributed to the ease-of-use of automated tests: they are adjustable, have a reusable nature, and can be applied in various ways. By implementing automation, one can dramatically reduce the cost of each test hour and find more hard-to-detect bugs.
Quick feedback on application health	Automation can give fast feedback concerning the state of the software product since it is revamped and assure higher efficiency of the dev team. It improves close integration between QA engineers, coders, managers, and more.
Earlier software defects detection	As test automation helps augment the entire development speed, it becomes more cost-effective and much easier to identify the glitch and fix it in the early SDLC phase.
Time savings	Automation of regression tests saves the time of the QA engineers to let them focus more on performing ad hoc testing and other testing activities. Furthermore, if involved properly, automated tests can be run with minimum or no human supervision.
Test data security	The effectiveness of checks highly depends on the quality of the used test data. Creating it manually is very resource-intensive, so testing is often conducted on copies of live databases. Automated tests can be designed so that they create and receive necessary data, change them without affecting other data, and return to the initial state.
Fast speed of checks execution	Test automation passes various steps quicker than a human does. This is particularly relevant for DDT (data-driven testing), as the same checks are conducted numerous times but with different data sets.