Interview with Mike Urbanovich: How to build a robust test automation strategy?

Today, we’ve discussed with Mike Urbanovich, Head of testing department at a1qa, how to set up a solid test automation strategy.

Mike is responsible for high-level team coordination, projects management, accounts management, and coaching. With 10 years of experience in test automation, performance, functional, penetration, localization and internationalization testing, he supports the range of projects for the Fortune 500 list clients representing diverse industries and coordinates a big team of technical and non-technical specialists.

Let’s get started to find out the benefits that companies receive with automated testing, what tests to automate, how to measure the effectiveness of a QA strategy, and the steps to make to implement test automation smartly.

Question #1. Test automation has been a leading QA trend for many years, and remains that way. Mike, what is so special about it?

Just have a look at the benefits (you know, dozens of them) that test automation brings to companies. Probably, this is the reason.

For instance, while manual QA engineers run only one test at a time, automated testing allows executing multiple tests simultaneously. Here, you aren’t limited to only one device, operating system, or platform. That’s why the coverage is higher, helping verify more features while curtailing testing time.

Or let’s take reduced operational costs as an example. By implementing test automation during long-term projects, companies make more profit while decreasing human efforts and redirecting them to core business processes. Of course, automation requires upfront investments at the initial stages. However, a well-designed test automation framework, a well-configured test environment, and properly selected tools help save a lot of money, time, and effort later on during the maintenance and support phase.

Not to mention accelerated velocity, optimized time spent on repetitive tasks, reusability of tests, and the possibility to conduct testing around the clock with the confirmation of the acceptable level of the system quality.

Question #2. If to speak about the automation strategy, what are the main factors to consider when choosing one?

To put it briefly, speed, budget, and reliability. Just three of them.

For me, the central one is speed, assisting in reacting rapidly to business risks and timely covering them. We are living in such an era where boosted release cadence is the priority, and that is indeed understandable. Companies move forward with their processes and wish everything could happen at lightning speed. QA automation can help ensure that. “Quality at speed,” like they say.

Question #3. But with all its benefits, test automation is not always the one to choose. Could you name some cases when automation is really a must?

In fact, test automation is a perfect match for long-term projects lasting more than 6 months. In this case, it speeds up the whole software testing process exponentially.

BTW, 100% automation is a common misconception. So, I suggest concentrating on which tests to automate to do it smartly.

First, smoke tests. By automating them, you timely detect critical errors, such as bugs in logging, loading an application, creating an account, and so on.

The next candidate for automation is regression testing, helping make sure an app functions as expected after code updates in the existing features.

Performance testing, of course, allows you to realize the maximum possible load, whether the system handles everyday load or overload, or is operable in the long run.

And verifying multi-platform operation. Test automation streamlines rapidly testing the application across all necessary environments, operating systems, browsers, and devices along with running them in parallel.

So, to summarize when automation is indispensable: you want to save time, the project is durable, the tests are repetitive and tedious enough to perform them manually, the tests run for every build of the software.

Question #4. What are the main difficulties that companies face when building a test automation strategy?

I’d say there are 3 most common challenges.

Firstly, an attempt to automate everything. Let’s remember that there are situations when manual testing helps receive more accurate results (we know them — usability, ad-hoc, exploratory testing).

Secondly, selecting inappropriate tools. The choice of test automation tools is like the Galaxy: free and open-source, fee-based, those customized by companies. But the point is not to choose the “buzzy” one, but the one that fits your project the best.

Thirdly, lack of skills of an in-house team. Does your QA experts specialize in a particular testing area, have enough knowledge to perform test automation and become a cross-functional engineer? That happens from time to time but it’s a big work to learn tons of things and upgrade to the next level. Good training, true wish to do that, and motivation will work in a bundle, I’m suppose. Otherwise, the dedicated team of QA automation Jedi is the option to save time and get ready-to-go people.

Question #5. About a test automation strategy. What are your suggestions to build it correctly, maybe some steps to follow to achieve the desired results?

Let’s go through the core steps together.

The initial step is to define the scope of testing — determine WHAT to automate. Automated testing is a win-game option for routine and voluminous tests running endlessly throughout the project.

The second step is to select the tools. Ask yourself, “Do I need a tool for a web or mobile IT solution? Do I have enough QA budget? Does the QA team have the necessary skills and possess the programming languages required to use it? Does the tool allow easily maintaining and reusing scripts? What have I forgotten to ask myself?” (smiles)

For the tools to operate smoothly, make sure they are able to interact with the given environment and platforms (iOS, Windows, Linux, and more).

The third step is to set up the environment — the very stage when the QA team prepares software and hardware to run the tests. To make the environment stable, consider a bunch of components: test data, DBs, OSs, tested IT product, network, and many more.

The following steps are writing, running, and supporting tests, as well as documenting the results.

After tests execution, it’s time to report the outcomes to analyze them and track the progress.

Question #6. What metrics to monitor to measure the effectiveness of test automation?

The list is vast. Test duration metric, I’d say, is the most imperative. It facilitates keeping track of the efforts spent on automated testing and monitoring the team’s productivity.

The percentage of passed and failed tests is another one. The name speaks for itself, but it also allows you to control the number of tests to execute further.

As we always stick to our clients’ needs, I find it crucial to monitor the percentage of covered requirements. This is how we show the ones reached, in progress, and those to manage later.

I think it’s also important to consider metrics that are vital for measuring business impact. For instance, a time to market metric helps gauge how automated testing reduces overall QA time, while a build stability metric demonstrates how often build failures occur due to problems with software under test. 

Question #7. What test automation trends do you think will rule the years ahead? 

The QA automation landscape is constantly evolving, so it’s a good idea to at least be aware of high-level trends that may help boost overall test automation efficiency and allow companies to stay ahead of the curve. Here are some key aspects to watch: 

• Integration of AI and ML. The World Quality Report 2023-34 respondents state that AI/ML contributes to improved productivity (65%), boosted speed (53%), and better CX (41%). And it’s not surprising. AI algorithms can scan the code and either generate required test cases or adjust existing ones as new functionality appears, automatically find and correct broken test cases, and even forecast probable issues. These activities allow scaling down maintenance costs, optimizing test coverage, and speeding up test execution. 

• Low-/no-code test automation. This practice allows the creation of automated tests without requiring proficient knowledge of sophisticated programming languages, using just visual interfaces or drag-and-drop components. On the one hand it greatly lowers time on developing scripts, simplifies project start, and doesn’t require an extensive learning curve for new team members. However, it may be inappropriate for complicated testing scenarios. So, I’d recommend that QA engineers still supervise low-/no-code test automation. 

• Hyperautomation. Gartner defines this concept as the process of automating the maximum possible number of IT activities within an organization relying on a wide range of advanced technologies, such as RPA, AI, iPaaS, and others. Such end-to-end automation boosts QA efficiency, decreases human error risks, and contributes to making testing workflows more intelligent, resilient, and adaptable. 

• Shift-left approach. Performing tests earlier in the development cycle has become a vivid trend that I think we’ll witness for a long time. Within shift-left testing, QA automation engineers are involved in the initial stages of the SDLC and perform tests continuously, rather than within a separate phase. This helps prevent software issues, optimize QA costs, and shorten testing cycles. 

Mike, thank you so much for this interview! 

Reach out to a1qa’s experts in case you need a personal consultation on how to build a test automation strategy.