Assuring quality of mobile app: how we at a1qa tested Pokemon GO

In the last article on mobile apps testing we’d like to tell you how we tested on our own accord Pokémon GO that became a smash hit in 2016.

It should be immediately noted that we performed testing at the beginning of August, immediately after the game was released to the market. To date, many of the defects have been fixed. But the objective of the article is not to list the relevant defects but rather what to focus on while assuring quality of mobile apps.

By the way, we recommend mobile testers to test the existing apps from time to time. It helps hone the outside-thinking and investigation skills.

So, last year in July we downloaded the app to our smartphones to immerse into those times when at school we collected Pokémon pogs and rushed home to watch Ash and Misty’s adventures.

While testing we discovered that the games has its fair share of bugs and glitches. What have we found out?

First, we gathered all the information we could about the product.

What did we know about Pokémon GO?

• Multiplayer online role playing game.
• Free-to-play.
• It uses real location information to encourage players to search far and wide in the real world to discover Pokémon.

What else? We also knew that the app was mainly targeted at the users from the USA, Australia and New Zealand and was built on the Unity game engine.

It supported iOS 8 and 9 versions and Android 4, 5, 6. Keep it in mind that testing was conducted in August, and it was still a month before iOS 10 was released.

It’s an important moment that testing on tablets wasn’t required as the app wasn’t adapted to tablets.

To detect user’s location the app would use the GPS module and constant internet connection so these functionalities also had to be tested.

Having analyzed the info and the available statistics we made up a list of the following test configurations:

For iOS we included the devices starting from iPhone 5.

Android devices were more numerous. This was due to the vast number of versions of the system and various devices popular in different regions. We also managed to check the app on 5 different screen resolutions.

What defects did we find in Pokémon GO?

For a number of reasons, such as no access to the app backend (the customer is hypothetical) and no opportunity to get the scope of test data (and test geolocation) we could hardly detect all defects. But even those that we were able to detect would have been enough to compile a multipage test report.

All the defects we grouped according to their severity level:

Blocked defects:

• Authorization Failure. It seemed like every second user of the Pokémon GO faced this issue. And it wasn’t not clear whether the problem’s roots lied on the client’s or server’s side.
• Another frequently reported bug took place when the user tried to create the in-game character. The mistakes of the indistinct nature prevented users from starting the game.

Critical defect:

• The so-called three-step glitch. Pokémon GO is the augmented reality game. It tracks the nearest monsters and the user has to catch them. Initially, the steps printed next to the Pokémon would show the user how far the monster was. Thanks to the three-step glitch this feature became useless. This glitch showed Pokémon in the vicinity as always being three-steps (three paw prints) away, no matter how much you should walk.

As well experienced testers we proved all the defects with screenshots.

Major defects:

• Broken UI. Relaunching the app or clearing the cache would resolve this issue.
• No in-game characters and Pokémons images. However, by some indirect indicators we guessed that they should be there.
• AR mode issues. It turned out that many users couldn’t use the mode. We guess that the essence of the problem was connected to the software and hardware specifications.

Some average defects were related to the application several functions. Catalogue challenges, frequent camera issues and GPS module problems spoilt the monster-catching experience.

Moreover, there were minor defects that would rather make user smile than swear. For example, UI defects (look at the screenshot below) and the opportunity to exceed the number of items in the inventory.

Shall the customer be real, we would have generated the detailed bug report and product quality report. We would also perform another iteration of testing.

Pokémon GO became widely available in July last year and generated $200 million in revenue within 30 days of launch. The game surpassed Candy Crush Soda Saga, the top grossing mobile games of all times.

However, it was unable to keep up the momentum and in September already more than 10 million users stopped entering the game. The reasons are many, but the detected defects can be among them. Who knows how many users would have preserved their loyalty to the game but for all these bugs?

That’s it. We’ve taught the theory and tested the app. We hope you’ve enjoyed the read and will take advantage of it.

Any questions left? Welcome to the comments!