With the global economic and social processes changing really fast before our eyes, it is almost a must to transfer a business to online mode in such a situation.

How to ensure the high performance of developed software products? How should those companies already presenting in the digital space adapt successfully to the conditions? Read on to find out.

Watching the global context change at the speed of light

Business representatives from many sectors of the economy were hit unexpectedly this year.

At the beginning of March, the world kept an eye on the largest sale of shares of the largest world companies since the economic crisis of 2008. Many experts have called this day “Black Monday”. In just one day, the shares price of USА and European companies decreased by 6-10%. In the new circumstances, the main goal of many large investors is not to increase profits but to mitigate the risks.

Before these events, the global economy was expected to grow by 3%. Now, experts speak about the recession, and companies are shifting to the “digital-first” paradigm to survive in the market.

Such a “path” has a flip side. On March 19, the European Commission proposed the largest streaming platforms to reduce the quality of visual content. YouTube worsened the default setting to a resolution of 480p. Amazon, Apple, and Netflix have changed the bitrate of the video stream for Europe. Facebook announced that it would cut the quality of their social networks. Even Disney+ decided to lower the bitrate.

Why so? Because high user interest increases the load caused on the servers, which can slow down or even pause the web service operations.

Even the largest global platforms were unprepared for such a sharp increase in the number of users. We can say that now we are witnessing the formation of new standards for the performance of web products.

Today, to maintain and increase the number of regular users, the platforms have to demonstrate stable operation under high and even stressful loads. But you can realize the capabilities of your software product after it has fallen or after accurate performance testing was conducted.

How to benefit from unstable conditions?

In the current situation, you can and should see the place for potential opportunities. For example, online delivery services, platforms for distance learning, and organization of remote work are already gaining momentum.

Now, it is important to prepare the product to smoothly functioning under peak loads, which can bring benefits to the business. According to Cloudflare, the current traffic growth is comparable to the period of the Olympic Games. After all, too slow or incorrect operation of the service will push away new users and force former ones to look for alternatives among the competitors.

This is why it is necessary to take into account long-term prospects. The rapid growth at the very moment may be followed by a recession process.

Ensuring software stability under risky conditions

Are you at risk? We suggest you draw attention to the following factors that may arise in case of no performance testing in place.

• Product release

Preparing to introduce a new software product on the market? Then you could have already conducted a series of tests. It is also a right decision to revise the performance requirements taking into account changes in the current potential load indicators on mobile and web applications.

Do not miss the opportunity to test your product before release, even under unexpectedly high loads.

• Unadapted mobile app to work on different platforms

Given the limited functioning of the software product, one should not expect a sharp increase in user load. But could have, actually! With the growing interest in the service use, business benefits are expected to grow as well.

Therefore, with no preparedness of the resource for several platforms, it will most likely to become an obstacle for business development rather than a constraint.

• New features

Software product upgrades help maintain user interest and meet current market needs. But when updating the current characteristics, one has to face difficulties. It is important to carry out not only fragmented checks of the performance of new sites, but also evaluate the capabilities of an already holistic system to function in the previous model and study the effect of changes on product performance.

Ignoring bugs when introducing new functionality can result in long-term malfunctions and huge losses because of untimely fixing.

Need professional assistance in ensuring software stability? The a1qa performance testing team is here to help.

• Ignorance of peak load points

Understanding the “boundary” capabilities of a software product allows you to analyze the behavior of the system now and build viable forecasts more accurately.

Performance testing can help you realize the critical load points on the system, thus, getting unambiguous answers to the following questions:

1. What are the peak load points?
2. How does the behavior of the system at these points differ?
3. For how long is the system able to work with a sharp increase in load?

• Switching to new software

Software amendments present risks for the product stability. For instance, to make sure that the system is operating as required after migrating to a new database management system (DBMS), it is worthwhile to conduct a series of load tests.

This approach will help identify differences in the behavior of the system in familiar and new conditions.

To mitigate the influence of these factors on the health of the software product, take some emergency measures to stabilize the system’s performance.

Supporting software product stability: three emergency measures

To speed up the stabilization of a software product while increasing user load, you can take the first steps by yourself and ensure:

• Download optimization

By temporarily reducing the amount of visual content or turning to file compression, you can reduce the loading time of each page. The scenario of minimizing the size of CSS and JavaScript files can also help the pages load faster.

• Server capabilities changes

You also need to make sure that the system has no limits on the permissible number of connections or other restrictions. At the moment of moving to the peak load, it is worth developing the site’s throughput capabilities.

• Resource scaling

Even if you are now handling the user load, tomorrow, you may need a room to maneuver. We advise to get some more software or hardware systems for processing requests.

In this situation, there are even more effective measures that can be taken.

• Set up system monitoring to determine the most loaded components.
• Increase the capacity of the infrastructure on which the application is deployed using cloud solutions.
• Configure the profiling of the DB and servers to identify the slowest queries, analyze them in order to optimize the query processing time.

The next active action to be undertaken is to conduct performance testing. The a1qa performance team is here to ascertain that your software product is ready for significant loads, to find system bottlenecks, and give recommendations of quality improvement:

• With the traffic increase, we see the number of users and the amount of information stored in DB growing. To ensure the ability of the system to cope with this growth, volume testing is carried out.
• Testing the software configuration helps determine the best combination of the server components, which reduces the cost of resources needed to maintain the environment.
• Stability testing helps determine the optimal scalability options with a sharp load increase or decrease. Automating the behavior of the system in such conditions helps save resources multifold.

In this article, we described in more detail the approach to performance testing and the main stages of this process.

Summarizing

Current global challenges that arise due to urgent transformation are a great opportunity to provide a new incentive for business development. The quarantine measures made people spend more time online, and such a rapid increase in interest to online products allow businesses to diversify risks.

Increased traffic leads to unstable application functioning, which can be solved by applying performance testing helping identify peak load points as well as determine the behavior of the system during prolonged use.

Does your product need a piece of a QA expert advice on quality assurance issues? Contact us to get a personal consultation.

System performance testing is carried out to check compliance with the specified requirements, ensure that software applications function smoothly under the expected and substantial workload and are ready for real-life operation.

In today’s article, we will talk about the sequence of checks conducted and the features of all 9 main stages:

Stage 1 – Software analysis and requirements preparation

The development of the system under test must be completed. Otherwise, the results received will be invalid.

Before conducting performance testing, it is important to make sure that the software configuration has been adjusted appropriately. When the system is deployed, functional testing should be carried out to ensure that the major functionality used for performance checks operates correctly.

System analysis includes the examination of its features, operation mode, and peculiarities. Detailed analysis is required to achieve the following purposes:

• simulate the most adequate user behavior patterns and load profile
• determine the necessary amount of test data
• find the system bottlenecks
• define the software methods for monitoring.

The criteria that the system technically corresponds to are called the requirements.

The major emphasis should be put on defining the success criteria for the tests conducted which are usually included in the SLA (service-level agreement).

The requirements defined during the first stage will be compared to the received results in order to evaluate the behavior of the product and system units and determine the bottlenecks.

The following metrics are used as success criteria:

Metric	Definition
Virtual users	Software that acts just like a real user would when making requests to the application.
Response time	The time system takes to process a request from a user.
Request per second	A number of requests sent to the destination server.
Transactions per second	A number of transactions sent to the destination server.
Error rate	The percentage of errors from the total number of responses per unit time.
CPU usage	The percentage of the time the CPU uses for processing the workflow.
Memory usage	The amount of RAM available for all applications and processes.
Disk space usage	The information on the usage of the disk space.
Network	The network usage gives information on the data transmission speed form the server to the network adapter.

Sometimes the system analysis and requirements collection take place throughout the whole project lifecycle. If the software is already in use, the analysis of the target audience and their actions will show, which parts of the application are of critical importance and how the interaction with the system occurs.

Book a consultation with the a1qa performance experts.

Stage 2 – Strategy design

The testing strategy is developed on the basis of detailed system analysis and describes an extended approach to performance testing.

System analysis includes all the information regarding the software, testing goals, application requirements, test stand configuration, testing monitoring, user behavior scenarios, load profile, load model, performance testing tools, planned test runs, and the way of providing the results.

Test stand configuration

The performance testing results may be influenced by various factors such as test stand configuration, network load, database content, and many more.

Therefore, to receive the most reliable results, performance testing should be conducted in a separate environment with features and configuration close to the parameters of the real software.

Test stand elements	Features
Hardware	CPU (number of cores, type, clock speed) Random access memory (space, type) Hard disk (type, speed)
Software	Operational system (version and service packs) Application server (version and patches) DBMS (version and type)
Network	Network mapping Bandwidth performance Communication protocol
Application	Architecture Database (structure, data) Software required for the system operation

Load profile and load model design

Statistics on the application usage are collected as a part of the performance testing process. The collected data is necessary for designing the load profile – a user behavior model.

Various load models can be applied to the same test. For example, one new virtual user may be added every five minutes or all users – at once. The key feature of the load model is a query rate, test duration, and the number of users.

Choosing optimal performance testing tool

The following tools can be used to conduct performance testing:

Tool	Advantages	Disadvantages
	Tests with complex logic and dynamic parameters correlation can be run. Web application testing can be conducted (including API and web services, database connections). Allows simulating the behavior of multiple users in several parallel threads and apply a heavy load to the web application.	Has some AJAX traffic reproduction problems. There’re limited reporting functions. Requires many resources of RAM and CPU for test launching.
	Offers distributed architecture (VuGen, Controller, Load Generator, Analysis). Tools for working with various protocols are available. Provides a possibility to generate reports and create report templates. Detailed action logs for each virtual user are possible.	The number of virtual users is limited. Updates are rare.
	Test results are kept in an MS SQL database and are well-structured. No limit for the number of virtual users under the license.	Declarative tests have limited functionality for scripts logic. Declarative tests are the only testing type available for SharePoint.
	It supports HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP, XMPP. Its’ convenient for API queries testing. Load can be distributed within a block of client machines.	There’s no GUI. Subqueries from HTML pages are not loaded automatically. WS is not suitable for server responses that are to be used in subsequent transactions. No detailed reports are available.

Stage 3 – Load generator configuration

In order to obtain high-quality results, performance testing tools should be installed on a load generator. It is a virtual or physical machine located in proximity to the application server(s).

If a high load is to be generated, the resources of one machine may be insufficient. In this case, distributed performance testing is conducted.

Get a free consultation.

Stage 4 – Server and load generator monitoring

The metrics defined at the first stage are used to evaluate the software performance and define the system bottlenecks.

The information obtained from the monitoring of all application servers and resources is used for performance testing results analysis.

Software monitoring can be conducted with the help of the tools for system hardware resources consumption control.

The most popular tools for hardware monitoring are enlisted below:

• Nmon

The tool saves and displays a brief overview of the following metrics: memory, hard disk, network, file systems. It does not consume much of the CPU resources (usually below 2%).

Nmon is available in two modes.

1. Online mode.

When the tool is launched, the command line screen will display the monitoring information.

2. Data record mode.

The data collected during the monitoring is saved in a CSV file that can be used as a data set for creating diagrams and manual results analysis.

• Performance Monitor

The data is gathered with the help of a data collector. The results of the monitoring are contained in a BLG file, which is later converted to a CSV file.

PAL (Performance Analysis of Logs) can be used to structure the data and create diagrams.

• Zabbix

The software follows the “server-agent” approach. The server collects all the data and allows viewing the monitoring history and setting up metrics and rules.

• Grafana

This very monitoring data visualization tool is able to analyze data from a time-series database for Graphite, InfluxDB, and OpenTSDB.

• New Relic

The performance tracking service provides analytics for each piece of the environment. The tool is an effective way of viewing and analyzing huge amounts of data and getting information about real-time actions.

Database profiling

Database profiling helps reveal, which queries, procedures and triggers decrease the application performance. This quick software diagnostic allows fixing errors instead of searching for them.

• PostgreSQL is a free object-relational database management system. pgBadger profiler can be used for reaching the performance testing goals.

pgBadger is a Postrgre SQL log analyzer that provides general query statistics, information on slow queries, data on connections and servers, and others.

• MS SQL Server Profiler is a tool for tracking, reconstruction, and debugging of the MS SQL Server. It allows creating and processing queries, reproducing and analyzing queries results.

Web servers profiling

All the systems are unique. The use of profiling helps understand how to increase software performance.

JProfiler combines CPU, memory, and thread profiling, so it’s easy to figure out what should be optimized, eliminated, or amended. This tool can be used both for local services and remote sessions.

PHP profiling

Code profiling helps reveal slow code elements and improve them if necessary.

• Xdebug is a powerful tool applied for analyzing PHP code, detecting its bottlenecks and slow elements.
• XHprof decomposes the application into function calls (methods) and creates the statistics on their resource consumption. The results include various metrics: the amount of allocated memory, number of function calls, execution time, and many more.

Stage 5 – Test data generation

Let’s discuss four types of test data generation:

• Code

Scripts developed in various programming languages (Java, Python) allow creating users, passwords, and other values necessary for correct data usage.

• SQL queries

A database can also be filled with the help of SQL queries. This method is available only if there’s access to the database form the server-side.

The approach can be implemented as follows: first, one creates a completed DB in MS Access with fields identical to the database on the server-side; secondly, a dump file that will contain requests to add information to the DB will be created.

• API calls

API requests can be used to fill the DB with items for sale or user data. One or two calls will be enough to send the data to the server.

• Interface

A script mimicking the steps of the user registration process can be developed for filling the database through the system interface.

New users created by the script are recorded in the database. A snapshot of the file system can be made in order to use the created users for test execution.

To get a consultation, drop us a few lines.

Stage 6 – Load scripts development

At this stage, performance testing scripts are created with the help of the chosen tool.

This stage includes:

1. Scripts analysis

The prepared scripts should be analyzed before the development of load test cases. When each of the steps has been checked manually and no errors have appeared, the load script can be recorded.

2. Test cases development

It’s important to create a template with no parametrization for each script.

Using Fiddler or any other proxy server, one may ensure that all the queries from the template are similar to the queries from Fiddler. When checked, each request should be parametrized to ensure that it works properly for various users.

3. Scripts debugging

When a query is parameterized, it should be executed X times for various users to ensure its proper functioning. If each scenario in the scripts works correctly, the script debugging stage can be considered complete.

Stage 7 – Preliminary tests launch

Preliminary checks help to determine the most appropriate load model for the system. They also show how the application resources are used and whether the load generator(s) power will be sufficient for executing full-scale tests.

The checks should be launched under different load models. Thus, the testing load is determined on the basis of software monitoring and results analysis.

Stage 8 – Tests execution

Performance testing process consists of the following types of checks:

1. During a stress test, the load will be gradually increasing until one of the test stopping criteria is reached.

Test stopping criteria:

• The response time exceeds the set value by several times.
• The critical level of the hardware resources usage is reached (CPU>80%, memory>90%).
• The number of HTTP errors exceeds 1% of the total request number.
• Fail of the system software.

Stress testing shows how the application reacts to an increased intensity of transactions and reveals the upper limit of the software performance.

2. A load test is executed for an extended period of time (4-8 hours).

If the results of the stress check showed that the system hadn’t coped with the required load, the load test is executed under the load comprising 80% of the maximum value reached during the stress test.

3. The stability test is conducted under an expected load for an extended period of time. When the maximum number of users has been reached, the load remains the same until the end of the check. The duration of the stability test can take up to several days.
   
4. Volume test is used to evaluate the system performance under increased data in the DB. To conduct volume testing, one should fill in the database with the necessary amount of information.
   
5. Scalability test shows how the software performance is modified when the changes in the size of the application are made.
   
6. During the component test, the load is independently applied to the software units.
7. Rush hour check is executed to evaluate the system behavior both during the peak load and right after it. The load is applied to the application in a jump-like mode.
8. Configuration test reveals the most appropriate software configuration, which can provide the required performance.

Stage 9 – Results analysis and reporting

The results of the previous stages are used to prepare a document describing the outcomes of the executed tests and the implementation of each stage.

A testing report includes the following information:

• Performance testing goals
• Test environment and load generator configuration
• Application requirements
• User behavior scripts and load profile
• Statistics on the key performance characteristics (response time, number of queries per second, number of transactions per second)
• Information on the maximum number of concurrent users under which the system is capable to cope with the load
• Information about the number of HTTP errors and their types
• Diagrams showing how the system performance depends on the number of concurrent users
• Conclusion on the software performance in general and its bottlenecks if any
• Recommendations on how to improve application performance.

Contact us to find out how performance testing experts can improve the quality of your solution.

Performance testing encompasses a wide range of goals and ways of achieving them. Today, we will compare two major techniques of software performance testing and explore how the most effective one is conducted.

To check the load the system can withstand under certain conditions, one of two methods can be used:

1. Implementation of online services to generate statistical load on certain pages of the application.
2. Simulation of real user’s actions by launching the developed load scripts in several parallel threads (or “user behavior method”).

The first method is easier to implement though it has several disadvantages. The requests created by performance testing online services are unpredictable in their intensity.

Online services do not consider some peculiarities of the user behavior, for example, dynamic parameters correct transmission, browser data caching emulation, correct time delays between the actions taken by a virtual user.

The load applied by such services is closer to a DDoS-attack rather than to an efficient performance testing. Quick testing, which does not correlate with the real situation, often turns out to be useless and even counterproductive, as the results of such tests may be unreliable and erroneous. It may cause reputational and financial risks.

The user behavior method gives precise results, helps define bottlenecks in the system’s work and eliminate them in time.

To obtain the results representing the actual system performance, you should reproduce user actions as accurately as possible. Virtual users can help reach the actual system performance load.

The virtual user is a script that makes requests to an application, mimicking the actions of a real user.

The high precision is achieved with the help of the request flow design. The request flow resembles the requests observed in the application production environment. Virtual users also interact with the system using a behavioral model.

Let’s discuss six rules and recommendations to apply the load on the system using the behavioral method:

1. User behavior scripts design

Before getting down to load scripts design, QA engineer should develop the most realistic patterns of virtual user behavior. To do this, it is necessary to answer a number of questions:

• What is the total number of the users working with the application (an average number per hour, day, month)?
• How many users work with the application at the moment of the peak load?
• What actions do users perform and how often?
• How are the roles in the system distributed among the users?

The answers can be obtained from the statistical data collected by web analytics tools, such as Google Analytics, Piwik, Adobe Analytics, etc. Transactions frequency analysis allows finding the key business transactions and design user behavior scenarios that will be used for load scripts development.

When the application is in the pre-release stage, test scripts can be created to check the main business processes and major functional modules. This information can be obtained from both the business analyst and the customer. The scripts can also include rare but heavy requests pointed out by the development team. Among such requests can be, for example, those processing information from many database tables and, therefore, applying a significant load on the system.

Besides determining the sequence of real user actions, the QA performance engineer studies the statistical distribution of time delays between transactions or the think time.

During the think time, the user does not perform any actions, but he or she is in the process of executing them or thinks over the next step (for example, enters a password, chooses goods, reads a text, writes a comment, etc). The time delay for the authorization transaction may take up to ten seconds, the website menu navigation or search process does not exceed five seconds. These delays must always be taken into consideration during the scripts’ development.

A test script for an online shop is given below as an example. The delay time is indicated in brackets.

The script “Authorized user: order payment”

1. A user opens the home page (5-10 seconds)
2. The user clicks “Login for registered users” (5-10 seconds)
3. Fills in the login form and clicks “Login” (10-60 seconds)
4. Chooses a good and clicks “Add to order” (30-60 seconds)
5. Clicks “Continue placing the order” (10-20 seconds)
6. Steps 4-5 are repeated a random number of times (from 1 to 5)
7. Clicks “Place the order” (10-20 seconds)
8. Clicks “Pay the order” (10-20 seconds)
9. Fills in the payment form and clicks “Pay” (10-20 seconds)
10. Clicks “Confirm payment” (10-20 seconds)

The number of developed scripts depends on the size of the application. As a rule, from 3 to 10 scripts are created. Each script contains up to 10 steps describing user actions.

Сomplex applications require more scripts. For example, a portal dedicated to facility building services, including repair works assignment, certificate and allowance creation, and many other functions required more than 20 test scripts written by the a1qa engineers. Each of the scripts included 15 to 20 steps.

Request a free consultation with the a1qa engineers.

2. Load profile design

A load profile is also created at the scripts’ development stage. The load profile shows how the transactions performed in the system are distributed among the users of different roles.

The load profile can also be created with the help of page traffic statistics for real users. Apparently, a user does not visit all the pages. For example, when he or she adds a good to the chart in an online shop, there is always a chance that the user will not get beyond the order placing page. Each action has a specified degree of possibility and is performed only by a certain percentage of users. The following ration was observed during the testing of the online shop:

Therefore, under the load of 1,000 users, 700 of them will be executing Script 1, 200 – Script 2, and 100 – Script 3.

The prepared load profile allows avoiding applying more load than it is done by the real users.

3. Load scripts design and debug

When the testing scripts are created and approved by the customer, the QA performance engineer proceeds with scripts development.

As a rule, the development starts with recording the data exchange process between the server and the browser at the level of data communication protocols. The testing tools allow running the recorded process in several parallel threads for generating the necessary system load.

At first glance, the task seems quite simple. However, it is important to make sure that the recorded scripts are working properly. The correlation of the dynamic parameters sent to the server (such as authorization token) should also be set up.

The performance testing tools allow parametrizing the request itself (the header, cookie files, body parameters) in order to achieve uniqueness. Parametrization is done with the help of built-in functions, however, sometimes code writing might be necessary. The uniqueness of virtual user actions is reached with the help of functions performing loops, forking, and value randomization.

When developing scripts, the QA performance engineer should carefully compare the network traffic generated by the load scripts with the traffic from the browser of a real user. This can be done with the help of the sniffing tool Fiddler.

The structure and the size of requests and responses must be exactly the same. The key server responses should be checked for proper performance as the modern systems may respond to a user with a special error notification rather than an HTTP code when an error occurs. The performance testing tools allow us to set up criteria for checking server responses.

Besides the steps described above, a particular emphasis is also paid to the browser data cashing emulation and simulation of other features of the client’s work. This approach helps to apply a load to the real one as close as possible.

During the final stage of the scripts debugging, a series of preliminary tests with a low load is conducted. The measure is necessary to ensure that the scripts work correctly for various user accounts: dynamic parameters’ transmission, HTTP error codes, and request/response structure are checked.

Preliminary tests also allow finding the most suitable load pattern, for example, +1 virtual user every 3 seconds.

The above-described steps are often missed, and this leads to errors emerging during load scripts execution at the stage of final testing. As a result, additional time may be required to find and fix the defects.

Get a free consultation with the a1qa performance engineer.

4. Generating a realistic set of test data

Before launching the final tests, test data must be generated in the amount sufficient for correct scripts operation. This step should never be omitted as it may influence testing results. When a thousand virtual users performs actions from one account, the requests will be cashed on the server-side.

Moreover, there may be conflicts among virtual users performing similar actions. For example, when users add goods to the chart in an online shop, there is a chance that they will click the “Pay the order” button at the same time. This coincidence will cause additional errors. Furthermore, this approach will not provide realistic performance results, as only one account will always be updating in the database.

There is no universal approach to the tests’ data generation. As a rule, the most appropriate method is chosen after the system analysis. At the stage, the types and the structure of the test data are discussed with the development team. Both the QA performance engineer and the development team may execute the data generation process.

The test data can be classified into three main types:

• User accounts with various roles

Special attention is paid to the uniqueness of the virtual user names and their passwords that must correspond to the specified safety criteria. For convenience, user names are created with the help of established patterns, for example, all test users may have the last name “a1qa” and the first name “PerfTestUserX”, where X is a number from 1 to 1,000. The use of such a pattern helps identify users in the database.

Sometimes it is necessary to create users with profiles of different sizes. A “heavy” user has a lot of database entries and connections among them. A “light” user has relatively few entries. Performance testing of a billing system showed that the results for the “heavy” users were significantly worse.

User accounts can be created with the help of the user interface; through API calls made by automation scripts or by updating entries in the database directly.

• Import files of different types and sizes

User accounts need data to work with. Sometimes it is necessary to measure not only the handling time for requests but also to evaluate the load speed of the files. These values may be of the highest importance, for example, for enterprise software products.

Test data is organized in a format, which corresponds to a particular type of upload documents and has specific features: type (MS Word. Excel, PDF, JPEG, etc.), size, and content. Data generation may be executed by the Fsutil command entered the command line or with the help of PowerShell commands, which generate files of the necessary size and quality.

• Valid data for filling in forms

Web pages often contain forms to fill in. These forms consist of various fields that should be completed with personal data. The server processes the received data and, as a rule, saves it to the system database. For successful data processing, the validity requirements must be observed. For example, the entered data (e.g. phone or credit card number) should have an appropriate format. If the data is not valid, the server may return an error message.

Another approach for valid data generation is to develop code, which generates the values in the appropriate formats. The performance testing tools built-in functions for randomizing values can also be applied.

When the test data types and means of their generation have been identified, the QA performance engineer should forecast the amount of data sufficient for successful test completion and identify the most efficient way of clearing the database.

5. Choosing load scheme

The aim of the performance testing should also be determined before the start. The testing process may be conducted with the help of several test types. The schemes below show how the load is applied during various types of tests.

Scheme №1 – a continuous linear increase in the number of virtual users.

Scheme №2 – a step-like increase in the number of virtual users.

There are time spans when the number of users does not change.

Scheme №3 – a linear increase in the number of users until a certain moment, after which the number of users remains the same for a long while.

Scheme №4 – the number of users is maintained at an average level, as in the scheme №3, but there is a sudden load increase.

Scheme №5 – a custom change in the number of users, which can combine the schemes described above (№1-4).

As a must-have, we recommend conducting stress testing and load testing.

Stress tests help find the upper limit of the system performance (schemes №1 and 2). Load tests allow ensuring that the system performance does not decrease under an average load lasting for a long time span (scheme №3).

At the traditional a1qa summer professional conference, the QA specialists made a presentation partly devoted to the topic of load testing. See below the reasons for conducting this testing type and the ideas on how to do it right.

The choice of the loading scheme may be conditioned by the peculiarities of the system and the requirement for its performance. For example, the system of limited bonuses’ distribution on a large gaming portal is scheduled for a specific date. The owners of the portal are expecting an influx of users and, therefore, want to be sure that the system is capable to cope with an extreme load. The scheme №1, with an increase of 3,000 per 10 seconds, can be applied for testing of the system. On the contrary, corporate systems are used by approximately the same number of users but during the whole working day. The most suitable approach for this situation is to apply the loading scheme № 3 during 8-10 hours.

Another example to look at is online shops that are exposed to an influx of users during seasonal sales. The expected number of visitors for the next sale can be predicted with the help of the statistics on the normal period and the period of peak loads. High system stability during the peak loads may be a requirement for the performance of such a system. In this case, we recommend using the scheme №4 as an additional load scheme.

Request a free consultation with the a1qa experts.

6. Distributed load testing from multiple geographic locations

We recommend placing load generators as close to the testing environment as possible. This approach allows avoiding problems related to the network carrying capacity as the delays may vary from milliseconds to dozens of seconds. Placing the load generator in one local network with the testing environment ensures the fastest speed ration between the server and the client. Thus, the application performance is tested in the conditions close to the ideal.

Sometimes the task is, on the contrary, to measure the key performance parameters by simulating the actions of the real users from remote regions. This is the case for a gaming news website visited by users from all over the world. In order to provide a realistic load, the geographical factor should be taken into consideration. The necessary information can be obtained from web analytics tools.

To accomplish this task, a distributed load test should be performed when virtual users are distributed among several load generators located in different geographical regions. Virtual machines rented from cloud services (Amazon WS, Microsoft Azure, Google Cloud, etc.) are used as load generators for this testing type.

During the test configuration, a client (master, test controller) and servers (slaves, test agents) are created in, for example, the US, Europe, Brazil, and Australia. A client is a machine collecting and showing the results, whereas servers are used as load generators. The load is applied from all servers simultaneously.

The testing results evaluate the system performance for the end users and the necessity to place additional servers in the regions with the worst performance results.

Now you have seen the basic rules of how to apply the load to the system. The result received with the help of the user behavior method is close to the actual state of the system performance.

An additional advantage of this approach is the possibility to find functional defects that appear only under the load. Scripts developed according to the actions of real users to help localize such defects.

Book a free consultation with the a1qa experts and find out how we can enhance the quality of your software product.

Have you ever faced the situation when everyone wants something from you at once? Your mailbox is full, the phone is ringing off the hook, Skype is abuzz with instant messages…

You keep responding and addressing all the issues. However, at some point, you exhale, take a break, and temporarily stop noticing what is happening around you.

The same thing can happen to your application or website. Unfortunately, such breaks can cost you a lot. A user won’t wait until the webpage is loaded, he or she will simply opt for your competitors.

Performance testing will help timely prevent all these possible breakdowns. How? Let’s discuss this in detail.

What is performance testing?

Performance directly affects the functioning of the entire system, ensuring the reliability and ongoing operation of the software even at the moments of the maximum load.

These tests help understand the system behavior in different scenarios. What will happen if the number of users increases from 1,000 to 10,000? Will the loading speed change? Will all the payment methods keep functioning correctly?

The necessary verifications are fulfilled by performance testing engineers who consequently provide the answers to these questions.

There are many types of performance tests that can be conducted on any system.

Let’s identify what services in terms of performance testing are in the highest demand nowadays and what types will be suitable for your business.

What do performance testing engineers verify?

There are six main types of performance verifications: load, volume, stress, stability, scalability, and client-side testing. Depending on the project requirements, these can be applied, mixed up and customized to provide maximum information on the solution performance.

Most often, performance testing engineers are hired to do the following:

• Check the compliance of performance characteristics with the stated requirements. This is especially important for product development companies that modify their solutions for each customer.
• Determine the level of maximum load the software can withstand. Stress testing helps to prepare a product for a successful release with a large-scale advertising campaign.
• Reproduce post-release defects and prepare the release of a product designed for numerous users. The companies constantly increasing the range of proposals through the release of additional products opt for such verifications.

Engineers assess the quality of the client’s system, find bottlenecks, and offer recommendations for their elimination.

QA specialists can also conduct an independent audit of internal or third-party performance testing team, as well as consult or train specialists.

To order a free consultation with a performance testing engineer, click here.

When is performance testing required?

Let us list three cases when you’ll definitely need to involve performance testing engineers into the project.

• You’ve undertaking system migration to a new hardware infrastructure or software platform, for instance, switching from one database to another

What should be done?

In this case, it’s vital to carry out load testing of the entire system, which functions using the old and the new database, and then compare the results.

The QA team monitors how migration affects the system speed (for example, the change of page load time) rather than the overall level of the database performance.

The client receives a comparative summary report with the results of system performance before and after the migration.

• You’ve decided on refactoring, or restructuring the existing code base.

Code changes are often associated with the need to improve performance, optimize queries or code itself, make it easy-to-understand, eliminate duplication, etc.

What should be done?

To evaluate the optimization, it is necessary to perform load testing before and after changing the code. Therefore, the client will see whether the refactoring has led to the desired result of the system optimization.

• The client part of the application has deteriorated after the system release.

What should be done?

Client-side testing is to be performed. The system is checked for a single user, and all verifications are performed on the side of the client.

These procedures can be carried out on any environment, since the system doesn’t undergo the load, and testing doesn’t affect the software performance.

Checking the speed of web application rendering helps identify issues regarding compression, caching, optimizing images, minifying/connecting resources.

It’s often cheaper to increase productivity on the client side. Moreover, if the front-end part of the product is weak, any optimization of the server part will be affected by the slowly operating client part.

Due to an integrated approach, the a1qa engineers analyze all internal system components, prepare detailed results and recommendations for improving product performance.

a1qa performance testing services

• Full-cycle testing by a dedicated team

The team provides full support of the product performance, from requirements gathering to the detailed report preparation on the performance level with a list of bottlenecks and recommendations for fixing them. The service may include various types of testing, chosen individually for each specific product.

If the QA and Dev teams cooperate, the process is improved by reducing the time for fixing defects.

How is the work organized? QA specialists run tests and fulfill the system assessment. If the defects are detected, the QA team passes them to the Dev team for quick fixing. Then the software testing engineers restart test cases, verify the correct system performance, and continue to test the solution.

• Regression testing

New versions of the solutions appear frequently. Every new version can contain changes in the operation of separate system modules or the implementation of new features.

Therefore, the engineers make sure the changes haven’t affected the system performance. The experts use the existing test cases supplemented by additional scenarios to test the modified functionality.

• Configuration of the online monitoring system

The engineers launch a new system that allows real-time monitoring of testing procedures.

Due to the well set up process, the client at any time can receive the access to the information on the testing scope, the number of activities fulfilled, their duration, and all the issues found.

• Database optimization

The a1qa team verifies the speed of database queries, suggests the ways for its optimization, seeks and detects bottlenecks in the database architecture, and checks whether the databases are built correctly.

Order free consultation to determine the appropriate service for your product.

The pros of the a1qa performance testing team

Today, more than 25 engineers work in the a1qa performance testing lab. The user behavior approach, which takes into account the aspects of real users’ activities, forms one of the core advantages of our team. How does it work?

Scenarios for applying the load to the system are closely connected to the actions completed by real users in the system.

The neglect of user behavior may lead to inaccurate test results that cause system failures.

The expertise in big data testing and database profiling allows the engineers to conduct an audit to check whether the speed of queries slows down due to an increase in the number of users, as well as suggest possible ways of optimization.

Summing up

Performance testing is an integral part of the overall solution verification. Defects can lead to system failures, a considerable loss of profit and harm the software reputation on the market.

Timely verifications can ensure smooth software functioning even during the periods of maximum load.

Obviously, performance testing is a very important aspect of launching and running any system, but where do you start from?

The first thing is to determine the optimal performance testing service that needs to be conducted on your product.

Book a free consultation with the a1qa experts to find bottlenecks in the performance of your solution.

In the previous articles we covered all the preparatory stages. Discussed the cautions you should keep in mind while the performance testing. When the specification is agreed, script is developed and the environment was carefully set-up, you are, finally, ready to run the tests.

Tests execution

Tests launch should be agreed with all project members that in charge of the platform where the tests will be launched. If the tests get launched at the PROD platform, you are to isolate virtual users from the real ones.

Furthermore, a day before launch you should run monitoring process. Collect data describing the system state when it doesn`t interacts with users. The information you get will assist in tracking down the influence of the background processes.

Result analysis

Before the analysis start collect all the data from the monitoring and profiling systems. You need to convert all them into one format to import in one system, which can be HP Analysis, MicroFocus Truelog Explorer or something else.

When running analysis consider the goals that were set when the performance testing was required and agreed. It is a good to compose 2 reports instead of 1. The first report contains graphics and basic information to explain the system state. Target audience of this report includes the project managers and the customer representatives. The second report should include profound information and statics about all the system components. It should also contain description of system problems and improvement recommendations. The target audience of this report includes developers and system administrators.

If this test launch wasn`t the initial one, then compare the performance of the previous tests and the held one. Keep in mind that all the important project artifacts should be stored.

Using this overview of DOs and DON’Ts you can move to practice performance testing being aware of what to do and how.

After the development specification and scripts, your next step is environment set up. That the stage where you provide the conditions to successfully run the testing process. Let`s go through the DOs you shouldn’t ignore.

To set up the environment you are to go through several points:

• Data input
• Load generator set up
• Adjust monitoring and profiling
• Application set-up

Data input

Since you need to see how the system behaves in the real conditions, you need to put it in the state that would be close to the reality. To emulate the real system state, fill the data base with the necessary information using the SQL script. For you emulation you also need to create necessary files in the storage, several accounts in the application and kit of objects usually utilized by users.

Load generator set up

Following the testing strategy generators can be installed within the limits of the local network or outside it. As long as you set the generator, check its connection with the controllers, it shouldn`t be blocked by the firewall. Moreover, carefully verify the settings of the security system to avoid identification of testing procedures as DDoS attacks, it can break the whole testing process.

Monitoring and profiling adjustment

During the process of performance testing remember to collect all the data characterizing system work under the load, you`ll need them for the analysis. To held full scale monitoring you should track the changes in the hardware & software, and run profiling of the application code. The track list can be shortened, if you don `t need a profound analysis on the project.

Application set up

Before launching the tests check that all the system components are compliant with the production configuration. As during the development process the parameters like: data caching, logging level, components clustering, load balancing and accessing the third party servers get changed for a better application adjustment. That is why don`t forget to verify that the system parameters are upgraded and transferred to PROD scheme.

Following these simple though important steps you can prepare the environment and start the performance testing. Next time I`ll cover the stages of test execution and result analysis.

Performance testing is testing that is performed, to determine how fast some aspect of a system performs under a particular workload. The process of performance testing includes the following stages:

1. Development and agreement of specification
2. Script development
3. Test environment set-up
4. Test execution
5. Test result analysis

Today we’ll cover first two stages.

Development and agreement of specification

The full information about system to be tested is exceptionally important on the first stage. Thus try to learn as much as possible about the software and hardware used. Of course, do not forget about the client’s requirements.

• Software. The information about the technologies used to develop and support software will tell you a lot about the bottlenecks and possible issues that may arise during the test execution.
• Hardware. Pay your special attention to the way the software components are arranged on the physical servers. Learn the configuration and parameters of all servers; find out how the system load is balanced. What is more discover the changes made in the dynamic parameters of the system. Why should you do that? All the information that you get on this step will help you to accurately set up the monitoring process while the test launch and carefully plan the performance model.
• Client’s requirements. Performance testing is quite a diversified procedure and includes different types of testing. To develop the testing strategy you are to know what the client awaits: evaluation of the maximum number of users the system can stand; detection of system bottlenecks, or check of the application stability. Try to get statistic of the software usage this assist in creating a more realistic performance model.

As soon as you get this information, you are able to assess the workload on the project and plan the workflow. A great advantage of getting the information before the project start allows you to avoid excessive communication with client afterwards.

Script development

As soon as you chose the tool for the automation of the performance tests, the scripts writing takes a start. Performance tests that emulate work of real user and are created on the basis of the test cases. You can adopt the test cases used in functional testing or you can write the original ones. There are several requirements to test cases:

• Iterations possibility
• Executed simultaneously in two or more streams
• Should not influence each other
• Compatibility with the dynamic parameters

When you have the appropriate test cases you can develop automated test. Then they are combined into the scenarios and groups depending upon the planned test load.

The tests are to include maximum of the system parameters. It is also recommended to create them in the way they could use data from the common source and exchange the information about the current situation. This allows managing the all running transactions.

Unlike automated functional tests, performance tests should return the system to the initial state, as in this case they don’t emulate the work of the real user.

Remember that some of the tests track the system state before and after the test launch. They fill the system with the necessary data and delete the non-valid and non-relevant data. These scripts don’t participate in the launch of the performance tests.