Portfolio
Complex testing of a flagship B2B marketplace
a1qa was contacted to help release SAP Hybris-based software and continuously ensure its failsafe operation.
Overview
The client is a prominent provider of consumer and commercial automotive services, specializing in paint, collision, glass, vehicle repair, oil changes, maintenance, and car washes. Their flagship IT product is a B2B marketplace built on the SAP Hybris platform, serving thousands of shops by offering high-quality products from top suppliers.
In the past, the client worked with a third-party QA vendor but was dissatisfied with the efficiency of their QA processes, which led to significant delays in launching the marketplace. This prompted them to search for a more QA partner that better suited their needs.
Services offered
User acceptance testing
Functional testing
Test automation
Performance testing
Project scope
a1qa’s team, led by a QA manager, joined the Scrum-based project. Its first goal was to help the client release their flagship marketplace as soon as possible.
To do this, the team first analyzed existing QA processes, suggested crucial improvements, and introduced new ISTQB-based approaches, namely:
- Established a system outlining what tests should be run on each environment
The client leverages 3 environments. A test environment, to which software engineers deploy new code, while a1qa tests it and performs demos of new features. A user acceptance testing (UAT) environment, to which the final build is delivered and can be used to perform demos for new vendors or to see how new features work. And a production environment itself.
To improve QA efficiency, a1qa recommended optimizing the use of test environments as follows:
- The test environment would focus on testing new features with full acceptance test coverage.
- The UAT environment would handle regression testing, with a code freeze implemented on the 9th sprint day to ensure minimal acceptance test coverage.
- The production environment would be used for smoke testing on the 15th sprint day, provided regression testing is successful.
This approach aimed to streamline the QA process and enhance the overall effectiveness of testing activities.
- Independently scheduled testing activities for each release
The QA manager implemented a detailed tracking system, outlining all functionalities to be delivered to each environment. This ensured clear communication and transparency among all QA specialists.
- Took over part of the client’s work
Previously, the client was responsible for approving the release of all new features. However, after a1qa demonstrated their reliability, the client entrusted them with validating nearly all new features. Business-critical features were either reviewed directly by the client or presented to them by a1qa during demos.
- Introduced a bug triage process
Initially held twice a week and later reduced to once a week, QA engineers conducted meetings with the client to analyze and revise issue priorities. These sessions helped identify potential risks, ensure all major and critical issues were addressed, and confirm that team members were aligned on priorities of defects.
- Set up a test documentation creation process
This ensured comprehensive test coverage, facilitated the development of well-structured test cases and scripts to minimize production issues, promoted collaboration among project stakeholders, streamlined knowledge sharing, and helped onboard new team members by familiarizing them with the specifics of the IT product.
- Introduced a universal description of defects
This approach ensured that all project members clearly understood the nature of each issue, enabling more accurate prioritization and reducing the risk of rework or missed deadlines due to miscommunication.
- Created a detailed knowledge base leveraged even by the client
This helped provide a more effective onboarding process, create a standardized approach to QA across the project, and solve arising process-related issues easier. It also served as a solid guarantee that all the expertise will remain even if key employees leave.
The second phase in rolling out the marketplace involved comprehensive UAT. a1qa’s QA engineers thoroughly tested the entire software solution using real-life user journeys to validate the previous QA vendor’s work and ensure all features aligned with requirements. After five months of rigorous testing and validation, the client approved the software for deployment, and the marketplace successfully went live.
After a successful release, a1qa performed the following activities:
QA engineers ran full-cycle new feature testing to support the quality of the latest functionality, regression testing to ensure code changes haven’t affected previously well-working software parts, smoke testing to verify core functionalities, and defect validation to confirm that all found defects were fixed by developers.
a1qa also ran compatibility testing using BrowserStack to provide correct software operation on diverse platforms and browsers. The team analyzed the most popular devices and compiled a compatibility matrix that included Safari, Google Chrome, Microsoft Edge and the latest models of iPhones, iPads, Samsung devices.
As the first step in the big data validation process, QA specialists verified business logic using an Excel document. They ensured that all goods were correctly categorized and aligned with predefined standards.
In addition, a1qa performed demos for the client’s financial team to help them understand the operation of newly developed functionality ― a system of refunds, allowing end users to automatically return goods and get money back.
Manual regression testing was a time-intensive process, requiring four QA engineers to dedicate four days per cycle. To streamline this, a1qa introduced Java-based test automation to cover regression, smoke, and build verification suite testing.
The team utilized an in-house AI-driven test automation framework designed for simplified, safer, and more stable interactions with web browsers. It has conditional waits, predictable element interactions, parallel execution support, and easy configuration.
To enhance test execution and reporting, a1qa implemented Jenkins CI for remote test runs and integrated it with Jira and Allure Report. This setup allowed the team to automatically generate specific test suites after checking each build and deliver detailed, visualized results, significantly improving efficiency and accuracy.
Test automation introduction halved regression testing time.
QA automation engineers supported the optimization of the search bar, enabling it to find goods by ID, item name, and category. The client didn’t have a complete vision of how it should operate. So, a1qa automated the search of diverse goods and provided reports to help client’s developers analyze results and improve the search.
In the second phase of big data validation, the team developed Python-based scripts to identify and resolve data anomalies impeding correct categorization.
The client wanted to understand how smoothly the marketplace would operate under specified acceptance criteria ― 250 concurrently operating users. a1qa’s engineers designed and conducted server-side stress and load testing, along with client-side testing of the entire product. These efforts uncovered heavy database queries that were slowing down the software operation. After the developers resolved the issue, QA engineers retested the system to ensure the fix was effective.
During post-release support, the team ran tests if there were user stories that could affect overall performance in a sprint. In this case, QA engineers passed detected issues to developers for resolution and then retested the software.
- Support migration to Oracle ERP
The client leveraged software that interacted with the marketplace using .txt files. They included data about goods, vendors, pricing, discounts. The marketplace sent .txt files containing order data, such as customer, invoice number, sum. However, the client decided to migrate to Oracle ERP. a1qa performed system integration testing to ensure unhindered interaction and smooth data flow between the marketplace and Oracle ERP.
- Support SAP Hybris upgrade
The marketplace operated on a SAP Hybris platform. Approximately once a year, SAP provides massive updates while cutting support of previous versions. When this happened last time, QA engineers reran all available test cases to confirm failsafe software operation after switching to the latest SAP Hybris version. In addition, each 3 months QA engineers supported major SAP Hybris upgrades by analyzing which software parts could be affected and testing them.
In addition, a1qa provided prompt support to address and resolve issues as they arose – be it contribution to fixing production incidents or swift scaling of the team to handle increased testing load.
Technologies & tools
- Postman
- BrowserStack
- Intellij IDEA
- AWS
- Jenkins
- Allure Reports
- Figma
- Git
- Jira
- Azure Storage Explorer
- Excel
- SQL
- Groovy/ImpEx scripts
- REST Assured
- Cucumber JVM
- Allure Framework
- TestNG
- Docker
Results
-
Set up QA processes from the ground up to ensure effective software delivery and eliminate any probability of missed deadlines.
-
Released high-performance flagship marketplace that serves thousands of shops in just five months after a1qa joined the project.
-
Enhanced the quality of delivery processes and the marketplace itself due to thorough testing throughout the entire SDLC.
-
Sped up regression testing time by 50% and optimized smoke and build verification suite testing by introducing Java-based test automation.
-
Ensured interchangeability of team members due to creation of a solid knowledge base and knowledge transfer system.
In numbers
2
years of project duration
13
QA engineers during peak load
50%
speed-up in regression testing time