Tech Talks vol.47－The KAIZEN Project Undertaken by All QA Members

About Tech Talks

In the Tech Talks series, we’re excited to bring you the unique culture and perspective of the PayPay Tech Team through conversations with our product members, representing diverse backgrounds from approximately 50 countries around the world. In this edition, we delve into the “KAIZEN” project for regression testing, spearheaded by the whole QA team at PayPay to uphold the quality of our services.

Maximizing the Efficiency of Massive Regression Testing was a Top Priority

What was the project outline and mission?

Kelvin:
Our project revolved around reducing manual effort for regression testing. Regression testing (assessing whether system changes have inadvertently affected other areas) is crucial for our QA team. Our goal was to compress the usual three-day testing period into just one day through optimizing test cases, improving test methodologies, and automating test execution. Normally, the QA team operates across dedicated areas based on service domains; however, achieving such a challenging target requires the united effort of the entire QA team. The members of the QA Service team, which I belonged to, alongside regular contributors like Yusaku in the financial services QA team , Prasad from the payment QA Service team, Amit from the QA Tech team who specialize in automation, joined forces to form one cohesive unit.

Amit:
Increasing the automation rate of our tests from 70% to 90% was a key mission in enhancing QA operations. Achieving a 90% automation rate is ambitious, even by industry standards, and I initially questioned its feasibility. However, through the reciprocal collaboration among QA members and coordination with non-QA team members like PMs who develop services firsthand, we managed to resolve issue after issue progressively.

What challenges were the QA team facing?

Yusaku:
As PayPay’s range of functionalities expanded with features like digital salary payments and PayPay Card information display, the scope of regression testing also ballooned. Many of these cases required manual intervention, taking three days and averaging several members to manually test around thousands of cases. However, our budget, manpower, and resources are finite. Implementing as much automation as possible was necessary, along with revisiting the frequency and priority of tests for sections that are technically challenging. On the other hand, with responsibility for maintaining the quality of PayPay services used by 67 million users (as of December 2024), balancing efficiency with quality became a compelling challenge.

Supporting PayPay as the “Last Line of Defense” while Achieving Close to 90% Test Automation

Could you describe each of your roles in the project?

Prasad:
My role revolved around reducing regression testing effort through automation. Initially forming a working group with other QA members, I focused on understanding and classifying test contents. After sorting the technical challenges encountered and identifying automation opportunities, I analyzed industry examples and QA tools, ultimately establishing a framework based on the “Page Object Model” that offers high maintainability and readability to expedite testing.

Amit:
Much like Prasad, I was tasked with promoting automation. As the leader of the automation team, I was also in charge of formulating strategies and project management to meet the automation targets. We strived for continuous improvements in an agile manner while ensuring close coordination with development departments by conducting weekly and monthly meetings and retrospectives, thereby ensuring our quality assurance aligned with their user-first expectations.

Kelvin:
Alongside Yusaku, I validated the necessity of test cases and established guidelines that were used by the team members to perform test case optimization. The guidelines included a set of rules and conditions that help determine the test execution frequency and priority of each test case, and they were utilized by each team. In the end, we successfully reduced regression manual effort by 70%, which increased the speed of regression delivery.

Yusaku:
I focused on optimizing tests in the financial services and bill payment domains, which I routinely engage with. We reviewed accumulated test results and defect occurrences to re-evaluate test frequency and priorities based on the guidelines that Kelvin and I created. Specifically, we downgraded the priority of cases with low defect frequencies and identified redundant or irrelevant cases to regression purposes, thus reducing workload. Simultaneously, I worked on creating mechanisms to prevent overlooking any defects post-reduction and ensuring that service quality wouldn’t be impacted by the cutbacks in test cases.

What was the most challenging aspect of the project?

Yusaku:
The constant looming worry: Would issues truly not arise with the flows we constructed? Our QA team is akin to the “last fortress” in catching defects before a service hits the market. If we overlook defects, flawed services could reach users, putting a lot of pressure on us. Yet, armed with a vast pool of past test data, I convinced myself, “The flow was designed with a clear rationale, so it’s bound to be alright,” and stayed committed to aggressively reducing tests.

Kelvin:
I completely agree. Reflecting back, the new guidelines we initially drafted were rather conservative. Realizing the unmet reduction target, we shifted towards more aggressive cuts, such as introducing a new priority level that allows test execution to be skipped for test cases when there are no code changes related to the affected area in a given release. The mental challenge was real, but knowing that by committing to these changes, we could save time and resources, redirecting them toward more impactful tasks, helped me persevere. As a parallel initiative, we developed a new strategy and tools that boosted productivity, reducing manual effort and enabling faster execution. Unlike test case optimization, which might affect quality, this approach minimized risk by maintaining all test cases, ensuring consistent quality.

Prasad:
Encountering test cases that were difficult to automate was a substantial challenge. For instance, the core payment code scan in PayPay involved human scanning, inherently making automation challenging. However, driven by requests from the frontline to find a way to make it happen, we thoroughly analyzed the features and documentation of the automation testing tools we used, brainstormed extensively, and discovered a path to automation. Tackling such seemingly intractable technical riddles found no straightforward solutions online, but finally, we managed to simulate code payment on the app, and successfully automate it.

Amit:
The classification and understanding of test cases necessary for framework improvement were demanding. We spent the initial two to three weeks not touching on the improvements themselves but engrossed in understanding and categorizing each case. Continuously visiting development teams responsible for individual functions was exhausting, but the precise understanding gained ultimately facilitated smoother framework improvements.

Could you share the outcomes of the project and upcoming prospects?

Amit:
We succeeded in reducing the regression testing period to one day, which was one of our chief goals. Additionally, we achieved near 90% automation. The lack of major defects from flow changes also provided a sigh of relief. I believe the success was in establishing thorough inter-departmental coordination centered around the QA team, and that every project member shared a sense of urgency and pursued improvement with the same intensity. I aspire to leverage this experience to set up an automation framework for the PayPay Card in my regular duties.

Kelvin:
Productivity has improved by more than 40%, while the total duration for regression testing has been reduced by more than 60%, resulting in significant time savings and enhanced efficiency. However, as this project was confined to QA team improvements, I’m keen to embark on full-scale operational improvements involving the entire development division going forward. Given the anticipated need for more extensive discussions with many stakeholders than ever, keeping in mind the PayPay principle “Ego is not welcome, Communication is necessary” from the “PayPay 5 senses,” I’ll focus on achieving results through productive conversations.

Finally, could you share a message with the readers?

Kelvin:
Administering QA for PayPay, distinguished by its vast user base, and securing service quality is a point of pride for me. I believe PayPay is an optimal place for those who, when faced with challenging goals, constantly pursue new challenges and relish the process.

Amit:
PayPay has been a revolutionary service in Japan’s payment market. While technical proficiency is, without a doubt, advantageous, supporting a large-scale service necessitates flexible thinking, cooperation among team members, and learning from past experiences. I plan to apply the lessons from this project to build a robust automation system for PayPay Card.

Yusaku:
There were challenging periods, but discovering and implementing proactive solutions for these challenging issues was rewarding. It’s not only important to safeguard the user experience we, as QA managers, vow to protect but to also enjoy the evolving environment and workflows, our company is best suited for such individuals.

Prasad:
Scrutinizing the vast specifications of the utilized tools to devise automation approaches, and engaging in extensive dialogues on new solutions was technically enlightening. It was also invaluable to lead and present technological solutions for the challenges faced by various departments and showcase leadership. For those aspiring to grow as professionals, I encourage you to step through PayPay’s doors.

*Job openings and employee affiliations are current as of the time of the interview.