The CoreApp & Growth Department, which develops new features and implements improvements at the heart of the PayPay app, is constantly creating new payment experiences for our 71 million users (as of September 2025). However, these achievements are not born from chance.
This article delves into “A/B testing,” a critical component of PayPay’s app design. We’ll walk you through our process, key considerations, and real-world results to show how the PayPay app continues to evolve.
CoreApp & Growth Team (Renate, Beau, Maki, Arysa)
CoreApp & Growth Department / Design Department, Payment Product Division, Product Group
Hello! We are designers and product managers on the CoreApp & Growth Design Team. We focus on shaping PayPay’s core experiences while driving growth through thoughtful design and experimentation. Our mission is to use structured experimentation and data‑informed design to improve key user journeys—without compromising overall usability and consistency. From onboarding through to core features, we use A/B testing to discover what really works—and make sure each improvement supports both user needs and business goals.
What is A/B Testing at PayPay?
A/B testing is a method where multiple designs or components are prepared and deployed to users to measure and analyze the differences in their responses, which then informs design improvements. At PayPay, however, we believe great design goes beyond aesthetics. It is about solving real problems, validating assumptions, and making informed decisions through measurable outcomes. Our primary focus is on its ability to contribute to solving users’ core problems.
This is why our process at PayPay doesn’t end with simply showing two versions and testing them. We begin by identifying the real challenges users face and clearly defining how we intend to solve them through design. To be specific, our UX researchers provide foundational insights that set the direction for improvement.
Next, designers and product managers collaborate closely to pinpoint user challenges and form hypotheses for improvement. To test these hypotheses, we work with data analysts to establish the A/B test’s conditions and metrics, such as how to target users, how to define the test group, and what margin of difference will signify success. Engineers are also involved from the outset to discuss technical feasibility and build the required infrastructure.
As you can see, cross-functional collaboration among professionals in various fields is indispensable for A/B testing at PayPay. We aren’t merely testing a design; we are gaining a deep understanding of the fundamental solutions that users are looking for.
Success Story: Changing How Options Are Shown More Than Doubled the Selection Rate of a Specific Payment Method
Let’s dive into a real‑world example of an A/B test at PayPay. This case doesn’t involve a major design overhaul; instead, it shows how subtle adjustments—such as fine‑tuning copy, restructuring a layout, or simplifying a button—can lead to a significant improvement in the user experience.
After installing PayPay, users add a payment method as part of the initial setup. On that addition screen, we improved the following points:
- Revised wording that was hard to understand from a user perspective
- Added explanations regarding the protection of personal information
- Emphasized the payment method that earns the most PayPay Point
- Set the payment method we want to recommend as the default and added a “Recommended” label
As a result of simplifying the options and layering in fine adjustments, the conversion rate for that payment method more than doubled. Because the content became simpler, we also saw improvements in preventing users from skipping the text and enabling them to start configuring their payment method more quickly.
For our users, being able to earn more points and choose a simple payment method without the need to charge in advance is a big benefit. Through design, we were able to support solving the core issues users truly care about.
Failure Case: Switching to Copy That Promotes Earning Points Did Not Improve Conversion Rates
While we have success stories like the one above, not all A/B tests lead to improvements. Some tests don’t perform as expected, and we believe it’s just as important to share these “failures.”
To encourage new credit card registrations, we hypothesized that promotional-style wording that highlights user benefits—such as “Get More 2x PayPay Points”—would increase conversions. Although the click-through rate did rise, few users actually converted. In contrast, the original wording that directly named the service—“PayPay Credit”—generated fewer clicks but more conversions.
We learned that for financial decisions, users are more persuaded by clear, direct descriptions of the service than by promotional language. However, these insights weren’t derived from the data alone; they emerged from our careful post-test reviews and honest reflection. We believe that even a failed experiment, if it’s built on a solid hypothesis and carried out with intention, can be the fastest path to the next success.
Post-A/B Test Reviews
After conducting an A/B test, we document its entire scope. Specifically, we record the problem we were trying to solve, the underlying hypothesis, the design patterns we considered, the actual results, and the key takeaways.
We conduct these reviews for all tests, but we are especially thorough when a test fails. We discuss as a team what didn’t work, why it missed the mark, and what we might try differently next time, documenting everything. These findings are shared with other teams to prevent repeating the same mistakes. We use failures as stepping stones, continuously refining and improving our processes.
At the same time, reviews can be time-consuming, so we have streamlined the process through systematization and dedicated tools. For hypothesis formation and test planning, we utilize an “A/B Testing Toolkit” that includes templates for the design team. Once results are available, they are documented on a dedicated “Result/Insight Card.” All learnings, from both successes and failures, are logged in a centralized “A/B Test Repository,” making it easy to reference past experiments.
This system enables us to make faster, more data-driven decisions. The review process doesn’t just improve the quality of our designs; it also fosters knowledge sharing throughout the organization.
A Message to Our Readers
Working at PayPay involves more than just designing screens or managing backlogs. If you are driven by the “why” behind the work, not just the “what,” and are as committed to outcomes as you are to your craft, we invite you to partner with us in shaping our product strategy. Let’s build hypotheses, run tests, share our findings, and create payment experiences that truly resonate with and impact our users!
Current job openings
*Job openings and employee affiliations are current as of the time of the interview.