Tech Talks vol.26 – Database Team

About Tech Talks

In this Tech Talks series, we will share with you the attitude and vibe of the PayPay Tech Team through the voices of the unique product members from around 50 countries!
In this issue, we talked to four engineers in the Database team.

Tell us about the team’s operational mission

Ninomiya:
The Database team comprehensively manages the databases used by PayPay. Currently we are running MySQL, TiDB, and Elasticsearch on AWS platform. Using these, we are working daily on how to efficiently create databases that meet the security and compliance specifications that financial services are expected to follow, in addition to performance and stability. The PayPay database handles a variety of data necessary to provide PayPay services, and if we look at that in terms of servers, it means about 2,000 or more databases are being managed by just a few members.

Kohei:
There are several important factors that come into play when managing databases, and one of them is stability. Since the platform, including the database, is the base for all services, it is fundamental to maintain healthy operation.
At the same time, it is also important to improve performance. For example, when we have a target for how many transactions can be handled in one second, taking a database-centered approach can yield dramatic improvements. With a little fine-tuning, we were able to complete a task that used to take six hours in three minutes.

Amogh:
Cost efficiency is another area where a database engineer can demonstrate their prowess. Database costs can be huge, comprising around half of AWS costs. When we were launching the business, the focus was on speed, so it is possible to reduce costs by half with some tweaks. Now that the organization is growing and we have many resources on board, we have moved to the cost-cutting stage.

Kohei:
In fact, Oono managed to complete a task that led to significant cost savings, in just a month.

Oono:
That’s right. We came up with an economical way to backup data and carried out these improvements. Because it is a large-scale database, even a small idea can have a big impact. If it is possible to do the same thing but at a reduced cost, then whoever developed that mechanism has very high engineering skills. Everyone is conscious of keeping costs as low as possible, and I think that is one of the interesting aspects of being a database engineer.

What are your individual challenges and specific initiatives?

Kohei:
To support PayPay’s database, which runs 24 hours a day, 365 days a year, we are working on the standardization of various operations to reduce dependency on individuals. Although TiDB, MySQL, and Elasticsearch are all under the same mechanism of data stores, they are each completely different and require specialist knowledge. For this reason, we are accelerating knowledge sharing among our members so that everyone in the team can support all the databases.

Specifically, we are working to improve the quality of our output by documenting what problems occurred and how they were handled. We adopted a buddy system where this knowledge is shared between two team members, then presented to the entire team. This not only allows the two members to share personal knowledge, but we also work to improve the quality of the output itself by discussing the content and disseminating it to the entire team.
I would like to promote knowledge transfer within the team and create a system that can produce more effective results with fewer people.

Ninomiya:
I have many opportunities to work with TiDB, and work on its operation and improvements, upgrades, and investigations into new features.
I can also leverage my previous experience from when I worked in a backend team. I often support the database team on application architecture and details which are difficult to know just from the database side perspective. This is an area where the specialism is very much concentrated with a few individuals and so one big challenge is how to impart that knowledge throughout the team.

Oono:
I am good at Elasticsearch, but I am aware that my knowledge of TiDB is not yet sufficient. So, as a personal challenge, I would like to work on deepening my understanding of TiDB and sharing my knowledge of Elasticsearch with the team.

Amogh:
The Database team also supports the delivery of the application, so there’s a lot of manual work that happens on request. To reduce that time as much as possible, we want to automate similar requests, or create steps that the Application team can carry out independently. Then with the resources that will open up, I would like to focus on enhancing database performance, seeking out new databases and working on database related improvements.

What challenges have you faced at PayPay?

Kohei:
I think the introduction of TiDB was the biggest challenge for PayPay’s database. We introduced distributed databases to deal with a spike in users as our existing databases could not handle the load. At that time, there were not so many resources experienced in TiDB in Japan so there was few information available online in Japanese. To learn this from scratch and run it without any issue was both challenging and a huge milestone in my career.

Ninomiya:
At that time, PayPay was already providing its payment services to more than 42 million users, so migrating the core database to TiDB was quite challenging.

I got involved in its implementation from the backend team, and we did everything necessary to make sure PayPay would have the confidence to deploy it into the production environment. More specifically, this involved making changes to applications to accommodate TiDB, the gathering and verification of non-functional requirements and risk assessments in addition to risk assessments for any issues and proactively handling them. It was grueling uncovering all of the specifications that were dependent on MySQL functionalities, but the documentation we created during the course of this migration have proved to be very helpful.

I believe that this experience has allowed me to look at applications not only from a technical perspective, but also from a service and business perspective. This project also gave me the opportunity to work with the Database team.

Oono:
For me personally, the disaster recovery project was very challenging. This project was to make the system redundant by using multiple bases so that the service can continue even in the event of a large-scale failure.
It wasn’t very easy to do that with Elasticsearch, and I ended up using the latest features that had just been released. It was a lot of fumbling and a lot of hard work, but it was a great experience to build a system like no other.

Tell us about your team

Kohei:
As many of us in the team did not come from a database engineering background, we have a culture of discussing ideas while leveraging our own knowledge and perspectives before reaching a decision. We also follow up with each other. For example, when I receive an on-call request which I am not sure about, there is always no shortage of volunteers willing to reach out and help me when I ask.

Oono:
Everyone is so nice. Everybody reacts as soon as possible, and when someone is stuck, our members can sense that and proactively react.

Amogh:
This culture proliferates throughout PayPay—not just the Database team. Even when there is a problem, everyone is focused on finding a solution. There is no blame game. At the Database team, when some unforeseen issue occurs, all members congregate in the form of a Zoom meeting, but there is no panic and everyone discusses the problem calmly. Doing so ensures you are focused on the result which ensures the solution is quick.

What challenges do you want to take on at PayPay?

Kohei:
I want to be able to provide the right database for each application as a member of the Database team. Right now, we have three options to choose from, but we should be able to give other advice. The best case would be increasing time spent on database research and validation so that we could use a greater number of databases. I want to be able to drive PayPay’s growth from the database side, and become the kind of team where people think, “What a brilliant Database team!”

Oono:
I agree. I think it is important to increase the range of proposals that can be made. I want to be familiar with various databases so that I can always propose something suitable for the purpose, rather than trying to do something with existing knowledge.

Kohei:
As the number of databases increase, keeping up with the learning is also quite a task. But we can say that because we enjoy it.

Finally, any words for job seekers?

Amogh:
PayPay is a perfect environment for those who want to contribute to the company’s growth and take ownership of what they do, as their opinions are heard, and they have a lot of freedom to make decisions. A good idea will be implemented quickly, so it helps technical growth, and you gain experience rolling out services to some 57 million users.

Oono:
In a good sense, PayPay is a very hungry company, so I would recommend it to those who want to grow as a database engineer. Our services must be available 24 hours a day-provide flawless security, and run smoothly. As we have grown bigger and often venture into unexplored territory, we welcome those who are ready to take on challenges.

Ninomiya:
In the Database team, it’s not just a matter of managing massive databases, but you can also experience the fun of taking on challenges from a database perspective as the service is continually growing. So, if you are someone who likes to get fully stuck into the product, and not just the technical aspect, then PayPay is for you.

Kohei:
It’s very appealing to be able to manage data for a service used so widely on a domestic scale, and a very exciting environment if you want to take on challenges in your daily work on how to achieve your goal using the latest technology. In addition, no one takes a negative view; everyone works in collaboration as a team and if your idea is logical, it will be proactively incorporated, regardless of your career level. So long as you’re motivated and responsible, you can go far at PayPay!

*Recruitment status and employee affiliations are correct at the time of the interview.