PayPay Data Engineer – Creating an Environment for Leveraging Data

This Professionals series showcases talented experts who support PayPay’s operations.
In this interview we talked with Takayuki Mieno, Senior Manager of the Data Management Department.

Data management is a hot topic these days. Let’s find out about the team at PayPay.

I am currently the head of the Data Management Department, which consists of the Data Infrastructure Team and the Anti Fraud Engineering Team.

The Data Infrastructure Team operates and maintains BigQuery (a data analysis platform), manages the company-wide data mart, and builds BI. The Anti Fraud Engineering team develops and maintains risk APIs, which block transactions regarding fraud prevention. The Data Science Team that will launched on November 1, 2022, will use machine learning and other technologies to improve operational efficiency (interview conducted in October 2022).

The Data Infrastructure Team is further divided into sub-teams called DWH, LAKE, BI, GCP, and Steward. The roles of each team is almost exactly what their names suggest, but the Steward Team may be unfamiliar to you, so let me explain. The Steward Team puts together sub-teams of the Data Infrastructure Team and connects them to external departments.

The Steward Team is a new team which was just formed.
Apart from the teams that maintain pipelines, write SQL queries, and manage the design and operations of GCP as data engineers, they are responsible for thinking about how data management should be across teams, understanding the status of data utilization in other departments, and sharing this information with each team.
The Steward Team has helped me handle projects a lot by interviewing other departments and joining kick-off meetings of projects which require data of these other departments.

Data infrastructure teams of other companies are probably in different situations. Some companies have one department that handles everything, while others have virtual teams across several departments. I am not sure if having an organization like the Steward Team is common practice, but if there is a necessary organizational structure, I would like to adopt it to promote better data utilization in the company.

Data Management and Projects

PayPay is a data-driven company where almost every department is involved in data analysis. That’s why, we tend to receive many smaller requests compared to other teams which develop systems, but I would like to take this opportunity to talk about two relatively large projects.

Utilizing vast amounts of data for business decisions

At PayPay, the department in charge issues reports for management quite frequently. It is important data that needs to be shared, but previously there was no system in place for cooperation among the business, system, and product members. This led to retrieving incorrect data and failed deliveries. Management pointed out that it was a problem if data that would be used as the basis for the company’s policy making was not being delivered correctly, so the Corporate Planning Department and the DaaS Team in the Product Division started the Data Governance Project (DGPJ).

The flow to deliver reports starts from the DaaS Team who links the data, which is processed by the Data Infrastructure Team in the middle, and the Corporate Planning Department creates the data mart and share as a PDF file via the Dataportal.

To ensure that this mechanism works, the tables to be used for reporting were identified, and the DaaS Team prioritized and monitored their performance. The Data Infrastructure Team established a process to detect delays and work with the Corporate Planning Department to ensure that reports are delivered in the morning.

However, putting the process in place was not easy. There were many challenges, such as fixing queries that refer to unmaintained tables and designing GCP infrastructure.
We simply communicated in depth many times with the Corporate Planning Department, which issues the reports. The queries were not too difficult to handle because there are staff in the Corporate Planning Department who can edit them, but we had to carefully check the details of GCP’s specifications with Google.
In any case, the data is directly related to management, so there is no room for mistakes. Really, the pursuit of accuracy was the hardest part of this project.

If you think about it, it is rare to find a company where a corporate planning department works so closely with a department that analyzes data.

Recently, we are finally able to deliver the reports in a fairly stable manner, so I am planning to improve on that and maintain the stability for future reports as well.

Leading projects involving the entire company by leveraging PayPay’s strengths

Another project we are currently working on in the Data Management Department is a project to build a company-wide data mart.

Originally, volunteers who are capable of writing SQL in operation teams created their own marts, which by the time we knew it was being used company-wide. But with personnel changes, they’re no longer maintained. So, we decided to take those data marts and build a new one to provide to the entire company.

PayPay folks are talented and eager to grow, and many of them study SQL on their own to acquire knowledge if the department needs it. Currently, there are tables being made by marketing, sales, and various other teams. That’s where data management comes in.
We are working steadily to ensure that data analysis throughout the company is sound and easy to use, such as by implementing common processing in the Data Infrastructure Team and writing cost-friendly queries.

But this project is also not as simple as just taking over from the operations teams. As mentioned above, data analysis was self-taught and done by people who knew what they were doing, so it was difficult to decipher and reconstruct each and every complex SQL or scheduled query where processing was written in a time-dependent manner. We contacted each department to inquire and understand how they were using the systems and to make connections to make it easier to proceed. It was also quite a patience-demanding process to grind through interviews with a dozen or so teams via Zoom.

In fact, while the original purpose of these meetings was to create a company-wide data mart, it was also an opportunity to collectively understand how data was being used and for what purpose; something we had not been able to have a grip on up to this point.We didn’t know how each department used the data, so we couldn’t decide policies, and it was a great opportunity, so we were like, “Let’s hear it from all departments!”

Thankfully, at PayPay, if you want to talk to another department, all you have to do is ask via Slack or email, and they will be able to talk to you right away on Zoom. This would not be the case if the system departments of a company only had a distant relationship to the business departments. So doing interviews may be time consuming, but there really is no obstacle between departments and divisions, so asking to have an interview or conducting a survey is not so hard.

As a result of asking around, we managed to gradually put the data mart in order and a support system is also being developed, such as a Slack channel where people can ask questions if they do not know how to write SQL.

Until Joining PayPay – The Ideal Match

I am not a natural-born engineer. Taking action enabled me to get to the current position.

Actually, I did not go into the IT industry after I graduated from college.
I majored in civil engineering throughout college and graduate school. I had some exposure to programming when I was in college, but I didn’t know much about it at the time and didn’t study it.
After graduation, I worked as a construction consultant, designing public buildings. At that time I was running a batch program (perl) to analyze the levee slope, and I was impressed to say the least. After that I read one technical book and found out how much fun programming was! I realized that I wanted to make a career out of it, so I decided to switch companies. I was already about 27 years old at the time, so I studied while looking for a job, and first joined a system integrator company after search for a place that would accept me even if I had no experience.
Also, my wife was living in Sapporo before we got married, and I had always wanted to move there, so it was a good timing for me to change my job.

Data analysis infrastructure that I just happened to come across

Before joining PayPay, I was a system developer at a system integrator company.
There, I was assigned to a team that would do data analysis and did data-related work. At that time people started saying, “Isn’t data management important?” so it was a good opportunity for me to gain experience in data infrastructure early on. I was then assigned to a project to develop a relatively large data analysis infrastructure.

While I was involved in that project, I started to feel that I want to build a data analysis infrastructure at an operating company. This is when I started my job search. In an operating company I would have more discretion, and when making decisions, it will be quicker with a different sense of speed. I would also be able to see where the data is actually being used, which makes it more rewarding.

So I looked for companies that were about to build a data analysis infrastructure and that seemed to have a large collection of data.

That’s how I came across PayPay. I did know about the PayPay app beforehand which means I knew what kind of application it was. I could immediately see that the company had a solid technological base. Above all, PayPay seemed to be handling a lot of data, you know? (laughs)

Also, one of the factors that encouraged me to apply for PayPay was the WFA system (fully remote work system). I have been working in Sapporo for a long time, and I can continue to work from Sapporo if I take advantage of the WFA system. Plus, the salary and benefits were perfect, so I definitely wanted to join the company since PayPay satisfied my three requirements: data volume, work style, and benefits.

The trick to WFA is simple

Obviously, WFA does away with commuting, so I can spend that time with my family, read books, and sleep. On the other hand, I no longer walk to work, and the only exercise I get is going up and down the stairs at home, so I take a walk during lunch hours.

Learn more about life at with WFA!

In terms of work, I place importance on pacing myself differently when communicating in ordinary meetings and in 1on1’s with members of my team. Meetings require logical conversations but while talking with members I want leeway for conversations to stray off from the subject, so we can share our views on other topics.

Calm and reliable in handling incidents

Compliment from a member who works with him

Actually, I tend to panic when I incidents occur (laughs).
In the past, I failed in incident handlings several times because I became too hasty.
After coming up with flows and steps for incident handling, I asked people to review my work or reconsidered the flow, only to find that I made mistakes in them.
It was a good reminder that the more you are in a hurry, the more dangerous it becomes to make assumptions about what might happen. It is very common to make assumptions and then make a move that turns out to be wrong. So I think it is important to check each assumption and build up the logic.

For everyday life, it might be okay to do things based on assumptions, but the system must be 100% logical to work correctly. I’m especially mindful about that.
But personality-wise, I tend to get panic and rush things with unexpected events. So I deliberately try to be calm and steady.

Diverse Members Looking for More Talent

Open to many possibilities.

The members of the Data Management Department come from diverse backgrounds. Some were custom software developers, some were data infrastructure developers, and some worked for analytical package software companies.
There is a tremendous amount of work to be done, so I think this is a department where people from various backgrounds can take an active role.
In terms of personalities, I would say that most of us are serene. It’s probably coincidence, though (laugh).

Spontaneity is especially important.

Data engineering skills are important, but you must also have the ability to take ownership of the tasks you are dealing with. If you are able to think and act on what needs to be done for things and issues that no one has yet acted on, it will be very rewarding and interesting.

If there are issues all over the place, we need to be proactive in reaching out to other departments. We can’t improve anything if we only sit and wait. You have to think of the big picture yourself and say, “Shouldn’t it be like this?” I often suggest ideas and ask others to green-light it.
It’s simply better to be willing to take on tasks that are static and that no one else is working on. The key is to be always spontaneous, not passive.

There are many things that have not yet been decided in PayPay, so you can go ahead and assign them to yourself and decide the policy on your own. If you can enjoy that, that would be a plus.

We want to keep introducing new things. Outlook and goals.

In the future, we will handle even more data; not only PayPay’s data, but also of group companies’. In fact, we already started sharing data amongst ourselves.
We will also be creating marts for machine learning in the future. We plan to adopt new mechanisms such as dbt and Analytics Hub as we go along.

As for future goals, I just want to make PayPay’s data analysis infrastructure even better! Although I have not set clear goals for myself, I am working to gain as much knowledge and experience as possible to create good software as a software engineer.
Not just in terms of technology, but in communication skills, management, and any other skill set!

For those interested in becoming a database engineer at PayPay

I think data engineering is on the rise right now, and I believe it will become even more popular in the future. If you want to build a career as a data engineer or further your career, we look forward to hearing from you!

*The recruitment status is current at the time of the interview.

RELATED ARTICLES

WFA × Life in Hokkaido

Thanks to: Takayuki Mieno / Author: Keiko（PayPay Inside-Out Editorial Team）/ Design: Mina / Translation: Language Communication Team / Translation Editor: Justin
*All employee affiliations are as of the time of this interview.