Tech Talks vol.12 – DaaS

About Tech Talks

In this Tech Talks series, we will share with you the attitude and vibe of the PayPay Tech Team through the voices of the unique product members from over 35 countries! This time, Zhang Ziliang from the DaaS Team will share his story.

*The Japanese version of the article is also available here.

What I’m working on currently

I’m a data engineer in the DaaS Team. Our team is responsible for building the data lake and near real-time pipeline. We ingest data from a variety of different data sources (Aurora, TiDB, AWS S3, etc.), load it into the Hudi data lake via batch or streaming pipelines, and then provide high-performance, high-availability data query interfaces for downstream services. We maintain a set of infrastructures on AWS by ourselves to meet more refined security and access control requirements.

A technical challenge I experienced recently

One of the problems we faced and solved recently is to use the idea of distributed computing to solve the problem of performance peaks in services. One of our backend development teams recently ran into an issue where they needed to upload over 40,000 files to a target server in a short period of time—once a day—and the number of files may increase over time. How to elastically complete such a huge file transfer task without wasting computing power was a problem. Even if the concurrency capability of a server is exhausted, its processing power is insufficient to support such a demand. If multiple servers are used for uploading at the same time, it’s difficult to distribute the upload tasks, track their status, deal with failed tasks, and dynamically adjust the number of required servers.

As I mentioned, our team has been working on distributed data processing, and such tasks are what we are good at. We transformed the original problem into a problem of executing a map function on a Spark DataFrame. We regard the correspondence between the source path and the target path of all the files to be uploaded as a Spark DataFrame, and we regard the upload operation of each file as a map function to be called against the DataFrame. With the help of data processing engines such as Spark, it’s no longer difficult to distribute tasks, track tasks, and retry failed tasks. Due to the auto-scaling strategy of our Spark cluster, it’s also easy to elastically increase or decrease the size of the cluster.

From such cases, we found that the experience of distributed data processing is helpful for other development teams to deal with some problems. So, we launched the Reconciliation as a Service (RaaS) project, which shields the complexity of distributed computing so that other development teams can easily solve many data processing scenarios with large amounts of data using only one Spark SQL statement containing business logic. Up to now, more than 100 tasks have used our RaaS services in the production environment, and the value of this project has also been affirmed by all other teams.

What I like to challenge at PayPay

After our data lake and data pipeline are stable, the next task will be how to use the data, which requires knowledge of data analysis and machine learning. I will gradually learn this knowledge in order to adapt to new projects. At the same time, I will focus on improving my project management capabilities, hoping to promote and lead new projects.

My typical daily & weekly schedules

I usually check and reply to emails in the morning, check system alerts, and complete tasks that require manual system operations. Sometimes there aren’t many emails or system alarms that need to be processed in the morning, so then I spend some time reading technical documents, whether they be internal or external to the company. As our team’s technology involves a wide range of areas, from infrastructure, frontend, backend, to distributed computing technology, and the programming languages we use are Scala, Kotlin, TypeScript, Python, etc. Reading such documents is necessary. It not only helps me understand the latest technical content, but also allows me to understand our technical architecture and details.

I have lunch at noon and take a walk. In the afternoon, there are usually some meetings inside and outside the team. After these meetings is usually my development time. I will spend a few hours on the current development task to promote the stories in the current sprint. Sometimes during the week, I also have interview tasks, such as screening resumes, checking code answers submitted by candidates, or an online technical interview.

I live in Tokyo and my favorite place to go on weekends is Yoyogi Park. I spend a few hours walking there and then stroll around. Sometimes I also play board games with friends on weekends. “The Resistance: Avalon” is one of my favorites.

My career before joining PayPay

Before 2018, I worked as a backend developer in China. Starting in October 2018, I moved to Tokyo and studied for a master’s degree in information science and technology at the University of Tokyo. Then I joined PayPay after graduation.

The reason I decided to join PayPay

What attracted me most about PayPay was the industry prospects, as well as the openness and diversity within the company. As the latest generation of payment methods, electronic payment not only provides you with incomparable convenience, but also the privacy and security of payment. Faster and more convenient payment methods will also stimulate emerging industries, such as bike sharing and food delivery. As a big fan of electronic payments, I undoubtedly have a strong interest in this industry. PayPay is based on electronic payments and is positioned as a super app. A promising industry, coupled with the company’s assertiveness, made me decide to join PayPay.

The openness and diversity of PayPay was also a big factor for me. The employees come from more than 40 countries. Japanese and English are both official languages, which makes me feel at ease as a foreigner. The WFA (Work From Anywhere At Anytime) system is even more exciting, especially during the epidemic. PayPay employees are not only living in Tokyo, but all over Japan. You can choose any city you want to live in, instead of having to endure several hours of public transportation every day for commuting. The WFA system is really convenient for me.

A message to aspiring PayPay employees

As I mentioned in my reasons for joining PayPay, the company’s industry prospects, openness, and diversity are what appealed to me the most. The WFA system was also another reason I was drawn to the company. Our team’s technology covers a wide range, which is a challenge for us. It requires our members to be familiar with many aspects of computer technology and have the ability to quickly learn and adapt to new technologies. But such challenges also allow us to upskill faster, have a global perspective, and more comprehensively consider the choice of a certain technology. I find such challenges and exercises to be quite attractive.

Perhaps one of the most critical qualities required in PayPay is the willingness to communicate. Due to our multinational, multilingual, multicultural, multi-time zone, and multi-workplace environment, this keenness towards communication is crucial. No matter the time or place, actively exchanging opinions with each other is one of the most important traits for adapting to this work style.

*The recruitment status is current at the time of the interview.

Author: Zhang Ziliang / Editorial Supervisor: Mune / Managing Editor: Az
*Employees’ affiliations are as of the time of the interview.