How to successfully migrate a critical microservice into monorepo
TL;DR: Bazel monorepo is a Engineering-wide initiative at Coinbase. Our Payments is one of the first teams to migrate a critical Ruby on Rails service for a major EU/UK payment rail into monorepo. After solving for a variety of challenges, we successfully migrated our service.
Background
The Coinbase Payments team’s mission is to empower customers to move money in and out of the crypto economy with a delightful experience. We take into account several factors including but not limited to scalability, security, and backup payment processors when we architect payment systems.
At present, we have multiple microservices integrating with our payment partners. The code for these services lives in numerous small repositories. As Coinbase expands geographically, we support more payment methods which can often mean more microservices. However, most of our microservices follow established design patterns, which resulted in frequent code replication.
We are continually thinking about ways to improve our systems and tools with the following high-level goals in mind:
- Decrease the cost of making changes on payments projects.
- Ensure payment services get security updates and fixes promptly.
- Increase code reuse within payments services.
- Align with Engineering-wide initiatives.
Earlier this year, Coinbase Engineering decided to use a Bazel monorepo. The Payments team chose to leverage the security, reliability features, and world-class tooling our Developer Productivity team has built for the Bazel monorepo by migrating our microservices. Our first initiative was to migrate a critical Ruby on Rails service for a major EU/UK payment rail which processes ~2M payments worth $3.5B annually into monorepo.
Challenges
However, migrating an already-deployed payment system into monorepo must account for a variety of challenges such as:
- Already processing payments in real-time.
- Adhering to predefined SLO and SLA.
- New features and bug-fixes being added and deployed.
- Integrations with third-party services that we have no control over.
To overcome these challenges we began with defining a successful migration. The following criteria were defined:
- Our project should work with the new monorepo tools: Our service is a Ruby on Rails application, and we use standard Rails tools for our SDLC. However, within Monorepo we use Bazel to build and test. Thus, a lot of our configurations needed to be changed to be compatible with new tools.
- Our project CI should work from monorepo: The number of steps we have in our projects CI increased in monorepo. We needed to spend time understanding what these steps were. Monorepo also used a different CI system. Our migration project involved making changes to successfully adapt to the new CI system and add configuration and code to pass these additional steps for the migrated service.
- Our service deployments should work from monorepo: Monorepo uses a different deployment tool so we needed to ensure our project can be deployed successfully in development, test, and production environments using the new tools.
- Our service connects to the same AWS resources as before: Depending on the payment network that moves money from a payer to a payee, payments go through several states. Some of these states are intermediate states, and we could not afford to lose such information before a payment reaches its terminal state. We also did not have the luxury of planning a data migration as a part of this project. Hence, it became vital that we have the right tools to deploy our service from monorepo, which can connect to the same cache and databases.
After identifying these critical requirements, we also maintained a git commit history across both repositories as a personal high standard. It took us about six weeks to ramp up on the new tools, make our project compatible with these tools, and migrate all the business logic.
The next step was to prepare for deployment and stop all commits to our old repository. Then we deployed the service to the development and test end-to-end payment flow. Once the team was satisfied and convinced that there is 100% parity in tooling and functionality, we proceeded to deploy the service to production. So the onboarding to using new tooling is smooth, we also trained new engineers to work on this service.
Conclusion
We managed to migrate our first payments microservice into monorepo, and today it is successfully processing the same volume of payments as before with no errors.
In the future, we plan to migrate more of our services into monorepo to solve our challenges around dependency management and code reuse.. We are also working closely with our Infrastructure team to fill any gaps and add new features before migrating more Ruby services. Follow us for more articles discussing in-depth components of our monorepo over the coming months.
If you are interested in solving complex technical challenges like this, Coinbase is hiring.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Payment services in Bazel monorepo was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.