You need to send cash from your bank to a friend, who happens to use another bank. The money should take a fairly straight line journey between them, right? In reality, it’s anything but simple.
To get from here to there, those funds must travel through a maze of payment schemes like FPS and BACS and SWIFT, each with their own protocols and APIs, all while making sure their path adheres to the complex and ever-changing requirements of a heavily regulated financial market.
Enter Form3, the enterprise-grade managed payment processing platform. The major financial institutions using Form3 need to integrate with only a single API in order to process their payment instructions securely and reliably across the galaxy of payment schemes, while the platform fully manages operational and compliance concerns. Form3 platform engineers Rogger Fabbri and Mario Morgado came to RoachFest 2023 to tell the story of migrating from PostgresSQL to CockroachDB, plus how to navigate the complexities of a multi-cloud environment while maintaining security, performance, and reliability.
This post is an overview of the topics covered in the talk; to learn all the nitty-gritty, cloud-spanning, latency-destroying details, complete with architecture diagrams, be sure to watch the entire presentation.
Three clouds, one platform
Form3 operates across AWS, Azure, and Google Cloud. They use CockroachDB as a critical component of this multi-cloud platform across AWS, with active-to-active replication so the service can survive even a full cloud outage. What does their multi-cloud architecture look like?
Cluster diagram courtesy Form3 All of the platform’s microservices run in Golang. The stack also makes use of NATS JetStream for messaging, with Kong as API gateway between the application and the managed Kubernetes service from each cloud provider. Clusters are formed by Kubernetes clusters, and the tenants — including CockroachDB and other microservices — are all running on Kubernetes. Ultimately, CockroachDB forms a cluster that spans across all the three clouds.
The diagram above is simplified to show basic architecture, but Form3 operates multiple clusters. The team runs clusters for critical platform components and as well as clusters for non-critical operations components such as reporting and auditing. Connection and routing is configured through Form3’s two physical data centers, which both sit on Equinix. To make that connection work, there is private connectivity in place for each of the three clouds.
Challenges of a multi-cloud environment
Several challenges emerged while creating Form3’s multi-cloud CockroachDB clusters, but the team quickly found solutions for each.
Connectivity. Addressing pod-to-pod connectivity within the CockroachDB cluster was the most pressing challenge, with pods needing to talk to each other across all three clouds.
Security Measures. Encryption at rest and encryption in transit both needed to be considered, including certificate management and an external secrets operator. Backup. With an RPO of five minutes, efficient backup was essential. Running in a multi-cloud environment required considering things like volumes, since all of this data would be on the networking and so possibly affect other workloads. Cross cluster communication also needed to be taken into account; with active-to-active replication, simply doing a full cluster backup would be a serious strain on networking.
See the full video for a deeper dive into the solutions and best practices Form3 evolved for these problems.
Migrating from Postgres to CRDB
Constructing Form3’s multi-cloud platform required migrating several microservices from PostgreSQL, which they initially had been running on AWS RDS, to CockroachDB.
“This migration was a natural step in our multi-cloud journey,” Mario Morgado explained. CockroachDB fulfilled the essential requirements Form3 needed from its new database:
Cloud agnostic
The ability to sustain a full cloud outage with no performance impact
A transactional database that works on a cross-data center setup
Zero downtime upgrades of the database, with rolling updates
Minimal migration work for the teams.
“Migrating from PostgreSQL to CockroachDB is not quite a shift and lift operation,” Morgado continued. “Indices, data cardinality, and multi-version concurrent control were the major things we had to learn, and how to best use them to our advantage.” For example, since CockroachDB has the same tree-like index coverage as DB3 indexes on PostgreSQL, they were able to plan queries in order to use fewer indices. Overall, though, the migration was a “fairly easy process” for the Form3 team. Since CockroachDB is PostgreSQL wire-compatible, Form3’s data structures and queries mostly just worked, with only a handful requiring a bit of tweaking. Also, since the Form3 platform is mostly written in Go., and since the majority of existing PostgreSQL tools also work with CockroachDB, the team could leverage the Go drivers they were already familiar with.
Key takeaways for future-proof multi-cloud operations
Rogger and Mario’s presentation covers much more, including: Schema shaping and performance tuning, implementing locality-restricted backups in multiple clouds for enhanced resiliency, and comprehensive observability and monitoring in a multi-cloud setup. Watch it on demand!
Form3’s journey exemplifies how to navigate the complexities of a multi-cloud environment while maintaining security, performance, and rock-solid reliability. The payment platform’s innovative approach to operating in a multi-cloud environment is a valuable guide for other organizations considering a move to multi-cloud… with CockroachDB playing a central part.
“CockroachDB is doing some amazing gymnastics under the bonnet,” Kevin Holditch, Form3’s head of platform engineering, said. “You might think it’s just a simple query, but it’s actually going off to different nodes where the data is physically stored to retrieve it. The performance is really good. And then you have this tremendous scaling capability. “Using CockroachDB almost feels a bit like magic.”
