Leap into action: Building fair and transparent finserv applications

The “Open Banking” market is booming and expected to create £7.2 billion in revenue opportunities by 2022. It’s popular because it ultimately creates greater financial transparency for account holders through the use of open APIs.

Headquartered in Germany, finleap connect built the leading European fintech ecosystem, enabling companies in banking, accounting, and lending to provide next-gen, mobile-first financial services to their customers. They attribute their success to providing a world-class customer experience and offering a suite of connective products that span the whole fintech value chain.

Open banking is a very regulated, very strict, very security-heavy, and very fast-growing market. In order to keep pace with the competition, finleap recognized they needed to build a cloud-native architecture designed with scale and compliance in mind. Four years ago, the company embarked on an initiative to transform their architecture from the application layer down to the persistent data layer — which is when they started using CockroachDB.

When Christian Hüning, finleap’s Director of Technology of Cloud & Switchkit, joined in 2018, the company was still in the process of building an MVP. They started to develop an open-banking API product that ran on a traditional set of virtual machines (VMs) and hardware nodes in a private bare-metal cloud. This quickly became problematic for several reasons: it was outdated, inflexible, and the company needed to scale, which this setup simply did not permit.

Furthermore, they wanted more control over how the software was rolled out, deployed and changed. So, like many other forward-thinking startups, finleap decided to move their API product from VMs to Kubernetes.

The initial project was to spin up a Kubernetes cluster and learn how to deploy and introduce GitOps, security mechanisms, availability, useability of deployments, Helm Charts, and Docker. They were on a tight 9 month timeline to roll out the first model using this new environment.

At the beginning they used MySQL as a backend datastore replicated to a secondary MySQL instance. Bottlenecks began happening right away:

1. They could only vertically scale MySQL, which created operational headaches for their team.

2. They were failing over to the secondary quite often (frequently in the middle of the night) which caused downtime, slowed performance, and upset customers.

3. In order to change the schema and data model, they had to take the database offline resulting in over an hour of downtime.

4. MySQL was not built to work with the distributed nature of Kubernetes

Finleap developers were eager to switch to PostgreSQL since many of them were familiar with the powerful relational database. However, PostgreSQL has similar limitations to MySQL: the need to shard manually, an active/passive setup (risk of downtime), additional downtime for maintenance, and requires implementing workarounds to ensure data consistency. All of these limitations were a burden on finleap’s teams and also greatly impacted the customer experience.

Most importantly, however, they needed to find a database that would work natively with their new distributed environment. Migrating a centralized database application (like MySQL, PostgreSQL, Oracle, SQL Server) to a distributed architecture is a difficult and complex task. You can’t just take a legacy RDBMS and wire it up to K8s.

Finleap’s search for a database that combined high availability, elastic horizontal scale, and strong data consistency with Kubernetes compatibility eventually led them to CockroachDB.

One thing that Christian and his team quickly noticed was that CockroachDB shares similarities with Kubernetes when it came to atomicity, scale, and survival. CockroachDB also manages clusters very similarly to how pods are orchestrated in Kubernetes. This is no coincidence: CockroachDB was architected to allow applications to manage workloads in Kubernetes, instead of alongside it. Christian summarizes the team’s experience as, “We deployed CockroachDB straight from the Helm Chart and it just worked.”

In 2019, finleap migrated its first customer to their new environment. The process went fairly smoothly and continued to get easier and easier after each customer migration. The developer team reported that they felt “immediate relief” after moving to CockroachDB because the simplicity and automation greatly improved their productivity.

For example, they now had the capability to roll out changes as a Helm Chart in a GitHub pipeline via a standard pull-request based mechanism. Once a PR in the pipeline gets approved, it’s rolled out onto the clusters. All of the developers can spin up — and spin down — clusters quickly because of CockroachDB’s built-in automation. They often run tests in the dev environment by creating a job that initializes the database, user, and database schema…And the app starts working.

“CockroachDB’s simplicity and automated features were big game changers for our team. And we were impressed by its ability to seamlessly work with other parts of our cloud-native stack.” - Christian Hüning, Director of Technology Cloud & Switchkit

After their successful migration to CockroachDB and Kubernetes, finleap’s customer base started to grow — and along with it, the need to support dozens of different environments. They deliver a standard SaaS offering but also have customers who want to customize their own stack. Regardless of which approach their customer chose, finleap wanted the ability to automate operations for everything: backups, rollbacks, feature updates, online schema changes, etc….All of which are possible with CockroachDB.

At the same time, finleap works in a heavily regulated space and so they are obligated to provide a secure, highly available environment. More specifically, they needed the ability to move applications around from one cloud computing environment (public and private) to another with minimal disruption – also known as cloud portability. This would ultimately benefit their customers since it would create a fault tolerant setup.

To address these challenges, they wanted to run CockroachDB across multiple cloud providers and across federated Kubernetes clusters. Fortunately, CockroachDB is cloud-neutral and can be deployed across multiple clouds and clusters while still functioning as a single logical database — so you don’t actually have to federate clusters. With this setup, finleap is able to run across private clouds, public clouds (AWS, GCP, Azure), OpenStack, VMware, and Equinix.

When you are running across multiple distributed environments, however, latency becomes a huge factor. Anything taking longer than 200 milliseconds to load is considered slow from an end user point of view. Finleap was able to minimize latency with CockroachDB’s multi-region capabilities to control where data resides at the table, or even row, level. So UK data is stored in the UK, German data in Germany, and so on. Without the need to make network hops to access data you get to enjoy very low write latencies — and, at the same time, satisfy data domiciling regulations like GDPR.

Finleap also needed a tool for adding observability, security, and reliability to their applications that could be inserted into the platform layer. They became early adopters of Linkerd, the now-popular open source network proxy tool / service mesh that makes running Kubernetes services easier and safer by providing runtime debugging, observability, reliability, and security — all without requiring any changes to your code. Linkerd also works with CockroachDB.

Finally, they wanted to have a uniform Kubernetes experience across different environments. They came across a project from SAP called Gardener that would deliver on that goal by providing an enterprise Kubernetes cluster manager with strong support from a vibrant open source community.

Now that they had a uniform Kubernetes layer and were building with cloud-native solutions, finleap was able to deploy their entire stack wherever they needed to. They are working with ~25 microservices that have dependencies on the underlying technologies that are part of the stack:

Instead of running one big CockroachDB cluster, they run 30 separate clusters — because the clusters require the separation, not because they have overly excessive usage. So if their customers want to transport layer security (TLS) and rotate their security certificates frequently (ex. every 60 days), they can do just that. Finleap injected CockroachDB into Linkerd to eliminate the maintenance overhead of certificate rotation — a huge time-savings made possible because CockroachDB allows you to rotate public key infrastructure (PKI) security certificates without restarting nodes.

They are also leveraging Buoyant Cloud, the cloud management product for Linkerd, so they don’t have to worry about management. It provides these metrics on their cluster performance:

This is just one example of how finleap is combining cloud-native tools in a very efficient way to solve real world challenges (frequent certificate rotation) and give their developers their time back.

Finleap’s devs had to learn some nuances behind working with a distributed database. For example it would sometimes require retries, so they had to make changes to the code base to eliminate these. Christian recommends introducing the developers early that will eventually work with the discs early on. While CockroachDB is 99% compatible with PostgreSQL it does not (yet) have triggers, stored procedures, or business logic.

He also recommends that you plan for integration time and take a look at what you might need to change. Finleap knew they would not be able to lift and shift their legacy app to CockroachDB and expect it to be performant: Adjustments would be required. Once finleap explored the migration process and understood the nuances ahead of time, they could move really quickly when restructuring the application. They recommend reading through CockroachDB Documentation and checking out Cockroach University.

“Efficiency is everything! From developing apps to running infrastructure to creating a front-end, we need to be able to execute as quickly and cheaply as possible. CockroachDB greatly reduces operational overhead for our team and makes us feel confident in our ability to grow to new regions in the future.” - Christian Hüning, Director of Technology Cloud & Switchkit

For their customers, one of the major benefits of leveraging finleap’s environment is that they are able to adhere to all the regulatory requirements while gaining control over their infrastructure. Their platform is a “pre-approved application” meaning that finleap has all the paperwork signed off by the authorities so you can just plug in and be ready to go — a very attractive offering in a heavily regulated market. Their services are available in 11 different European countries today, and they plan to expand in the future

The finleap team members are huge fans of the CNCF ecosystem and have adopted many of the community’s values. They remarked that the cloud-native community wants to help you, vs. a proprietary vendor being obligated to help. Overall, finleap believes that using cloud-native technologies helps companies build better, more transparent products.

They remarked that they’ve been impressed with CockroachDB’s continuous release cycle and the team’s openness to receiving feedback. Many of finleap’s suggestions to CockroachDB have been incorporated into the product. As they continue to expand to new countries, they are looking forward to scaling their application because, Christian says, “Multi-region is where CockroachDB really shines.”