How to build an infinitely scalable document storage solution

Optimal Systems is a 30-year-old IT company that provides document management solutions to organizations across Europe. They serve customers who need to store huge amounts of documents: electronic records, invoices, contracts, videos, audio recordings, and more. All told, OPTIMAL SYSTEMS is storing billions of documents — they use more than a petabyte of storage – for customers in 30 different countries.

The company was founded in 1991 because the European Space Agency needed an archive for satellite photos. At the time, they didn’t have a way to store, display, or search for their images of space. The file sizes were too large. Fortunately, the team at OPTIMAL SYSTEMS came up with a solution, and the company was born.

As their business grew over time, so did their customers. OPTIMAL SYSTEMS weathered storms including the dot-com bubble, the 2008 economic crisis, and most recently COVID-19, growing from a small firm to a company with billions of documents to store.

But the company’s growth led them to a new kind of challenge: they had become too successful, and were close to maxing out their capacity to scale.

Wisely, instead of pushing the limits of their existing solution, OPTIMAL SYSTEMS made it a priority to modernize their infrastructure.

Given the massive shift to the cloud over the past few years, the OPTIMAL SYSTEMS team wanted to build a new product from scratch that would be super fast, super scalable, and cheap to operate in the cloud.

Ultimately, they built a cloud-native document storage system using Kubernetes, Elastic, and CockroachDB as their primary system of record. Here’s how it happened…

OPTIMAL SYSTEMS’ original product offering (which they still have today) is a document storage software package that customers host on-premises using their own databases. Because OPTIMAL SYSTEMS had limited control over their customers' infrastructure, they ran into some challenges. For example, many of their customers’ IT teams chose to use legacy systems such as SQL Server and Oracle.

While the software works with legacy databases, customers ran into performance issues when storing and searching for hundreds of millions of documents. Their legacy systems just weren’t built for this type of heavy transactional use. Additionally, SQL Server and Oracle are both difficult and expensive to scale. Around five years ago, the OPTIMAL SYSTEMS team realized they needed to modernize their infrastructure and offer a new solution for document storage. Their vision was something that could be “infinitely scalable.” That meant they needed to develop a cloud strategy.

OPTIMAL SYSTEMS CTO Nikola Milanovic did not want to lift and shift their 25-year-old on-premises software solution. Instead, he thought it would be better (and easier) to build a new cloud-native solution from scratch so they could have the freedom to build exactly what they wanted. The design criteria for the new product included the following:

• Postgres compatibility

• Open standards, no proprietary software

• Horizontal scale in the cloud

• Free of all Windows dependency

• High availability and fault tolerance

• Managed cloud database and/or built on containers

To meet these requirements, Nikola and his team had to create a backend that included a full text search engine and relational database. They selected Elasticsearch for their full text search and liked that it supported SQL. They either wanted a managed cloud database (to run AWS or GCP) or the ability to run their database in Docker containers and scale using Kubernetes.

Their search for a cloud database product that met these requirements led them to Google Spanner, which (after a brief evaluation) then led them to investigate CockroachDB. They ultimately chose CockroachDB because it delivers on all of their requirements:

• Speaks SQL

• Bulletproof resiliency

• High availability

• Failover protocols

• Compatibility with Docker and Kubernetes

CockroachDB, they saw, could truly provide infinite scalability in the cloud.

“When it came to selecting a database solution, our choice was pretty simple. CockroachDB 100% fits our vision for an infinitely scalable database in the cloud. It’s how a software system should look.” - Nikola Milanovic

Once they had decided on the technologies that would make up their backend, OPTIMAL SYSTEMS started building their infrastructure. Basing everything on Kubernetes made it extremely easy to scale and distribute. And because CockroachDB supports a hybrid deployment model, OPTIMAL SYSTEMS’ customers can determine where their workloads live instead of being tied to a single provider.

For example, OPTIMAL SYSTEMS can have CockroachDB in the cloud (a cheap, fault tolerant, managed solution) and they can have their artificial intelligence (AI) system on-premises since it’s expensive to operate in the cloud. Because their product is made up of microservices, they can completely distribute them. CockroachDB can run in Asia while the AI system is running in Europe, and these distributed elements still talk to each other.

Here’s how their new system looks:

The core service is the main service that is needed by the customers, and it is the main framework for the other peripheral services, as illustrated below:

Here’s how the platform works: when a document comes into OPTIMAL SYSTEMS’ system it is registered. This creates a transaction that includes metadata about the document, as well as an ID number. It’s extremely important that this data isn’t lost, so it is written to CockroachDB.

From there, the document can be sent to the optical character recognition (OCR) system, so that all text can be indexed and searched with Elasticsearch. Alternatively, the document can be processed by the AI system, which determines additional metadata about the document.

Nikola says that the OCR and AI systems can crash, and they do crash. This is why OPTIMAL SYSTEMS uses CockroachDB as a recovery point. Since all document information is safely stored in CockroachDB from the beginning, they can reconstruct any document after a crash with zero data loss.

“Every system will fail at some point, it is not the question if, but how to cope with failure. For us, data loss is not an option. Thanks to CockroachDB, we can recover the entire system. It’s our single source of truth – and that’s a huge benefit.” - Nikola Milanovic

This newly developed technology stack is called yuuvis® Momentum. It enables their customers to develop, deploy and operate powerful, domain- and customer-specific information management applications on premise. It’s fast and helps them save money with little effort.

To better understand the impact of OPTIMAL SYSTEMS’ new solution, let’s look at two of their use cases.

Use Case #1: System of Record Archive

One of the customers using yuuvis® Momentum is the German National Library. Their mission was to archive all German publications dating back to 1945. That includes all print media (newspapers, magazines, books), video, audio – really anything that has a serial number.

They were looking for a future-proof system that could run in the cloud and on-premises, and had trouble finding one until they came across OPTIMAL SYSTEMS. yuuvis® Momentum by OPTIMAL SYSTEMS ended up being exactly what they were looking for. The hybrid deployment model that CockroachDB enables allows the German National Library to run the software however they like and scale their archive easily.

Use Case #2: Real-Time Video Surveillance Alerting

A few years ago, OPTIMAL SYSTEMS hosted a series of hackathons in the United States using yuuvis® Momentum. One of the teams had an idea to put surveillance videos into the system. Since yuuvis® Momentum can support compound documents, the videos could be sliced into intervals for AI to investigate offline. If the system detects something unusual, such as a burglar, it triggers an alarm, essentially making it a real-time surveillance alert system. CockroachDB is important to this system because of its ability to scale and deliver immediate access to the metadata.

Nikola manages a team of 85 engineers and the team came up with a way to learn more about this new system and test its limitations. After another company challenged itself to scale a relational system to store 11 billion documents, OPTIMAL SYSTEMS decided to take this challenge one step further. They’re building a relational system that’ll host 11 billion fully searchable documents. All of this is happening with CockroachDB, Elasticsearch, and is hosted in Azure.

In order to test the limits of the system and see how it scales, they’ve also been building measurement tools. Right now, they are at one billion documents and say the results look promising. As more documents are entered into the system, CockroachDB automatically rebalances the data and distributes it across different nodes.

They plan to finish the challenge by the end of this summer. If they are successful, they will break the industry record for a scalable, full text search database. (Stay tuned for more information!)

Right now, OPTIMAL SYSTEMS caters to customers in 30 different countries, primarily in Europe. However, in 2020 they joined the Kyocera Group, which is headquartered in Japan and has locations across the globe. They foresee a future where they have the opportunity to expand their business to new regions in Asia and North America.

Nikola and his team are happy that they built their “infinitely scalable” solution on CockroachDB. With a truly global, infinitely scalable database, the possibilities are endless!