Engineering

It’s been over six years since CockroachDB became a GitHub project. In that time, the project has racked up more than 20,000 GitHub stars, which is a simple way for GitHub users to bookmark repositories that interest them. Naturally, we’ve wondered how people find out about our project. Are there things we could do to accelerate awareness and interest?

Spencer Kimball

February 22, 2021

As long as you’re willing to follow the rules, deploying on Kubernetes and air travel can be quite pleasant. More often than not, things will “just work”. However, if one is interested in traveling with an alligator that must remain alive or scaling a database that must remain available, the situation is likely to become a bit more complicated. It may even be easier to build one’s own plane or database for that matter. Traveling with reptiles aside, scaling a highly available stateful system is no trivial task.

While not exactly envious of our current crop of interns (because, you know, the whole work from home thing), I’ll admit I find myself reminiscing back to when I was one myself. I’m still surprised the engineering team let me anywhere near the stuff they did. When I first interned four years ago, we had declared a just code yellow to focus our energy towards stabilizing CRDB. Having joined the newly-formed distributed query execution1 team, but now with its focus directed elsewhere, what this meant for me was free rein to flesh out distributed hash and merge joins2, few aggregation primitives (think SUM, COUNT, DISTINCT, etc.), and some sorting algorithms.

CockroachDB users trust us with their most sensitive data (see: healthcare, finance). And the best way for us to maintain that trust is for Cockroach Labs to never see this data at all. In CockroachDB v20.2, our tooling is able to automatically redact users' sensitive data out of log files, so that Cockroach Labs never even receives it. We also do this always for crash report telemetry.

The 2021 Cloud Report stands on benchmarks. Now in its third year, our report is more precise than ever, capturing an evaluation of Amazon Web Services (AWS), Microsoft Azure (Azure), and Google Cloud Platform (GCP) that tells realistic and universal performance stories on behalf of mission-critical OLTP applications.

Support for spatial data and spatial indexing is one of the most requested features in the history of CockroachDB. The first issue requesting spatial data in CockroachDB was opened in October 2017, and closed on November 12, 2020 with the release of spatial data storage in CockroachDB 20.2.

Last year the BulkIO team at Cockroach Labs replaced the implementation of our IMPORT bulk-loading feature with a simpler and faster data ingestion pipeline. In most of our tests, it looked like a major improvement: the release notes for CockroachDB v19.2 touted "4x faster" IMPORT. Many a 🎉 reaction was clicked, and the team moved on to new projects. But over the following months, it became clear we had celebrated too soon: we started to get reports of some IMPORTs that, instead of being faster, were much slower or even getting stuck. Armed with a test that could reproduce such a case, we started to dig.

This blog is written by guest authors Robin de Silva Jayasinghe, Thomas Pötzsch, and Joachim Mathes. Robin de Silva Jayasinghe, is a Sr. software engineer based in Germany working at synyx GmbH & Co. KG. synyx is an agile software provider in Karlsruhe, Germany that works together with different companies to find the best possible IT-solutions for their challenges. Thomas and Joachim are working as software and systems engineers at Contargo, one of the leading container hinterland logistics in Europe.

Since its inception, CockroachDB has relied on RocksDB as its key-value storage engine. The choice of RocksDB has served us well. RocksDB is battle tested, highly performant, and comes with a rich feature set. We’re big fans of RocksDB and we frequently sing its praises when asked why we didn’t choose another storage engine. Today we’re introducing Pebble.

Peter Mattis

September 15, 2020