View all

In August, we published a blog post entitled “Why Can’t I Run a 100-Node CockroachDB Cluster?”. The post outlined difficulties we encountered stabilizing CockroachDB. CockroachDB stability (or the lack of) had become significant enough that we designated it a “code yellow” issue, a concept borrowed from Google that means a problem is so pressing that it merits promotion to a primary concern of the company. For us, the code yellow was more than warranted; a database program isn’t worth the bytes to store its binary if it lacks stability. In this post, I’ll set the stage with some background, then cover hypotheses for root causes of instability, our communication strategy, some interesting technical details, outcomes for stabilization efforts, and conclusions. It’s a long post, so bear with me!

Spencer Kimball

November 16, 2016

In this blog post, we provide some details on how CockroachDB uses system memory on each node, and what you can do to keep memory usage in CockroachDB under control.

Some months ago I started work on a way to test random SQL statements with CockroachDB. This is important to expose unintended behavior in our server. For example, we want to prevent valid SQL statements from unexpectedly crashing a server or using all of its CPU or memory. We have already performed some small-scale fuzz testing, but fuzz testing often produces un-parseable input since it modifies bytes (although some fuzzers like AFL do attempt to produce clean input). The goal here was to produce valid SQL statements that the parser would accept and the system would then execute. These statements would essentially attempt to try various combinations of valid SQL to panic or otherwise render the system unusable (like consuming all CPU).

CockroachDB is a distributed SQL database built on top of a transactional key value store. We don’t (yet) expose the kv layer but it’s general purpose enough that we’ve used it to implement SQL without any special trickery. The particulars of how we represent data in a SQL table as well as the table metadata are internally called the “format version”. Our first format version was deliberately simple, causing some performance inefficiencies. We recently improved performance with a technique called column families, which pack multiple columns in one kv entry.

Scaling effortlessly over multiple nodes is one of the defining properties of CockroachDB. By maintaining a strongly-consistent database state across a network of machines, the distributed system can provide reliability and availability, while transparently tolerating disk, machine, and even datacenter failures. But not everyone has access to a datacenter at their fingertips, so we recently began looking into what it would take to run CockroachDB on a Raspberry Pi, one of the go-to tools of the modern day computer tinkerer.

CockroachDB is designed to be a scalable, survivable, and strongly consistent SQL database. Building a distributed system with these capabilities is a big task. Beyond the required functionality, it must also be correct, performant, and stable, or it isn’t worth the bits used to copy the binary.

Spencer Kimball

August 25, 2016

Until recently, CockroachDB’s SQL was suffering from a serious, long-standing bug – a schrödinbug, in fact – in its handling of table and column references. This blog post outlines how fuzz testing uncovered the error, how we discovered that our way of using Go was partly to blame, and how we addressed the issue using a form of strong typing.

As CockroachDB has grown over the last 30+ months of development, avenues for communicating between users and developers have proliferated. We started with GitHub, then created a Google Group, then an info@ email, then another Google Group, a Gitter room, and on we went. And while getting in touch with our developers is easier than ever (I suppose we don’t have Snapchat yet…), we hadn’t made a concerted effort to centralize our community’s brainpower – but that’s changing! Today we are launching the CockroachDB Community Forum, a place to ask how-tos, query others about best practices, and receive dev notices like our recent upgrade to Go 1.7.

Jessica Edwards

August 10, 2016

CockroachDB’s JOIN: An Early Implementation When our VP of engineering, Peter Mattis, made the decision in 2015 to support SQL, little did he know that the team would get as far as shipping the first implementation of CockroachDB’s JOIN exactly one year after that. A celebration is in order!