For organizations that want to avoid risk the first step is often to safeguard their most important asset: data.
Data loss usually happens by accident (primarily human error and system failures) and not because of malicious intent or bad actors. Maybe an employee neglects to update their software, or an entire cloud region goes down, or there’s an unexpected influx of traffic that the company wasn’t prepared for – we’ve all heard of or have been part of these stories. Recently, data protection has become more complicated because businesses are running many applications in the cloud that employees are accessing on a daily basis. The more applications, the more risk.
In 2018, Nightfall was founded to help organizations prevent data loss and protect data across cloud applications. For example, if a company is using Slack, Nightfall offers a Slack integration that can identify sensitive data such as social security numbers or credit card numbers, then send alerts to the appropriate people letting them know that sensitive information is at risk.
There are two main components to Nightfall’s product offering:
integrations with popular cloud applications like GDrive, Slack, GitHub, Jira, Confluence
a cloud-agnostic platform that uses AI/machine learning to identify sensitive data.
The Nightfall Developer Platform is a set of APIs that allow you to build data protection features into any app and protect customer data across its lifecycle, starting at ingestion.
Many of their customers are generating terabytes of data, and those scan request volumes can fluctuate suddenly depending on their business operations (customers have the ability to upload any amount of data at any time and request scans). Given this need for tremendous scale while still ensuring an ideal customer experience, they turned to CockroachDB as a backend for their API gateway and interactive analytics dashboard.
In this presentation from our first-ever customer conference Nightfall took the stage to describe all their various CockroachDB use cases:
Consolidate the tech stack 
When Nightfall started building their product 4 years ago, they wanted to deliver an MVP quickly and they wanted it to be cloud-native. At the time, a majority of the tech they used was chosen by the company’s co-founders – tech that had worked well for them in the past.
They were running Cassandra, Postgres, RocksDB, and TimescaleDB for a couple different use cases. And this all worked well until their customer base started to grow. Not only was managing several different databases an inefficient use of time, but they needed a solution that could scale horizontally to accommodate more customers. They also wanted to consolidate their databases down to one and have a single source of truth for all of their data.
Around the same time, they started hiring more engineers to help support business growth now that they had found product-market-fit. They decided to migrate from Docker on EC2 to Kubernetes. Given the distributed nature of Kubernetes, they now had the ability to operate applications at scale.
To take full advantage of that, they needed a database system that could match the distributed scale Kubernetes could provide. They wanted a solution that…
Delivered high availability so that their customers’ data was always available
Scaled out to multiple regions for so their customers would have low latency access and they could meet global data regulations
Allowed for easy horizontal scale to keep up with their growing user base and could support unpredictable spikes in traffic
Lowered the learning curve for developers and provided SQL compatibility
They started to shop around for distributed databases, keeping in mind that they wanted a cloud-native, agnostic solution. After coming across CockroachDB, they were impressed with its ability to scale across multiple regions, which the team thought would help them expand business into new locations in the future while still meeting data locality requirements.
“We are looking to expand into EMEA in the near future and are aware of the complications that can arise when entering new territories. CockroachDB provides tremendous value when it comes to scaling your app across multiple geographies. It handles the complexity in the database layer, so we wouldn’t have to deal with it in the application layer.” - Dan Hertz, Platform Team Manager
How CockroachDB fits into supply chain stack
After evaluating CockroachDB, they realized how easy it was to and scale across multiple regions. Since they are in the security space, they handle a lot of sensitive data while trying to mitigate regulations. The ability to control where data resides is a huge benefit.
CockroachDB fit seamlessly into their software delivery supply chain model. They use a variety of AWS tools like Lambda, Glue, and SQS alongside their Kubernetes clusters. They dogfood their own developer platform with CircleCI testing to scan their own code for sensitive data like PII or API keys. They push this data to their container image registry and then automate their deploys with Terraform.
They use Kafka for streaming data and Kafka queues to help manage large influxes in traffic. They also use Temporal (for workflow management) on top of Kafka which allows them to more predictably conduct asynchronous operations so all their jobs get completed on time.
They are running 5 CockroachDB clusters in a single AWS region now which is in close proximity to their current customer base which is primarily in the US.
It won’t be long before Nightfall transitions into a multi-region setup so that they can accommodate a user base in EMEA. Then, their deployment will look something like this:
As you can see from the diagram, they scaled their application to two additional regions, but CockroachDB still functions as a single logical database.
CockroachDB for metadata storage & system of record
The Nightfall team is running two major use cases in production on CockroachDB. The first is using CockroachDB as a metadata storage layer for their API gateway (which they built themselves). They have a high volume of API requests coming into their system and they need to return quick responses to confirm questions such as… is the API key valid? Is this customer within their quota for the request they are trying to make?
They use CockroachDB because it can deliver performant reads AND writes. For example, Nightfall might have 100 requests coming in per second and they need to write request logs for each of them and read out quickly to see if the API key is valid. They also use CockroachDB to store key entities such as telemetry, usage tiers, rate limits, request logs, etc.
“We have very high bandwidth use cases that require the ability to deliver performant reads and writes. With CockroachDB, we’ve never had to worry about any bottlenecks and have been impressed with its ability to scale as our customers generate tremendous amounts of data.” - Evan Fuller, Engineer
Their second use case is an interactive analytics dashboard that uses CockroachDB as the system of record because it can support a high volume of data ingestion. The data generated from Nightfall’s integrations with Google Drive, Slack, and other cloud apps can spread out among many different locations. Because CockroachDB functions as a single logical database they have a centralized place to host information even when they scale out their application to multiple regions. And because CockroachDB delivers high availability, their customers can always view/interact with their data.
Like the API gateway, this is not just a read-only use case because of the interactivity element. Users can take action on what they see in the dashboard and respond to alerts. They pull analytics straight from CockroachDB and use change data capture (CDC) to write to S3. CDC ensures that they are providing the correct data for their customers.
Tips for using CockroachDB
The Nightfall team said they felt very comfortable with PostgreSQL, which was actually a huge selling point for CockroachDB. Their team could reuse their SQL knowledge and get started quickly.
The API use case was pretty straightforward given CockroachDB’s high read and write bandwidth. However, their analytics dashboard was a little more complicated to build. It’s important to remember that CockroachDB is a distributed database and has unique capabilities that traditional relational databases do not have.
The Nightfall team has a few tips they recommend you keep in mind:
If you are performing complex queries with a lot of JOINs, you can end up with hotspots and subpar performance. You can make better index choices to improve SLAs.
If you need to control where data is distributed to, make sure you set up primary keys properly and understand how follower reads work.
Remember that CockroachDB has ACID guarantees which can have an impact on code design.
Nightfall worked with the Cockroach Labs architects to help them develop a setup that worked best for their analytics use case. If you are migrating from PostgreSQL to CockroachDB, here’s some other tips to keep in mind.
