Last week, while running a workload consisting of 200 different queries, we noticed right away that a CPU imbalance was causing a performance issue. Looking at the first graph, below, you can see right away that one of the three CockroachDB nodes was operating at near 100% CPU. Not ideal.
At the time we installed that cluster, we utilized version 21.1.11 of CockroachDB, which was the most current. Now fast forward in time to today – version 22.1 will be released in days, with a new feature called Admission Control. From the docs for this new feature, one scenario where Admission Control can help is when
The node has high CPU usage, visible in the CPU percent graph …
Bingo! We decided to give it a go and see if Admission Control could solve our problem.
We performed a zero-downtime, rolling upgrade to v. 22.1.0 and then enabled Admission Control via three SQL commands, as shown here:
set cluster setting admission.kv.enabled = true;
set cluster setting admission.sql_kv_response.enabled = true;
set cluster setting admission.sql_sql_response.enabled = true;
The CPU percent graph in DB Console showed an immediate improvement. We let the workload run for several hours and took a screenshot of the current CPU utilization across the cluster.
CPU utilization on the three nodes now tracks to within about 3%, which is about as close as one could hope for. What else could we call this feature? “My New Super Power”, “Auto Pilot”, “Guardrails”? Amazing, in any case, and a great example of the type of thoughtful features the brilliant development team at Cockroach Labs builds into each new release.
I encourage you to try this yourself. Stay in the loop with all the 22.1 release features right here.
