Publication date: May 20, 2020
Description
Range deletions (as caused by certain table drops, table truncates, and certain rebalancing operations during node decommissioning or failover), when occurring concurrently with replica garbage collection, can yield permanent data corruption or loss in rare cases.
Statement
A defect in RocksDB allowed it to violate its guarantees for snapshots, causing corruption under specific and extremely rare circumstances. This corruption is irrecoverable using regular node failover and must be addressed by restoring a backup when it occurs.
This bug has been fixed in versions v19.1.9, v19.2.7, and v20.1.0. All users are encouraged to upgrade (see Mitigation below).
The public issue is tracked as #48860.
Mitigation
There is no mitigation possible without an upgrade. Cockroach Labs recommends that customers upgrade to the latest release of v19.1, v19.2, or v20.1, all of which are currently under Maintenance Support.
Because CockroachDB v2.1 will reach End of Life on July 1, 2020, no new maintenance release for v2.1 will be issued containing a fix for this bug. Affected deployments running v2.1 should be upgraded to at least the latest release of v19.1.
Impact
All CockroachDB v2.x and v19.x releases up to and including v19.1.8 and v19.2.6 are affected. When encountered, the bug causes data corruption or loss.
Questions about any technical alert can be directed to our support team.
Was this helpful?
