Publication date: May 3, 2021
Description
Cockroach Labs has discovered a race condition, where a long running read request submitted during replica removal may be evaluated on the removed replica, returning an empty result. Common causes of replica removal are range merges and replica rebalancing in the cluster. When a replica is removed there is a small window, where the data is already deleted but a read request will still see it as a valid replica. In this case a read request will always return an empty result for the part of the query that relied on this range.
This is a rare scenario that can only be encountered if a number of conditions are met:
- A read request checks that the node is holding the lease and begins processing
- The node loses the lease before the read request is able to lock the replica for reading
- The read request checks the state of the replica and makes sure it is alive
- The new leaseholder decides to immediately remove the old leaseholder replica from the range, due to a rebalance or range merge.
- The removal is processed before the read requests starts reading the data from the replica
- The read request observes the replica in a post-deletion state returning an empty result set.
Due to the nature of this race condition, we believe that it is only likely for this to happen on an extremely overloaded node that is struggling to keep up with processing requests and experiences natural delays between each step in the processing of a read request.
This issue affects CockroachDB versions 20.1 and later.
Statement
This is resolved in CockroachDB by PR #64324, which fixes the race condition between read requests and replica removal.
The fix has been applied to maintenance releases of CockroachDB v20.1, v20.2, and v21.1.
This public issue is tracked by #64325.
Mitigation
Users of CockroachDB v20.1 or v20.2 are encouraged to upgrade to v20.1.16, v20.2.9, or a later version.
Impact
This issue may impact all read operations, while the window of opportunity to see is very limited it may have happened to without your knowledge. Because this issue affects reads, it may impact backups. Our support team has tools to detect and troubleshoot this condition. If you would like help to check this in your environment, please contact them.
Was this helpful?
