Publication date: November 18, 2021
Description
During an upgrade of a CockroachDB cluster from v21.1.x → v21.2.0, backups will fail until the upgrade is finalized.
After the upgrade is complete and finalized, backups will continue as normal.
This issue will only occur if the upgrade coincides with a backup. For small clusters, where the upgrade is quick, there may be no overlap, and you will not experience this issue.
For larger clusters, or clusters with frequent backups, the likelihood of such an overlap increases. You should consider the mitigations below.
Newly created (non-upgrade) clusters are unaffected.
Statement
Cockroach Labs has diagnosed the above-described problem, and is tracking progress in GitHub issue #72839.
We plan to improve this issue in an expedited patch version v21.2.1 in the coming days. This advisory will be updated after that release.
You can track releases on our releases page.
Mitigation
If you plan to upgrade to v21.2, please wait until the v21.2.1 release or later.
If an upgrade is already in progress, we recommend either of the following:
- Rolling back the upgrade and waiting for v21.2.1.
- Finalizing the upgrade. See: Upgrade to CockroachDB v21.2 for more details. Note: You cannot roll back after finalization is complete.
If you have already completed the upgrade to v21.2, no further intervention is necessary. However, you may have skipped backups during that upgrade process, so you may wish to review your archives.
If you are running v21.1.x, and anticipate running backups during an upgrade to v21.2, we recommend upgrading to the v21.1.12 patch version (or later), prior to the major-version upgrade. Once on this version, we recommend the following setting:
SET CLUSTER SETTING bulkio.backup.experimental_21_2_mode.enabled = true;
This setting will reduce the likelihood of backups failing during a subsequent upgrade to v21.2.1.
To minimize likelihood of backup issues, we recommend against running in a mixed-major-version state for long periods of time.
Please review our upgrade documentation.
Impact
Backups may fail during the upgrade process from v21.1.x → v21.2.0.
For a large cluster, or a cluster with frequent backups, this upgrade time may overlap with backup execution. These backups are likely to fail.
For a small cluster, the upgrade time might not overlap with backup execution. In this case, backups will be unaffected.
Please reach out to the support team if you need more information or assistance.
Was this helpful?
