Publication date: September 3, 2024
Description
In all versions of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3, changefeeds could emit events on the same row out of order in some cases, which violates changefeed ordering guarantees. This issue was caused by a bug in the sarama Kafka client library, a third-party library that CockroachDB uses to talk to Kafka clusters. The bug manifested when a workload had mutations to the same key in rapid succession, because it was possible for sarama to re-order the messages if it encountered a retryable Kafka error.
To resolve this issue, CockroachDB now uses a new version of the Kafka sink that uses franz-go, a different third-party Kafka client library that does not have this bug. The new Kafka sink can be enabled using the cluster setting changefeed.new_kafka_sink.enabled. The new Kafka sink was added in PR #126213.
Statement
This is resolved in CockroachDB by PR #126213, which replaces the Kafka client library sarama with franz-go.
The fix has been applied to maintenance releases of CockroachDB v23.2.10, v24.1.4, and v24.2.0-rc.1.
This public issue is tracked by issue #122372.
Mitigation
Users of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3 are encouraged to upgrade to v23.2.10, v24.1.4, v24.2.0, or a later version, and enable the cluster setting changefeed.new_kafka_sink.enabled:
SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true;
changefeed.new_kafka_sink.enabled will be enabled by default in versions v24.2.1, v24.3, and all later versions.
Impact
Changefeeds could emit events on the same row out of order in some cases, violating the changefeed ordering guarantee. Versions affected include CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3.
Questions about any technical alert can be directed to our support team.
Was this helpful?
