Cut Over from a Primary Cluster to a Standby Cluster

Physical cluster replication is supported in CockroachDB self-hosted clusters. New in v23.2: Cutover in allows you to switch from the active primary cluster to the passive standby cluster that has ingested replicated data. When you complete the replication stream to initiate a cutover, the job stops replicating data from the primary, sets the standby to a point in time (in the past or future) where all ingested data is consistent, and then makes the standby virtual cluster ready to accept traffic. Cutback using a new PCR stream switches operations back to the original primary cluster (or a new cluster) after a cutover event. This page describes:

• Cutover from the primary cluster to the standby cluster.
• Cutback from the original standby cluster (after it was promoted during cutover) to the original primary cluster.
• Job management after a cutover or cutback.

​

Cutover

The cutover is a two-step process on the standby cluster:

1. Initiating the cutover.
2. Completing the cutover.

​

Before you begin

During PCR, jobs running on the primary cluster will replicate to the standby cluster. Before you cut over to the standby cluster, or cut back to the original primary cluster, consider how you will manage running (replicated) jobs between the clusters. Refer to Job management for instructions.

​

Step 1. Initiate the cutover

To initiate a cutover to the standby cluster, you can specify the point in time for the standby’s promotion. That is, the standby cluster’s live data at the point of cutover. Refer to the following sections for steps:

• LATEST: The most recent replicated timestamp.
• Point-in-time:
  • Past: A past timestamp within the .
  • Future: A future timestamp for planning a cutover.

​

Cut over to the most recent replicated time

To initiate a cutover to the most recent replicated timestamp, you can specify LATEST when you start the cutover. The latest replicated time may be behind the actual time if there is in the stream. Replication lag is the time between the most up-to-date replicated time and the actual time.

1. To view the current replication timestamp, use:
   SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;
   
   id |     name           | data_state  | service_mode | source_tenant_name |                                                  source_cluster_uri                                 | replication_job_id |       replicated_time        |         retained_time         | cutover_time
   -----+--------------------+-------------+--------------+--------------------+-----------------------------------------------------------------------------------------------------+--------------------+------------------------------+-------------------------------+---------------
   5 | application        | replicating | none         | application        | postgresql://user:redacted@host?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted | 911803003607220225 | 2023-10-26 17:36:52.27978+00 | 2023-10-26 14:36:52.279781+00 |         NULL
   
2. Run the following from the standby cluster’s SQL shell to start the cutover:
   ALTER VIRTUAL CLUSTER application COMPLETE REPLICATION TO LATEST;
   
   The cutover_time is the timestamp at which the replicated data is consistent. The cluster will revert any replicated data above this timestamp to ensure that the standby is consistent with the primary at that timestamp:
           cutover_time
   ----------------------------------
   1695922878030920020.0000000000
   (1 row)
   

​

Cut over to a point in time

You can control the point in time that the PCR stream will cut over to.

1. To select a in the past, use:
   SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;
   
   The retained_time response provides the earliest time to which you can cut over.
   id |        name        |     data_state     | service_mode | source_tenant_name |                                                     source_cluster_uri                                               | replication_job_id |        replicated_time        |         retained_time         | cutover_time
   ---+--------------------+--------------------+--------------+--------------------+----------------------------------------------------------------------------------------------------------------------+--------------------+-------------------------------+-------------------------------+---------------
   3  | application        | replicating        | none         | application        | postgresql://{user}:redacted@{hostname}:26257?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted    | 899090689449132033 | 2023-09-11 22:29:35.085548+00 | 2023-09-11 16:51:43.612846+00 |     NULL
   (1 row)
   
2. Specify a timestamp:
   ALTER VIRTUAL CLUSTER application COMPLETE REPLICATION TO SYSTEM TIME '-1h';
   
   Refer to for more information. Similarly, to cut over to a specific time in the future:
   ALTER VIRTUAL CLUSTER application COMPLETE REPLICATION TO SYSTEM TIME '+5h';
   
   A future cutover will proceed once the replicated data has reached the specified time.

To monitor for when the replication stream completes, use to find the replication stream’s replication_job_id, which you can pass to SHOW JOB WHEN COMPLETE job_id as the job_id. Refer to the SHOW JOBS page for and an .

​

Step 2. Complete the cutover

1. The completion of the replication is asynchronous; to monitor its progress use:
   SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;
   
     id |        name         |         data_state          | service_mode | source_tenant_name |                                                source_cluster_uri                                                   | replication_job_id |       replicated_time        |         retained_time         |          cutover_time
   -----+---------------------+-----------------------------+--------------+--------------------+---------------------------------------------------------------------------------------------------------------------+--------------------+------------------------------+-------------------------------+---------------------------------
     4  | application         | replication pending cutover | none         | application        | postgresql://{user}:{password}@{hostname}:26257?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted | 903895265809498113 | 2023-09-28 17:41:18.03092+00 | 2023-09-28 16:09:04.327473+00 | 1695922878030920020.0000000000
   (1 row)
   
   Refer to for the and of SHOW VIRTUAL CLUSTER... WITH REPLICATION STATUS fields.
2. Once complete, bring the standby’s virtual cluster online with:
   ALTER VIRTUAL CLUSTER application START SERVICE SHARED;
   
     id |        name         |     data_state     | service_mode
   -----+---------------------+--------------------+---------------
     1  | system              | ready              | shared
     2  | template            | ready              | none
     3  | application         | ready              | shared
   (3 rows)
   
3. To make the standby’s virtual cluster the default for connection strings, set the following :
   SET CLUSTER SETTING server.controller.default_target_cluster='application';
   

At this point, the primary and standby clusters are entirely independent. You will need to use your own network load balancers, DNS servers, or other network configuration to direct application traffic to the standby (now primary). To manage replicated jobs on the promoted standby, refer to Job management. To enable PCR again, from the new primary to the original primary (or a completely different cluster), refer to Cut back to the primary cluster.

​

Cut back to the primary cluster

After cutting over to the standby cluster, you may need to move back to the original primary cluster, or a completely different cluster. This process is manual and requires starting a new PCR stream. For example, if you had between a primary and standby cluster and then cut over to the standby, the workflow to cut back to the original primary cluster would be as follows:

• Original primary cluster = Cluster A
• Original standby cluster = Cluster B

1. Cluster B is now serving application traffic after the cutover.
2. Drop the application virtual cluster from the cluster A with .
3. Start a PCR stream that sends updates from cluster B to cluster A. Refer to .

At this point, Cluster A is once again the primary and Cluster B is once again the standby. The clusters are entirely independent. To direct application traffic to the primary (Cluster A), you will need to use your own network load balancers, DNS servers, or other network configuration to direct application traffic to Cluster A. To manage replicated jobs on the promoted standby, refer to Job management. To enable physical cluster replication again, from the primary to the standby (or a completely different cluster), refer to .

​

Job management

During a replication stream, jobs running on the primary cluster will replicate to the standby cluster. Once you have completed a cutover (or a cutback), refer to the following sections for details on resuming jobs on the promoted cluster.

​

Backup schedules

will pause after cutover on the promoted cluster. Take the following steps to resume jobs:

1. Verify that there are no other schedules running backups to the same , i.e., the schedule that was running on the original primary cluster.
2. Resume the backup schedule on the promoted cluster.

​

Changefeeds

will fail on the promoted cluster immediately after cutover to avoid two clusters running the same changefeed to one sink. We recommend that you recreate changefeeds on the promoted cluster. will continue on the promoted cluster. You will need to manage or the schedule on the promoted standby cluster to avoid two clusters running the same changefeed to one sink.

​

See also