Configure Replication Zones

On this page

Replication zones give you the power to control what data goes where in your CockroachDB cluster. Specifically, they are used to control the number and location of replicas for data belonging to the following objects:

• Databases
• Tables
• Rows (enterprise-only)
• Indexes (enterprise-only)
• All data in the cluster, including internal system data (via the default replication zone)

For each of the above objects you can control:

• How many copies of each range to spread through the cluster.
• Which constraints are applied to which data, e.g., "table X's data can only be stored in the German availability zones".
• The maximum size of ranges (how big ranges get before they are split).
• How long old data is kept before being garbage collected.
• Where you would like the leaseholders for certain ranges to be located, e.g., "for ranges that are already constrained to have at least one replica in region=us-west, also try to put their leaseholders in region=us-west".

This page explains how replication zones work and how to use the CONFIGURE ZONE statement to manage them.

Overview

Every range in the cluster is part of a replication zone. Each range's zone configuration is taken into account as ranges are rebalanced across the cluster to ensure that any constraints are honored.

When a cluster starts, there are two categories of replication zone:

1. Pre-configured replication zones that apply to internal system data.
2. A single default replication zone that applies to the rest of the cluster.

You can adjust these pre-configured zones as well as add zones for individual databases, tables, rows, and secondary indexes as needed. Note that adding zones for rows and secondary indexes is enterprise-only.

For example, you might rely on the default zone to spread most of a cluster's data across all of your availability zones, but create a custom replication zone for a specific database to make sure its data is only stored in certain availability zones and/or geographies.

Replication zone levels

There are five replication zone levels for table data in a cluster, listed from least to most granular:

For system data

In addition, CockroachDB stores internal system data in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular:

Level priorities

When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data:

1. If there's a replication zone for the row, CockroachDB uses it.
2. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone.
3. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone.
4. If there's no applicable table replication zone, CockroachDB uses the database replication zone.
5. If there's no applicable database replication zone, CockroachDB uses the default cluster-wide replication zone.

Manage replication zones

Use the CONFIGURE ZONE statement to add, modify, reset, and remove replication zones.

Replication zone variables

Use the ALTER ... CONFIGURE ZONE statement to set a replication zone:

> ALTER TABLE t CONFIGURE ZONE USING range_min_bytes = 0, range_max_bytes = 90000, gc.ttlseconds = 89999, num_replicas = 5, constraints = '[-region=west]';

Replication constraints

The location of replicas, both when they are first added and when they are rebalanced to maintain cluster equilibrium, is based on the interplay between descriptive attributes assigned to nodes and constraints set in zone configurations.

Descriptive attributes assigned to nodes

When starting a node with the cockroach start command, you can assign the following types of descriptive attributes:

Types of constraints

The node-level and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance:

• When locality is the only consideration for replication, it's recommended to set locality on nodes without specifying any constraints in zone configurations. In the absence of constraints, CockroachDB attempts to spread replicas evenly across the cluster based on locality.
• Required and prohibited constraints are useful in special situations where, for example, data must or must not be stored in a specific country or on a specific type of machine.
• Avoid conflicting constraints. CockroachDB returns an error if you:
  • Redefine a required constraint key within the same constraints definition on all replicas. For example, constraints = '[+region=west, +region=east]' will result in an error.
  • Define a required and prohibited definition for the same key-value pair. For example, constraints = '[-region=west, +region=west]' will result in an error.

Scope of constraints

Constraints can be specified such that they apply to all replicas in a zone or such that different constraints apply to different replicas, meaning you can effectively pick the exact location of each replica.

Node/replica recommendations

See Cluster Topography recommendations for production deployments.

Troubleshooting zone constraint violations

To see if any of the data placement constraints defined in your replication zone configurations are being violated, use the system.replication_constraint_stats report as described in Replication Reports.

View replication zones

Use the SHOW ZONE CONFIGURATIONS statement to view details about existing replication zones.

You can also use the SHOW PARTITIONS statement to view the zone constraints on existing table partitions, or SHOW CREATE TABLE to view zone configurations for a table.

Basic examples

Setup

The following examples use MovR, a fictional vehicle-sharing application, to demonstrate CockroachDB SQL statements. For more information about the MovR example application and dataset, see MovR: A Global Vehicle-sharing App.

To follow along, run cockroach demo with the --geo-partitioned-replicas flag. This command opens an interactive SQL shell to a temporary, 9-node in-memory cluster with the Geo-Partitioned Replicas Topology applied to the movr database.

$ cockroach demo --geo-partitioned-replicas

These examples focus on the basic approach and syntax for working with zone configuration. For examples demonstrating how to use constraints, see Scenario-based examples.

For more examples, see CONFIGURE ZONE and SHOW ZONE CONFIGURATIONS.

View all replication zones

> SHOW ALL ZONE CONFIGURATIONS;

                       target                      |                               raw_config_sql
+--------------------------------------------------+-----------------------------------------------------------------------------+
  RANGE default                                    | ALTER RANGE default CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 90000,
                                                   |     num_replicas = 3,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  DATABASE system                                  | ALTER DATABASE system CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 90000,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  TABLE system.public.jobs                         | ALTER TABLE system.public.jobs CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 600,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  RANGE meta                                       | ALTER RANGE meta CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 3600,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  RANGE system                                     | ALTER RANGE system CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 90000,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  RANGE liveness                                   | ALTER RANGE liveness CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 600,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  TABLE system.public.replication_constraint_stats | ALTER TABLE system.public.replication_constraint_stats CONFIGURE ZONE USING
                                                   |     gc.ttlseconds = 600,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  TABLE system.public.replication_stats            | ALTER TABLE system.public.replication_stats CONFIGURE ZONE USING
                                                   |     gc.ttlseconds = 600,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
...

For more information, see SHOW ZONE CONFIGURATIONS.

View the default replication zone

> SHOW ZONE CONFIGURATION FOR RANGE default;

     target     |              raw_config_sql
+---------------+------------------------------------------+
  RANGE default | ALTER RANGE default CONFIGURE ZONE USING
                |     range_min_bytes = 134217728,
                |     range_max_bytes = 536870912,
                |     gc.ttlseconds = 90000,
                |     num_replicas = 3,
                |     constraints = '[]',
                |     lease_preferences = '[]'
(1 row)

For more information, see SHOW ZONE CONFIGURATIONS.

Edit the default replication zone

To edit the default replication zone, use the ALTER RANGE ... CONFIGURE ZONE statement to define the values you want to change (other values will remain the same):

> ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR RANGE default;

     target     |              raw_config_sql
+---------------+------------------------------------------+
  RANGE default | ALTER RANGE default CONFIGURE ZONE USING
                |     range_min_bytes = 134217728,
                |     range_max_bytes = 536870912,
                |     gc.ttlseconds = 100000,
                |     num_replicas = 5,
                |     constraints = '[]',
                |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a system range

In addition to the databases and tables that are visible via the SQL interface, CockroachDB stores internal data in what are called system ranges. CockroachDB comes with pre-configured replication zones for some of these ranges:

To control replication for one of the above sets of system ranges, use the ALTER RANGE ... CONFIGURE ZONE statement to define the relevant values (other values will be inherited from the parent zone):

> ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 7;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR RANGE meta;

    target   |            raw_config_sql
+------------+---------------------------------------+
  RANGE meta | ALTER RANGE meta CONFIGURE ZONE USING
             |     range_min_bytes = 134217728,
             |     range_max_bytes = 536870912,
             |     gc.ttlseconds = 3600,
             |     num_replicas = 7,
             |     constraints = '[]',
             |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a database

To control replication for a specific database, use the ALTER DATABASE ... CONFIGURE ZONE statement to define the relevant values (other values will be inherited from the parent zone):

> ALTER DATABASE movr CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR DATABASE movr;

     target     |              raw_config_sql
+---------------+------------------------------------------+
  DATABASE movr | ALTER DATABASE movr CONFIGURE ZONE USING
                |     range_min_bytes = 134217728,
                |     range_max_bytes = 536870912,
                |     gc.ttlseconds = 100000,
                |     num_replicas = 5,
                |     constraints = '[]',
                |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a table

To control replication for a specific table, use the ALTER TABLE ... CONFIGURE ZONE statement to define the relevant values (other values will be inherited from the parent zone):

> ALTER TABLE users CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR TABLE users;

    target    |             raw_config_sql
+-------------+----------------------------------------+
  TABLE users | ALTER TABLE users CONFIGURE ZONE USING
              |     range_min_bytes = 134217728,
              |     range_max_bytes = 536870912,
              |     gc.ttlseconds = 100000,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a secondary index

The secondary indexes on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes.

To control replication for a specific secondary index, use the ALTER INDEX ... CONFIGURE ZONE statement to define the relevant values (other values will be inherited from the parent zone).

> ALTER INDEX vehicles@vehicles_auto_index_fk_city_ref_users CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR INDEX vehicles@vehicles_auto_index_fk_city_ref_users;

                         target                        |                                 raw_config_sql
+------------------------------------------------------+---------------------------------------------------------------------------------+
  INDEX vehicles@vehicles_auto_index_fk_city_ref_users | ALTER INDEX vehicles@vehicles_auto_index_fk_city_ref_users CONFIGURE ZONE USING
                                                       |     range_min_bytes = 134217728,
                                                       |     range_max_bytes = 536870912,
                                                       |     gc.ttlseconds = 100000,
                                                       |     num_replicas = 5,
                                                       |     constraints = '[]',
                                                       |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a partition

Once partitions have been defined for a table or a secondary index, to control replication for a partition, use ALTER PARTITION <partition> OF INDEX <table@index> CONFIGURE ZONE:

> ALTER PARTITION us_west OF INDEX vehicles@primary
    CONFIGURE ZONE USING
      num_replicas = 5,
      constraints = '[+region=us-west1]';

CONFIGURE ZONE 1

> ALTER PARTITION us_west OF INDEX vehicles@vehicles_auto_index_fk_city_ref_users
    CONFIGURE ZONE USING
      num_replicas = 5,
      constraints = '[+region=us-west1]';

CONFIGURE ZONE 1

To define replication zones for identically named partitions of a table and its secondary indexes, you can use the <table>@* syntax to save several steps:

> ALTER PARTITION us_west OF INDEX vehicles@*
    CONFIGURE ZONE USING
      num_replicas = 5,
      constraints = '[+region=us-west1]';

To view the zone configuration for a partition, use SHOW ZONE CONFIGURATION FOR PARTITION <partition> OF INDEX <table@index>:

> SHOW ZONE CONFIGURATION FOR PARTITION us_west OF INDEX vehicles@primary;

                    target                    |                             raw_config_sql
+---------------------------------------------+------------------------------------------------------------------------+
  PARTITION us_west OF INDEX vehicles@primary | ALTER PARTITION us_west OF INDEX vehicles@primary CONFIGURE ZONE USING
                                              |     range_min_bytes = 134217728,
                                              |     range_max_bytes = 536870912,
                                              |     gc.ttlseconds = 90000,
                                              |     num_replicas = 5,
                                              |     constraints = '[+region=us-west1]',
                                              |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Reset a replication zone

> ALTER TABLE t CONFIGURE ZONE USING DEFAULT;

CONFIGURE ZONE 1

For more information, see CONFIGURE ZONE.

Remove a replication zone

> ALTER TABLE t CONFIGURE ZONE DISCARD;

CONFIGURE ZONE 1

For more information, see CONFIGURE ZONE.

Constrain leaseholders to specific availability zones

In addition to constraining replicas to specific availability zones, you may also specify preferences for where the range's leaseholders should be placed. This can result in increased performance in some scenarios.

The ALTER TABLE ... CONFIGURE ZONE statement below requires that the cluster try to place the ranges' leaseholders in zone us-east1; if that is not possible, it will try to place them in zone us-west1.

For more information about how the lease_preferences field works, see its description in the Replication zone variables section.

> ALTER TABLE users CONFIGURE ZONE USING num_replicas = 3, constraints = '{"+region=us-east1": 1, "+region=us-west1": 1}', lease_preferences = '[[+region=us-east1], [+region=us-west1]]';

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR TABLE users;

    target    |                           raw_config_sql
+-------------+--------------------------------------------------------------------+
  TABLE users | ALTER TABLE users CONFIGURE ZONE USING
              |     range_min_bytes = 134217728,
              |     range_max_bytes = 536870912,
              |     gc.ttlseconds = 100000,
              |     num_replicas = 3,
              |     constraints = '{+region=us-east1: 1, +region=us-west1: 1}',
              |     lease_preferences = '[[+region=us-east1], [+region=us-west1]]'
(1 row)

For more information, see CONFIGURE ZONE.

Scenario-based examples

Even replication across availability zones

Scenario:

• You have 6 nodes across 3 availability zones, 2 nodes in each availability zone.
• You want data replicated 3 times, with replicas balanced evenly across all three availability zones.

Approach:

1. Start each node with its availability zone location specified in the --locality flag:

   Availability zone 1:

   $ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=az=us-1 \
   --join=<node1 hostname>,<node3 hostname>,<node5 hostname>
   $ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=az=us-1 \
   --join=<node1 hostname>,<node3 hostname>,<node5 hostname>
   

   Availability zone 2:

   $ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=az=us-2 \
   --join=<node1 hostname>,<node3 hostname>,<node5 hostname>
   $ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=az=us-2 \
   --join=<node1 hostname>,<node3 hostname>,<node5 hostname>
   

   Availability zone 3:

   $ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=az=us-3 \
   --join=<node1 hostname>,<node3 hostname>,<node5 hostname>
   $ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=az=us-3 \
   --join=<node1 hostname>,<node3 hostname>,<node5 hostname>
   

2. Initialize the cluster:

   $ cockroach init --insecure --host=<any node hostname>
   

There's no need to make zone configuration changes; by default, the cluster is configured to replicate data three times, and even without explicit constraints, the cluster will aim to diversify replicas across node localities.

Per-replica constraints to specific availability zones

Scenario:

• You have 5 nodes across 5 availability zones in 3 regions, 1 node in each availability zone.
• You want data replicated 3 times, with a quorum of replicas for a database holding West Coast data centered on the West Coast and a database for nation-wide data replicated across the entire country.

Approach:

1. Start each node with its region and availability zone location specified in the --locality flag:

   Start the five nodes:

   $ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=region=us-west1,az=us-west1-a \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
   $ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=region=us-west1,az=us-west1-b \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
   $ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=region=us-central1,az=us-central1-a \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
   $ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=region=us-east1,az=us-east1-a \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
   $ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=region=us-east1,az=us-east1-b \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
   

   Initialize the cluster:

   $ cockroach init --insecure --host=<any node hostname>
   

2. On any node, open the built-in SQL client:

   $ cockroach sql --insecure
   

3. Create the database for the West Coast application:

   > CREATE DATABASE west_app_db;
   

4. Configure a replication zone for the database:

   > ALTER DATABASE west_app_db
   CONFIGURE ZONE USING constraints = '{"+region=us-west1": 2, "+region=us-central1": 1}', num_replicas = 3;
   

   CONFIGURE ZONE 1
   

5. View the replication zone:

   > SHOW ZONE CONFIGURATION FOR DATABASE west_app_db;
   

            target        |                           raw_config_sql
   +----------------------+--------------------------------------------------------------------+
     DATABASE west_app_db | ALTER DATABASE west_app_db CONFIGURE ZONE USING
                          |     range_min_bytes = 134217728,
                          |     range_max_bytes = 536870912,
                          |     gc.ttlseconds = 90000,
                          |     num_replicas = 3,
                          |     constraints = '{+region=us-central1: 1, +region=us-west1: 2}',
                          |     lease_preferences = '[]'
   (1 row)
   

   Two of the database's three replicas will be put in region=us-west1 and its remaining replica will be put in region=us-central1. This gives the application the resilience to survive the total failure of any one availability zone while providing low-latency reads and writes on the West Coast because a quorum of replicas are located there.

6. No configuration is needed for the nation-wide database. The cluster is configured to replicate data 3 times and spread them as widely as possible by default. Because the first key-value pair specified in each node's locality is considered the most significant part of each node's locality, spreading data as widely as possible means putting one replica in each of the three different regions.

Multiple applications writing to different databases

Scenario:

• You have 2 independent applications connected to the same CockroachDB cluster, each application using a distinct database.
• You have 6 nodes across 2 availability zone, 3 nodes in each availability zone.
• You want the data for application 1 to be replicated 5 times, with replicas evenly balanced across both availability zones.
• You want the data for application 2 to be replicated 3 times, with all replicas in a single availability zone.

Approach:

1. Start each node with its availability zone location specified in the --locality flag:

   Availability zone 1:

   $ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=az=us-1 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
   $ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=az=us-1 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
   $ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=az=us-1 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
   

   Availability zone 2:

   $ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=az=us-2 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
   $ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=az=us-2 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
   $ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=az=us-2 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
   

   Initialize the cluster:

   $ cockroach init --insecure --host=<any node hostname>
   

2. On any node, open the built-in SQL client:

   $ cockroach sql --insecure
   

3. Create the database for application 1:

   > CREATE DATABASE app1_db;
   

4. Configure a replication zone for the database used by application 1:

   > ALTER DATABASE app1_db CONFIGURE ZONE USING num_replicas = 5;
   

   CONFIGURE ZONE 1
   

5. View the replication zone:

   > SHOW ZONE CONFIGURATION FOR DATABASE app1_db;
   

          target      |               raw_config_sql
   +------------------+---------------------------------------------+
     DATABASE app1_db | ALTER DATABASE app1_db CONFIGURE ZONE USING
                      |     range_min_bytes = 134217728,
                      |     range_max_bytes = 536870912,
                      |     gc.ttlseconds = 90000,
                      |     num_replicas = 5,
                      |     constraints = '[]',
                      |     lease_preferences = '[]'
   (1 row)
   

   Nothing else is necessary for application 1's data. Since all nodes specify their availability zone locality, the cluster will aim to balance the data in the database used by application 1 between availability zones 1 and 2.

6. Still in the SQL client, create a database for application 2:

   > CREATE DATABASE app2_db;
   

7. Configure a replication zone for the database used by application 2:

   > ALTER DATABASE app2_db CONFIGURE ZONE USING constraints = '[+az=us-2]';
   

8. View the replication zone:

   > SHOW ZONE CONFIGURATION FOR DATABASE app2_db;
   

          target      |               raw_config_sql
   +------------------+---------------------------------------------+
     DATABASE app2_db | ALTER DATABASE app2_db CONFIGURE ZONE USING
                      |     range_min_bytes = 134217728,
                      |     range_max_bytes = 536870912,
                      |     gc.ttlseconds = 90000,
                      |     num_replicas = 3,
                      |     constraints = '[+az=us-2]',
                      |     lease_preferences = '[]'
   (1 row)
   

   The required constraint will force application 2's data to be replicated only within the us-2 availability zone.

Stricter replication for a table and its secondary indexes

Scenario:

• You have 7 nodes, 5 with SSD drives and 2 with HDD drives.
• You want data replicated 3 times by default.
• Speed and availability are important for a specific table and its indexes, which are queried very frequently, however, so you want the data in the table and secondary indexes to be replicated 5 times, preferably on nodes with SSD drives.

Approach:

1. Start each node with ssd or hdd specified as store attributes:

   5 nodes with SSD storage:

   $ cockroach start --insecure --advertise-addr=<node1 hostname> --store=path=node1,attrs=ssd \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node2 hostname> --store=path=node2,attrs=ssd \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node3 hostname> --store=path=node3,attrs=ssd \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node4 hostname> --store=path=node4,attrs=ssd \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node5 hostname> --store=path=node5,attrs=ssd \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   

   2 nodes with HDD storage:

   $ cockroach start --insecure --advertise-addr=<node6 hostname> --store=path=node6,attrs=hdd \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node7 hostname> --store=path=node7,attrs=hdd \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   

   Initialize the cluster:

   $ cockroach init --insecure --host=<any node hostname>
   

2. On any node, open the built-in SQL client:

   $ cockroach sql --insecure
   

3. Create a database and table:

   > CREATE DATABASE db;
   

   > CREATE TABLE db.important_table;
   

4. Configure a replication zone for the table that must be replicated more strictly:

   > ALTER TABLE db.important_table CONFIGURE ZONE USING num_replicas = 5, constraints = '[+ssd]'
   

5. View the replication zone:

   > SHOW ZONE CONFIGURATION FOR TABLE db.important_table;
   

                target             |                config_sql
   +-------------------------------+---------------------------------------------+
        TABLE db.important_table   | ALTER DATABASE app2_db CONFIGURE ZONE USING
                                   |     range_min_bytes = 1048576,
                                   |     range_max_bytes = 536870912,
                                   |     gc.ttlseconds = 90000,
                                   |     num_replicas = 5,
                                   |     constraints = '[+ssd]',
                                   |     lease_preferences = '[]'
   (1 row)
   

   The secondary indexes on the table will use the table's replication zone, so all data for the table will be replicated 5 times, and the required constraint will place the data on nodes with ssd drives.

Tweaking the replication of system ranges

Scenario:

• You have nodes spread across 7 availability zones.
• You want data replicated 5 times by default.
• For better performance, you want a copy of the meta ranges in all of the availability zones.
• To save disk space, you only want the internal timeseries data replicated 3 times by default.

Approach:

1. Start each node with a different locality attribute:

   $ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=az=us-1 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>   
   $ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=az=us-2 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=az=us-3 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=az=us-4 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=az=us-5 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=az=us-6 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   $ cockroach start --insecure --advertise-addr=<node7 hostname> --locality=az=us-7 \
   --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
   

   Initialize the cluster:

   $ cockroach init --insecure --host=<any node hostname>
   

2. On any node, open the built-in SQL client:

   $ cockroach sql --insecure
   

3. Configure the default replication zone:

   > ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5;
   

4. View the replication zone:

   > SHOW ZONE CONFIGURATION FOR RANGE default;
   

        target     |              raw_config_sql
   +---------------+------------------------------------------+
     RANGE default | ALTER RANGE default CONFIGURE ZONE USING
                   |     range_min_bytes = 134217728,
                   |     range_max_bytes = 536870912,
                   |     gc.ttlseconds = 90000,
                   |     num_replicas = 5,
                   |     constraints = '[]',
                   |     lease_preferences = '[]'
   (1 row)
   

   All data in the cluster will be replicated 5 times, including both SQL data and the internal system data.

5. Configure the meta replication zone:

   > ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 7;
   

   > SHOW ZONE CONFIGURATION FOR RANGE meta;
   

       target   |            raw_config_sql
   +------------+---------------------------------------+
     RANGE meta | ALTER RANGE meta CONFIGURE ZONE USING
                |     range_min_bytes = 134217728,
                |     range_max_bytes = 536870912,
                |     gc.ttlseconds = 3600,
                |     num_replicas = 7,
                |     constraints = '[]',
                |     lease_preferences = '[]'
   (1 row)
   

   The meta addressing ranges will be replicated such that one copy is in all 7 availability zones, while all other data will be replicated 5 times.

6. Configure the timeseries replication zone:

   > ALTER RANGE timeseries CONFIGURE ZONE USING num_replicas = 3;
   

   > SHOW ZONE CONFIGURATION FOR RANGE timeseries;
   

          target      |               raw_config_sql
   +------------------+---------------------------------------------+
     RANGE timeseries | ALTER RANGE timeseries CONFIGURE ZONE USING
                      |     range_min_bytes = 134217728,
                      |     range_max_bytes = 536870912,
                      |     gc.ttlseconds = 90000,
                      |     num_replicas = 3,
                      |     constraints = '[]',
                      |     lease_preferences = '[]'
   (1 row)
   

   The timeseries data will only be replicated 3 times without affecting the configuration of all other data.

See also

• SHOW ZONE CONFIGURATIONS
• CONFIGURE ZONE
• SHOW PARTITIONS
• SQL Statements
• Table Partitioning
• Replication Reports

Was this helpful?