The Hardware dashboard lets you monitor the hardware utilization of your cluster. This includes CPU usage, disk throughput, storage capacity, and memory.
To view this dashboard, access the DB Console, click Metrics in the left-hand navigation, and select Dashboard > Hardware.
Dashboard navigation
Use the Graph menu to display metrics for your entire cluster or for a specific node.
To the right of the Graph and Dashboard menus, a time interval selector allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected in the SQL Activity pages. However, if you select 10 or 30 minutes, the interval defaults to 1 hour in SQL Activity pages.
Hovering your mouse pointer over the graph title will display a tooltip with a description and the metrics used to create the graph.
When hovering on graphs, crosshair lines will appear at your mouse pointer. The series' values corresponding to the given time in the cross hairs are displayed in the legend under the graph. Hovering the mouse pointer on a given series displays the corresponding value near the mouse pointer and highlights the series line (graying out other series lines). Click anywhere within the graph to freeze the values in place. Click anywhere within the graph again to cause the values to change with your mouse movements once more.
In the legend, click on an individual series to isolate it on the graph. The other series will be hidden, while the hover will still work. Click the individual series again to make the other series visible. If there are many series, a scrollbar may appear on the right of the legend. This is to limit the size of the legend so that it does not get endlessly large, particularly on clusters with many nodes.
The Hardware dashboard displays the following time series graphs:
CPU Percent
This graph displays the current user and system CPU percentage consumed by the CockroachDB process, normalized by number of cores, as tracked by the sys.cpu.combined.percent-normalized metric.
This graph shows the CPU consumption by the CockroachDB process, and excludes other processes on the node. Use the Host CPU Percent graph to measure the total CPU consumption across all processes.
In the node view, the graph shows the percentage of CPU utilization of the CockroachDB process as measured by the host for the selected node.
In the cluster view, the graph shows the percentage of CPU utilization of the CockroachDB process as measured by the host across all nodes.
Expected values for a healthy cluster: CPU utilized by CockroachDB should not persistently exceed 80%. Because this metric does not reflect system CPU usage, values above 80% suggest that actual CPU utilization is nearing 100%.
For multi-core systems, the percentage of CPU usage is calculated by normalizing the CPU usage across all cores, whereby 100% utilization indicates that all cores are fully utilized.
Host CPU Percent
The Host CPU Percent graph displays the current user and system CPU percentage consumed by all processes on the host OS, normalized by number of cores, as tracked by the sys.cpu.host.combined.percent-normalized metric. If the CockroachDB process is run in a containerized environment, the host OS is the container since the CockroachDB process cannot inspect CPU usage beyond the container.
In the node view, the graph shows the percentage of CPU in use by all processes for the selected node.
In the cluster view, the graph shows the percentage of CPU in use by all processes across all nodes.
For multi-core systems, the percentage of CPU usage is calculated by normalizing the CPU usage across all cores, where 100% utilization indicates that all cores are fully utilized.
Memory Usage
This graph shows the memory consumption by the CockroachDB process only and is useful as long as there are no other processes consuming significant memory on the node. In case you have other processes running on the node, use a separate monitoring tool to measure the total memory consumption across all processes.
In the node view, the graph shows the memory in use by CockroachDB for the selected node.
In the cluster view, the graph shows the memory in use by CockroachDB across all nodes in the cluster.
Disk Read Bytes
In the node view, the graph shows the 10-second average of the number of bytes read per second by all processes, including CockroachDB, for the selected node.
In the cluster view, the graph shows the 10-second average of the number of bytes read per second by all processes, including CockroachDB, across all nodes.
Disk Write Bytes
In the node view, the graph shows the 10-second average of the number of bytes written per second by all processes, including CockroachDB, for the node.
In the cluster view, the graph shows the 10-second average of the number of bytes written per second by all processes, including CockroachDB, across all nodes.
Disk Read Ops
In the node view, the graph shows the 10-second average of the number of disk read ops per second for all processes, including CockroachDB, for the selected node.
In the cluster view, the graph shows the 10-second average of the number of disk read ops per second for all processes, including CockroachDB, across all nodes.
Disk Write Ops
In the node view, the graph shows the 10-second average of the number of disk write ops per second for all processes, including CockroachDB, for the node.
In the cluster view, the graph shows the 10-second average of the number of disk write ops per second for all processes, including CockroachDB, across all nodes.
Disk Ops In Progress
In the node view, the graph shows the number of disk reads and writes in queue for all processes, including CockroachDB, for the selected node.
In the cluster view, the graph shows the number of disk reads and writes in queue for all processes, including CockroachDB, across all nodes in the cluster.
Expected values for a healthy cluster: This value should be 0 or single-digit values for short periods of time. If the values persist in double digits, you may have an I/O bottleneck.
On macOS, this graph is currently not populated and shows zero disk ops in progress.
Available Disk Capacity
| Metric | Description |
|---|---|
| Available Disk Capacity | Free disk space available to CockroachDB data on each node. |
Capacity metrics
The available disk capacity equals the amount of empty disk space, up to the value of the maximum store size. The store size is determined as follows:
- If a store size was specified using the
--storeflag when starting nodes, this value is used as the limit for CockroachDB data. - If no store size has been explicitly set, the actual disk capacity is used as the limit for CockroachDB data.
The disk usage of the Cockroach binary, operating system, and other system files is not shown on the Available Disk Capacity graph.
If you are testing your deployment locally with multiple CockroachDB nodes running on a single machine (this is not recommended in production), you must explicitly set the store size per node in order to display the correct capacity. Otherwise, the machine's actual disk capacity will be counted as a separate store for each node, thus inflating the computed capacity.
Summary and events
Summary panel
A Summary panel of key metrics is displayed to the right of the timeseries graphs.
| Metric | Description |
|---|---|
| Total Nodes | The total number of nodes in the cluster. Decommissioned nodes are not included in this count. |
| Capacity Used | The storage capacity used as a percentage of usable capacity allocated across all nodes. |
| Unavailable Ranges | The number of unavailable ranges in the cluster. A non-zero number indicates an unstable cluster. |
| Queries per second | The total number of SELECT, UPDATE, INSERT, and DELETE queries executed per second across the cluster. |
| P99 Latency | The 99th percentile of service latency. |
If you are testing your deployment locally with multiple CockroachDB nodes running on a single machine (this is not recommended in production), you must explicitly set the store size per node in order to display the correct capacity. Otherwise, the machine's actual disk capacity will be counted as a separate store for each node, thus inflating the computed capacity.
Events panel
Underneath the Summary panel, the Events panel lists the 5 most recent events logged for all nodes across the cluster. To list all events, click View all events.
The following types of events are listed:
- Database created
- Database dropped
- Table created
- Table dropped
- Table altered
- Index created
- Index dropped
- View created
- View dropped
- Schema change reversed
- Schema change finished
- Node joined
- Node decommissioned
- Node restarted
- Cluster setting changed
See also
Was this helpful?
