Metrics are reported per node. Therefore, it is necessary to retrieve metrics from every node in the cluster. For example, if you are monitoring whether a backup fails, it is necessary to track
scheduled_backup_failed on each node.Prometheus endpoint
You can access the (http://<host:<http-port/_status/vars) for backup and restore metrics.
Refer to the tutorial for guidance on installing and setting up Prometheus and Alertmanager to track metrics.
Available metrics
We recommend the following guidelines:- Use the
schedules.BACKUP.last-completed-timemetric to monitor the specific backup job or jobs you would use to recover from a disaster. - Configure alerting on the
schedules.BACKUP.last-completed-timemetric to watch for cases where the timestamp has not moved forward as expected.
| Metric | Description |
|---|---|
schedules.BACKUP.failed | The number of scheduled backup jobs that have failed. Note: A stuck scheduled job will not increment this metric. |
schedules.BACKUP.last-completed-time | The Unix timestamp of the most recently completed scheduled backup specified as maintaining this metric. Note: This metric only updates if the schedule was created with the . |
schedules.BACKUP.protected\_age\_sec | The age of the oldest protected by backup schedules. |
schedules.BACKUP.protected\_record\_count | The number of held by backup schedules. |
schedules.BACKUP.started | The number of scheduled backup jobs that have started. |
schedules.BACKUP.succeeded | The number of scheduled backup jobs that have succeeded. |
schedules.round.reschedule\_skip | The number of schedules that were skipped due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the schedule option. |
schedules.round.reschedule\_wait | The number of schedules that were rescheduled due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the schedule option. |
jobs.backup.currently\_paused | The number of backup jobs currently considered . |
jobs.backup.currently\_running | The number of backup jobs currently running in Resume or OnFailOrCancel state. |
jobs.backup.fail\_or\_cancel\_retry\_error | The number of backup jobs that failed with a retryable error on their failure or cancelation process. |
jobs.backup.fail\_or\_cancel\_completed | The number of backup jobs that successfully completed their failure or cancelation process. |
jobs.backup.fail\_or\_cancel\_failed | The number of backup jobs that failed with a non-retryable error on their failure or cancelation process. |
jobs.backup.protected\_age\_sec | The age of the oldest protected by backup jobs. |
jobs.backup.protected\_record\_count | The number of held by backup jobs. |
jobs.backup.resume\_failed | The number of backup jobs that failed with a non-retryable error. |
jobs.backup.resume\_retry\_error | The number of backup jobs that failed with a retryable error. |
jobs.restore.currently\_paused | The number of restore jobs currently considered . |
jobs.restore.currently\_running | The number of restore jobs currently running in Resume or OnFailOrCancel state. |
jobs.restore.fail\_or\_cancel\_failed | The number of restore jobs that failed with a non-retriable error on their failure or cancelation process. |
jobs.restore.fail\_or\_cancel\_retry\_error | The number of restore jobs that failed with a retryable error on their failure or cancelation process. |
jobs.restore.protected\_age\_sec | The age of the oldest protected by restore jobs. |
jobs.restore.protected\_record\_count | The number of held by restore jobs. |
jobs.restore.resume\_completed | The number of restore jobs that successfully resumed to completion. |
jobs.restore.resume\_failed | The number of restore jobs that failed with a non-retryable error. |
jobs.restore.resume\_retry\_error | The number of restore jobs that failed with a retryable error. |