Configure logs

On this page

This page describes how to configure CockroachDB logs with the --log or log-config-file flag and a YAML payload. Most logging behaviors are configurable, including:

The log sinks that output logs to different locations, including over the network.
The logging channels that are mapped to each sink.
The format used by the log messages.
The redaction of log messages.

For examples of how these settings can be used in practice, see Logging Use Cases.

Note:

The logging flags previously used with cockroach commands are now deprecated. Instead, use the YAML definitions as described below. The default logging configuration uses YAML settings that are backward-compatible with the log files used in v20.2 and earlier.

Flag

To configure the logging behavior of a cockroach command, include one of these flags with the command:

--log={yaml}, where {yaml} is the YAML payload
--log-config-file={yaml-file}, where {yaml-file} is the path to a YAML file

Tip:

All cockroach commands support logging and can be configured with --log or --log-config-file. However, note that most messages related to cluster operation are generated by cockroach start or cockroach start-single-node. Other commands generate messages related to their own execution, which are mainly useful when troubleshooting the behaviors of those commands.

YAML payload

All log settings for a cockroach command are specified with a YAML payload in one of the following formats:

Block format, where each parameter is written on a separate line. For example, after creating a file logs.yaml, pass the YAML values with either --log-config-file or --log:
```
$ cockroach start-single-node --certs-dir=certs --log-config-file=logs.yaml
```
```
$ cockroach start-single-node --certs-dir=certs --log="$(cat logs.yaml)"
```
Inline format, where all parameters are specified on one line. For example, to generate an ops log file that collects the OPS and HEALTH channels (overriding the file groups defined for those channels in the default configuration):
```
$ cockroach start-single-node --certs-dir=certs --log="sinks: {file-groups: {ops: {channels: [OPS, HEALTH]}}}"
```
Note that the inline spaces must be preserved.

For clarity, this article uses the block format to describe the YAML payload, which has the overall structure:

file-defaults: ...       # defaults inherited by file sinks
fluent-defaults: ...     # defaults inherited by network sinks
sinks:
   file-groups: ...      # file sink definitions
   fluent-servers: ...   # network sink definitions
   stderr: ...           # stderr sink definitions
capture-stray-errors: ... # parameters for the stray error capture system

Note:

Providing a logging configuration is optional. Any fields included in the YAML payload will override the same fields in the default logging configuration.

Tip:

You can view your current settings by running cockroach debug check-log-config, which returns the YAML definitions and a URL to a visualization of the current logging configuration.

Configure log sinks

Log sinks route events from specified logging channels to destinations outside CockroachDB. These destinations currently include log files, Fluentd-compatible servers, and the standard error stream (stderr).

All supported output destinations are configured under sinks:

file-defaults: ...
fluent-defaults: ...
sinks:
  file-groups:
    {file group name}:
      channels: {channels}
      ...
  fluent-servers:
    {server name}:
      channels: {channels}
      ...
  stderr:
      channels: {channels}
      ...

All supported sink types use the following common sink parameters:

Parameter	Description
`filter`	Minimum severity level at which logs enter the channels selected for the sink. Accepts one of the valid severity levels or `NONE`, which excludes all messages from the sink output. For details, see Set logging levels.
`format`	Log message format to use for file or network sinks. Accepts one of the valid log formats. For details, see file logging format and network logging format.
`redact`	When `true`, enables automatic redaction of personally identifiable information (PII) from log messages. This ensures that sensitive data is not transmitted when collecting logs centrally or over a network. For details, see Redact logs.
`redactable`	When `true`, preserves redaction markers around fields that are considered sensitive in the log messages. The markers are recognized by `cockroach debug zip` and `cockroach debug merge-logs` but may not be compatible with external log collectors. For details on how the markers appear in each format, see Log formats.
`exit-on-error`	When `true`, stops the Cockroach node if an error is encountered while writing to the sink. We recommend enabling this option on file sinks in order to avoid losing any log entries. When set to `false`, this can be used to mark certain sinks (such as `stderr`) as non-critical.
`auditable`	If `true`, enables `exit-on-error` on the sink. Also disables `buffered-writes` if the sink is under `file-groups`. This guarantees non-repudiability for any logs in the sink, but can incur a performance overhead and higher disk IOPS consumption. This setting is typically enabled for security-related logs.

If not specified for a given sink, these parameter values are inherited from file-defaults (for file sinks) and fluent-defaults (for network sinks).

Output to files

CockroachDB can write messages to one or more log files.

file-groups specifies the channels that output to each log file, along with its output directory and other configuration details. For example:

file-defaults: ...
fluent-defaults: ...
sinks:
  file-groups:
    default:
      channels: [DEV]
    health:
      channels: [HEALTH]
      dir: health-logs
    ...

Tip:

A file group name is arbitrary and is used to name the log files. The default file group is an exception. For details, see Log file naming.

Along with the common sink parameters, each file group accepts the following parameters:

Parameter	Description
`channels`	List of channels that output to this sink. Use a YAML array or string of channel names, `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names. For more details on acceptable syntax, see Logging channel selection.
`dir`	Output directory for log files generated by this sink.
`max-file-size`	Approximate maximum size of individual files generated by this sink.
`max-group-size`	Approximate maximum combined size of all files to be preserved for this sink. An asynchronous garbage collection removes files that cause the file set to grow beyond this specified size.
`buffered-writes`	When `true`, enables buffering of writes. Set to `false` to flush every log entry (i.e., propagate data from the `cockroach` process to the OS) and synchronize writes (i.e., ask the OS to confirm the log data was written to disk). Disabling this setting provides non-repudiation guarantees, but can incur a performance overhead and higher disk IOPS consumption. This setting is typically disabled for security-related logs.

If not specified for a given file group, the parameter values are inherited from file-defaults (except channels, which uses the default configuration).

Log file naming

Log files are named in the following format:

{process}-{file group}.{host}.{user}.{start timestamp in UTC}.{process ID}.log

For example, a file group health will generate a log file that looks like this:

cockroach-health.work-laptop.worker.2021-03-15T15_24_10Z.024338.log

For each file group, a symlink points to the latest generated log file. It's easiest to refer to the symlink. For example:

cockroach-health.log

Note:

The files generated for a group named default are named after the pattern cockroach.{metadata}.log.

Access in DB Console

Tip:

Log files can be accessed using the DB Console, which displays them in JSON format.

Access the DB Console and then click Advanced Debug in the left-hand navigation.
Under Raw Status Endpoints (JSON), click Log Files to view the JSON of all collected logs.
Copy one of the log filenames. Then click Specific Log File and replace the cockroach.log placeholder in the URL with the filename.

Known limitations

Log files can only be accessed in the DB Console if they are stored in the same directory as the file sink for the DEV channel.

Output to network

CockroachDB can send logs over the network to a Fluentd-compatible log collector (e.g., Elasticsearch, Splunk). fluent-servers specifies the channels that output to a server, along with its configuration details. For example:

file-defaults: ...
fluent-defaults: ...
sinks:
  fluent-servers:
    health:
      channels: [HEALTH]
      address: 127.0.0.1:5170
    ...

Note:

A network sink can be listed more than once with different address values. This routes the same logs to different Fluentd servers.

Along with the common sink parameters, each Fluentd server accepts the following parameters:

Parameter	Description
`channels`	List of channels that output to this sink. Use a YAML array or string of channel names, `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names. For more details on acceptable syntax, see Logging channel selection.
`address`	Network address and port of the log collector.
`net`	Network protocol to use. Can be `tcp`, `tcp4`, `tcp6`, `udp`, `udp4`, `udp6`, or `unix`. Default: `tcp`

A Fluentd sink buffers at most one log entry and retries sending the event at most one time if a network error is encountered. This is just sufficient to tolerate a restart of the Fluentd collector after a configuration change under light logging activity. If the server is unavailable for too long, or if more than one error is encountered, an error is reported to the process's standard error output with a copy of the logging event, and the logging event is dropped.

For an example network logging configuration, see Logging use cases.

Output to `stderr`

CockroachDB can output messages to the standard error stream (stderr), which prints them to the machine's terminal but does not store them. stderr specifies the channels that output to the stream. For example:

file-defaults: ...
fluent-defaults: ...
sinks:
  stderr:
    channels: [DEV]

Along with the common sink parameters, stderr accepts the following parameters:

Note:

The format parameter for stderr is set to crdb-v2-tty and cannot be changed.

Parameter	Description
`channels`	List of channels that output to this sink. Use a YAML array or string of channel names, `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names. For more details on acceptable syntax, see Logging channel selection.
`no-color`	When `true`, removes terminal color escape codes from the output.

Because server start-up messages are always emitted at the start of the standard error stream, it is generally difficult to automate integration of stderr with log analyzers. We recommend using file logging or network logging instead of stderr when integrating with automated monitoring software.

Note:

By default, cockroach start and cockroach start-single-node do not print any messages to stderr. However, if the cockroach process does not have access to on-disk storage, all messages are printed to stderr.

Logging channel selection

Each sink can select multiple channels. The names of selected channels can be specified as a YAML array or as a string.

# Select just these two channels. Space is important.
channels: [OPS, HEALTH]

# Selection is case-insensitive.
channels: [ops, HeAlTh]

# Same configuration, as a YAML string. 
# Avoid space around commas when using the YAML inline format.
channels: OPS,HEALTH

# Same configuration, as a quoted string.
channels: 'OPS, HEALTH'

# Same configuration, as a multi-line YAML array.
channels:
- OPS
- HEALTH

To select all channels, using the all keyword:

channels: all
channels: 'all'
channels: [all]
channels: ['all']

To select all channels except for a subset, using the all except keyword prefix:

channels: all except ops,health
channels: all except [ops,health]
channels: 'all except ops, health'
channels: 'all except [ops, health]'

Configure logging defaults

When setting up a logging configuration, it's simplest to define shared parameters in file-defaults and fluent-defaults and override specific values as needed in file-groups, fluent-servers, and stderr. For a complete example, see the default configuration.

Tip:

You can view your current settings by running cockroach debug check-log-config, which returns the YAML definitions and a URL to a visualization of the current logging configuration.

This section describes some recommended defaults.

Set file defaults

Defaults for log files are set in file-defaults, which accepts all common sink parameters and the file group parameters dir, max-file-size, max-group-size, and buffered-writes.

Logging directory

By default, CockroachDB adds log files to a logs subdirectory in the first on-disk store directory (default: cockroach-data):

cockroach-data/logs

Tip:

Each Cockroach node generates log files in the directory specified by its logging configuration. These logs detail the internal activity of that node without visibility into the behavior of other nodes. When troubleshooting, it's best to refer to the output directory for the cluster log files, which collect the messages from all active nodes.

In cloud deployments, the main data store will be subject to an IOPS budget. Adding logs to the store directory will excessively consume IOPS. For this reason, cloud deployments should output log files to a separate directory with fewer IOPS restrictions. You can override the default logging directory like this:

file-defaults:
  dir: /custom/dir/path/

File logging format

The default message format for log files is crdb-v2. Each crdb-v2 log message starts with a flat prefix that contains event metadata (e.g., severity, date, timestamp, channel), followed by the event payload. For details on the metadata, see Log formats.

If you plan to read logs programmatically, you can switch to a JSON or compact JSON format:

file-defaults:
  format: json

Note:

format refers to the envelope of the log message, which contains the event metadata. This is separate from the event payload, which corresponds to its event type.

Set network defaults

Defaults for network sinks are set in fluent-defaults, which accepts all common sink parameters.

Note that the server parameters address and net are not specified in fluent-defaults:

address must be specified for each sink under fluent-servers.
net is not required and defaults to tcp.

Network logging format

The default message format for network output is json-fluent-compact. Each log message is structured as a JSON payload that can be read programmatically. The json-fluent-compact and json-fluent formats include a tag field that is required by the Fluentd protocol. For details, see Log formats.

fluent-defaults:
  format: json-fluent

Note:

format refers to the envelope of the log message. This is separate from the event payload, which is structured according to event type.

Set logging levels

Log messages are associated with a severity level when they are generated. Each logging sink is configured to output messages that meet a minimum severity level, specified with filter. Messages with severity levels below the filter threshold do not enter logging channels and are discarded.

The default configuration uses the following severity levels for cockroach start and cockroach start-single-node:

file-defaults and fluent-defaults each use filter: INFO. Since INFO is the lowest severity level, file and network sinks will emit all log messages.
stderr uses filter: NONE and does not emit log messages.

Note:

All other cockroach commands use filter: WARNING and log to stderr by default, with these exceptions:

cockroach workload uses filter: INFO.
cockroach demo uses filter: NONE (discards all log messages).

You can also override the file-defaults and fluent-defaults severity levels on a per-sink basis. For example, use filter: NONE to disable logging for certain channels:

file-defaults: ...
fluent-defaults: ...
sinks:
  file-groups:
    dev:
      channels: [DEV]
      filter: NONE

Redact logs

CockroachDB can redact personally identifiable information (PII) from log messages. The logging system includes two parameters that handle this differently:

redact is disabled by default. When enabled, redact automatically redacts sensitive data from logging output. We do not recommend enabling this on the DEV channel because it impairs our ability to troubleshoot problems.
redactable is enabled by default. This places redaction markers around sensitive fields in log messages. These markers are recognized by cockroach debug zip and cockroach debug merge-logs, which aggregate CockroachDB log files and can be instructed to redact sensitive data from their output.

When collecting logs centrally (e.g., in data mining scenarios where non-privileged users have access to logs) or over a network (e.g., to an external log collector), it's safest to enable redact:

file-defaults:
  redact: true
fluent-defaults:
  redact: true

Tip:

In addition, the DEV channel should be output to a separate logging directory, since it is likely to contain sensitive data. See DEV channel.

External log collectors can misinterpret the cockroach debug redaction markers, since they are specific to CockroachDB. To prevent this issue when using network sinks, disable redactable:

fluent-defaults:
  redactable: false

DEV channel

The DEV channel is used for debug and uncategorized messages. It can therefore be noisy and contain sensitive (PII) information.

We recommend configuring DEV separately from the other logging channels. When sending logs over the network, DEV logs should also be excluded from network collection.

In this example, the dev file group is reserved for DEV logs. These are output to a cockroach-dev.log file in a custom disk dir:

file-defaults: ...
fluent-defaults: ...
sinks:
  file-groups:
    dev:
      channels: [DEV]
      dir: /custom/dir/path/
    ...
  fluent-servers:
    ...

Tip:

To ensure that you are protecting sensitive information, also redact your logs.

Stray error capture

Certain events, such as uncaught software exceptions (panics), bypass the CockroachDB logging system. However, they can be useful in troubleshooting. For example, if CockroachDB crashes, it normally logs a stack trace to what caused the problem.

To ensure that these stray errors can be tracked, CockroachDB does not send them to stderr by default. Instead, stray errors are output to a cockroach-stderr.log file in the default logging directory.

You can change these settings in capture-stray-errors:

file-defaults: ...
fluent-defaults: ...
sinks: ...
capture-stray-errors:
  enable: true
  dir: /custom/dir/path/

Note:

When capture-stray-errors is disabled, redactable cannot be enabled on the stderr sink. This is because stderr will contain both stray errors and logged events and cannot apply redaction markers in a reliable way. Note that redact can still be enabled on stderr in this case.

Default logging configuration

The YAML payload below represents the default logging behavior of cockroach start and cockroach start-single-node. To retain backward compatibility with v20.2 and earlier, these settings match the log filenames used in previous versions (except sql-auth, which should be changed to auth to match the log filename used in v20.2 and earlier).

file-defaults:
  max-file-size: 10MiB
  max-group-size: 100MiB
  buffered-writes: true
  filter: INFO
  format: crdb-v2
  redact: false
  redactable: true
  exit-on-error: true
  auditable: false
fluent-defaults:
  filter: INFO
  format: json-fluent-compact
  redact: false
  redactable: true
  exit-on-error: false
  auditable: false
sinks:
  file-groups:
    default:
      channels: [DEV, OPS, HEALTH, SQL_SCHEMA, USER_ADMIN, PRIVILEGES]
    pebble:
      channels: [STORAGE]
    sql-audit:
      channels: [SENSITIVE_ACCESS]
      auditable: true
    sql-auth:
      channels: [SESSIONS]
      auditable: true
    sql-exec:
      channels: [SQL_EXEC]
    sql-slow:
      channels: [SQL_PERF]
    sql-slow-internal-only:
      channels: [SQL_INTERNAL_PERF]
  stderr:
    channels: all
    filter: NONE
    redact: false
    redactable: true
    exit-on-error: true
capture-stray-errors:
  enable: true
  max-group-size: 100MiB

Note that a default dir is not specified for file-defaults and capture-stray-errors:

The default dir for file-defaults is inferred from the first on-disk store directory. See Logging directory.
The default dir for capture-stray-errors is inherited form file-defaults.

Pricing

Contact us

Sign In

Configure logs

Flag

YAML payload

Configure log sinks

Output to files

Log file naming

Access in DB Console

Known limitations

Output to network

Output to `stderr`

Logging channel selection

Configure logging defaults

Set file defaults

Logging directory

File logging format

Set network defaults

Network logging format

Set logging levels

Redact logs

DEV channel

Stray error capture

Default logging configuration

See also

Configure logs

Flag

YAML payload

Configure log sinks

Output to files

Log file naming

Access in DB Console

Known limitations

Output to network

Output to stderr

Logging channel selection

Configure logging defaults

Set file defaults

Logging directory

File logging format

Set network defaults

Network logging format

Set logging levels

Redact logs

DEV channel

Stray error capture

Default logging configuration

See also

Output to `stderr`