Logging and Debugging
Typically, when you add debugging to your Ceph configuration, you do so at runtime. You can also add Ceph debug logging to your Ceph configuration file if you are encountering issues when starting your cluster. You may view Ceph log files under /var/log/ceph (the default location).
Tip
When debug output slows down your system, the latency can hide race conditions.
Logging is resource intensive. If you are encountering a problem in a specific area of your cluster, enable logging for that area of the cluster. For example, if your OSDs are running fine, but your metadata servers are not, you should start by enabling debug logging for the specific metadata server instance(s) giving you trouble. Enable logging for each subsystem as needed.
Important
Verbose logging can generate over 1GB of data per hour. If your OS disk reaches its capacity, the node will stop working.
If you enable or increase the rate of Ceph logging, ensure that you have sufficient disk space on your OS disk. See Accelerating Log Rotation for details on rotating log files. When your system is running well, remove unnecessary debugging settings to ensure your cluster runs optimally. Logging debug output messages is relatively slow, and a waste of resources when operating your cluster.
See Subsystem, Log and Debug Settings for details on available settings.
Runtime
If you would like to see the configuration settings at runtime, you must log in to a host with a running daemon and execute the following:
ceph daemon {daemon-name} config show | less
For example,:
ceph daemon osd.0 config show | less
To activate Ceph’s debugging output (i.e., dout()) at runtime, use the ceph tell command to inject arguments into the runtime configuration:
ceph tell {daemon-type}.{daemon id or *} config set {name} {value}
Replace {daemon-type} with one of osd, mon or mds. You may apply the runtime setting to all daemons of a particular type with *, or specify a specific daemon’s ID. For example, to increase debug logging for a ceph-osd daemon named osd.0, execute the following:
ceph tell osd.0 config set debug_osd 0/5
The ceph tell command goes through the monitors. If you cannot bind to the monitor, you can still make the change by logging into the host of the daemon whose configuration you’d like to change using ceph daemon. For example:
sudo ceph daemon osd.0 config set debug_osd 0/5
See Subsystem, Log and Debug Settings for details on available settings.
Boot Time
To activate Ceph’s debugging output (i.e., dout()) at boot time, you must add settings to your Ceph configuration file. Subsystems common to each daemon may be set under [global] in your configuration file. Subsystems for particular daemons are set under the daemon section in your configuration file (e.g., [mon], [osd], [mds]). For example:
[global]
debug ms = 1/5
[mon]
debug mon = 20
debug paxos = 1/5
debug auth = 2
[osd]
debug osd = 1/5
debug filestore = 1/5
debug journal = 1
debug monc = 5/20
[mds]
debug mds = 1
debug mds balancer = 1
See Subsystem, Log and Debug Settings for details.
Accelerating Log Rotation
If your OS disk is relatively full, you can accelerate log rotation by modifying the Ceph log rotation file at /etc/logrotate.d/ceph. Add a size setting after the rotation frequency to accelerate log rotation (via cronjob) if your logs exceed the size setting. For example, the default setting looks like this:
rotate 7
weekly
compress
sharedscripts
Modify it by adding a size setting.
rotate 7
weekly
size 500M
compress
sharedscripts
Then, start the crontab editor for your user space.
crontab -e
Finally, add an entry to check the etc/logrotate.d/ceph file.
30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
The preceding example checks the etc/logrotate.d/ceph file every 30 minutes.
https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/