Notice

This document is for a development version of Ceph.

Cephadm Operations

Watching Cephadm Log Messages

The cephadm orchestrator module writes logs to the cephadm cluster log channel. You can monitor Ceph’s activity in real time by reading the logs as they fill up. Run the following command to see the logs in real time:

ceph -W cephadm

By default, this command shows info-level events and above. To see debug-level messages as well as info-level events, run the following commands:

ceph config set mgr mgr/cephadm/log_to_cluster_level debug
ceph -W cephadm --watch-debug

Warning

The debug messages are very verbose!

You can see recent events by running the following command:

ceph log last cephadm

Monitors that cephadm deploys with Podman 2.1.0 or later send these events to journald by default. With Docker, or with an older Podman, the events go to the Monitor container’s stderr. If you enable logging to files (see Ceph Daemon Logs), the Monitors also write them to /var/log/ceph/<cluster-fsid>/ceph.cephadm.log on Monitor hosts.

Ceph Daemon Control

Starting and Stopping Daemons

You can stop, start, or restart a daemon with:

ceph orch daemon stop <daemon-name>
ceph orch daemon start <daemon-name>
ceph orch daemon restart <daemon-name>

A daemon name has the form <type>.<id>, for example osd.0. ceph orch ps lists the names.

You can also do the same for all daemons for a service with:

ceph orch stop <service-name>
ceph orch start <service-name>
ceph orch restart <service-name>

A service name is the name that ceph orch ls shows, for example mds.myfs.

Note

It is usually not safe to run ceph orch restart osd.myosdservice on a running cluster, as attention is not paid to CRUSH failure domains, and parallel OSD restarts may lead to temporary data unavailability or in rare cases even data loss.

Cluster-wide shutdown and startup

Cephadm provides commands to gracefully shut down and start up an entire Ceph cluster. This is useful for planned maintenance such as datacenter power-downs, hardware upgrades, or disaster recovery scenarios.

Warning

These commands will stop all Ceph services across all hosts in the cluster. Ensure all clients have unmounted CephFS filesystems and disconnected from RBD images and RGW before proceeding.

Shutting down the cluster

To shut down the entire cluster, run the following command from the admin host:

cephadm cluster-shutdown --fsid <fsid> --yes-i-really-mean-it

The --yes-i-really-mean-it flag is required to confirm the operation.

You can preview what would happen without making changes by using the dry-run mode:

cephadm cluster-shutdown --fsid <fsid> --dry-run

The shutdown process performs the following steps in order:

  1. Checks cluster health. The command stops with an error unless the cluster reports HEALTH_OK and all placement groups are active+clean. With --force it logs a warning and continues.

  2. Pauses the cephadm orchestrator to prevent daemon restarts

  3. Sets the OSD noout flag to prevent rebalancing on startup

  4. Fails all CephFS filesystems to prevent client I/O

  5. Stops client-facing daemons (NVMe-oF, iSCSI, NFS, SMB, RGW, MDS)

  6. Saves cluster state to a local file for later startup

  7. Stops all hosts sequentially (admin host last)

The cluster state is saved to /var/lib/ceph/<fsid>/cluster-shutdown-state.json on the admin host. This file is required for the startup process.

Note

During shutdown, the cephadm SSH private key is retrieved from the cluster and cached locally on the admin host. This is necessary because the cluster (and its key store) will be unavailable during startup. The cached key is stored with restricted permissions (0600) and is automatically removed after a successful cluster startup.

If the cluster health check fails but you still want to proceed, you can use the --force flag:

cephadm cluster-shutdown --fsid <fsid> --yes-i-really-mean-it --force

Danger

Using --force to shut down an unhealthy cluster is dangerous and may result in data loss. The cluster may not be able to recover properly on startup if there are degraded or undersized placement groups. Only use this option in emergency situations when you fully understand the risks and have verified that your data is safely replicated or backed up.

Starting the cluster

To start a previously shut-down cluster, run the following command from the admin host:

cephadm cluster-start --fsid <fsid>

You can preview what would happen without making changes:

cephadm cluster-start --fsid <fsid> --dry-run

The startup process performs the following steps:

  1. Loads the saved cluster state from the shutdown state file

  2. Starts all hosts in parallel (configurable)

  3. Waits for the cluster to become accessible

  4. Waits for all placement groups to become active+clean

  5. Unsets the OSD noout flag

  6. Sets CephFS filesystems back to joinable

  7. Resumes the cephadm orchestrator

  8. Cleans up the state file

By default, up to 5 hosts are started in parallel. You can adjust this with the --parallel option:

cephadm cluster-start --fsid <fsid> --parallel 10

Checking cluster status

To check whether a cluster is in a shutdown state or running normally:

cephadm cluster-status --fsid <fsid>

This command displays:

  • Whether a shutdown state file exists and when the shutdown occurred

  • The list of hosts and their status

  • Current cluster health (if the cluster is running)

Troubleshooting

Stale state file: If you believe the cluster is actually running but a state file exists from a previous incomplete operation, you can rename or remove the state file and retry:

mv /var/lib/ceph/<fsid>/cluster-shutdown-state.json /var/lib/ceph/<fsid>/cluster-shutdown-state.json.bak

Hosts fail to start: If some hosts fail to start, check:

  • SSH connectivity from the admin host to the failed hosts

  • That the ceph-<fsid>.target systemd unit exists on those hosts

  • System logs on the failed hosts (journalctl -u ceph-<fsid>.target)

Cluster not becoming healthy after startup: After startup, if PGs don’t become active+clean, check:

  • That all OSDs have started (ceph osd tree)

  • OSD logs for errors (journalctl -u ceph-<fsid>@osd.*)

  • That the noout flag has been cleared (ceph osd dump | grep flags)

Redeploying or reconfiguring a daemon

The container for a daemon can be stopped, recreated, and restarted with the redeploy command:

ceph orch daemon redeploy <name> [--image <image>]

A container image name can optionally be provided to force a particular image to be used (instead of the image specified by the container_image config value).

If only the configuration needs to be regenerated, you can also issue a reconfig command, which rewrites the daemon’s configuration files. For mon, mgr, osd, mds, rgw, rbd-mirror, cephfs-mirror, crash and ceph-exporter daemons this does not restart the daemon. For other daemon types, cephadm normally restarts the daemon after it rewrites the files.

ceph orch daemon reconfig <name>

Rotating a Daemon’s Authentication Key

All Ceph and gateway daemons in the cluster have a secret key that is used to connect to and authenticate with the cluster. The ceph orch daemon rotate-key command is disabled in releases beginning with Squid 19.2.6 and Tentacle 20.2.4. It did not work for some daemon types that have client keys (for example NFS and NVMe-oF) and could take those daemons down.

Ceph Daemon Logs

Logging to Journald

Ceph daemons traditionally write logs to /var/log/ceph. Ceph daemons log to journald by default and Ceph logs are captured by the container runtime environment. They are accessible via journalctl.

Note

When the container engine is Docker, or a Podman version without split cgroups support, Ceph daemons log to stderr and the container runtime captures the output.

Example of Logging to Journald

For example, to view the logs for the daemon mon.foo for a cluster with ID 5c5a50ae-272a-455d-99e9-32c6a013e694, the command would be something like:

journalctl -u ceph-5c5a50ae-272a-455d-99e9-32c6a013e694@mon.foo

This works well for normal operations when logging levels are low.

Logging to Files

You can also configure Ceph daemons to log to files instead of journald if you prefer logs to appear in files (as they do in clusters that are not deployed with cephadm). When Ceph logs to files, the logs appear in /var/log/ceph/<cluster-fsid>. If you choose to configure Ceph to log to files instead of journald, remember to configure Ceph so that it will not log to journald (the commands for this are covered below).

Enabling Logging to Files

To enable logging to files, run the following commands:

ceph config set global log_to_file true
ceph config set global mon_cluster_log_to_file true

Disabling Logging to Journald

If you choose to log to files, we recommend disabling logging to journald or else everything will be logged twice. Run the following commands to disable logging to stderr:

ceph config set global log_to_stderr false
ceph config set global mon_cluster_log_to_stderr false
ceph config set global log_to_journald false
ceph config set global mon_cluster_log_to_journald false

Note

You can change the default by passing --log-to-file during bootstrapping a new cluster.

Modifying the Log Retention Schedule

By default, cephadm sets up log rotation on each host to rotate these files. You can configure the logging retention schedule by modifying /etc/logrotate.d/ceph-<cluster-fsid>.

Per-node Cephadm Logs

The cephadm executable, either run directly by a user or by the cephadm orchestration module, may also generate logs. It does so independently of the other Ceph components running in containers. By default, this executable logs to the file /var/log/ceph/cephadm.log.

This logging destination is configurable and you may choose to log to the file, to the syslog/journal, or to both.

Setting a Cephadm Log Destination During Bootstrap

The cephadm command may be executed with the option --log-dest=file or with --log-dest=syslog or both. These options control where cephadm will store persistent logs for each invocation. When these options are specified for the cephadm bootstrap command the system will automatically record these settings for future invocations of cephadm by the cephadm orchestration module.

For example:

cephadm --log-dest=syslog bootstrap # ... other bootstrap arguments ...

If you want to manually specify exactly what log destination to use during bootstrap, independent from the --log-dest options, you may add a configuration key mgr/cephadm/cephadm_log_destination to the initial configuration file, under the [mgr] section. Valid values for the key are: file, syslog, and file,syslog.

For example:

cat >/tmp/bootstrap.conf <<EOF
[mgr]
mgr/cephadm/cephadm_log_destination = syslog
EOF
cephadm bootstrap --config /tmp/bootstrap.conf # ... other bootstrap arguments ...

Setting a Cephadm Log Destination on an Existing Cluster

An existing Ceph cluster can be configured to use a specific cephadm log destination by setting the mgr/cephadm/cephadm_log_destination configuration value to one of file, syslog, or file,syslog. This will cause the cephadm orchestration module to run cephadm so that logs go to /var/log/ceph/cephadm.log, the syslog/journal, or both, respectively.

To log to syslog:

ceph config set mgr mgr/cephadm/cephadm_log_destination syslog

To log to both the log file and syslog:

ceph config set mgr mgr/cephadm/cephadm_log_destination file,syslog

To log to the log file only:

ceph config set mgr mgr/cephadm/cephadm_log_destination file

Note

If you execute cephadm commands directly, such as cephadm shell, this option will not apply. To have cephadm log to locations other than the default log file when running cephadm commands directly use the --log-dest options described in the bootstrap section above.

Setting Cephadm Log Verbosity

The verbosity of cephadm’s persistent logging (/var/log/ceph/cephadm.log and syslog, when enabled) can be controlled separately from its terminal output. Valid levels are info, debug, error, and warning. The default is debug.

When the cephadm orchestration module runs cephadm on cluster hosts, it uses the mgr/cephadm/cephadm_binary_logging_level configuration value:

ceph config set mgr mgr/cephadm/cephadm_binary_logging_level info

For manual invocations, pass --logging-level on the command line:

cephadm --logging-level info check-host

Note

The logging level applies only to persistent log destinations (the log file and syslog). Console output during bootstrap and other interactive use is unchanged. When you run cephadm directly, the mgr configuration does not apply; use --logging-level as shown above.

Data Location

Cephadm stores daemon data and logs in different locations than clusters that are not deployed with cephadm:

  • /var/log/ceph/<cluster-fsid> contains all cluster logs. By default, Ceph daemons log to journald (to stderr when the container engine is Docker or a Podman version older than 2.1.0), so these logs will not exist unless you have enabled logging to files as described in Ceph Daemon Logs.

  • /var/lib/ceph/<cluster-fsid> contains all cluster daemon data (besides logs).

  • /var/lib/ceph/<cluster-fsid>/<daemon-name> contains all data for an individual daemon.

  • /var/lib/ceph/<cluster-fsid>/crash contains crash reports for the cluster.

  • /var/lib/ceph/<cluster-fsid>/removed contains old daemon data directories for stateful daemons (e.g., Monitor, Prometheus) that have been removed by cephadm.

Disk Usage

Because a few Ceph daemons (notably, the Monitors and Prometheus) store a large amount of data in /var/lib/ceph , we recommend moving this directory to its own disk, partition, or logical volume so that it does not fill up the root file system.

Health Checks

The cephadm module provides additional health checks to supplement the default health checks provided by the cluster. These additional health checks fall into two categories:

  • cephadm operations: Health checks in this category are always executed when the cephadm module is active.

  • cluster configuration: These health checks are optional, and focus on the configuration of the hosts in the cluster.

Cephadm Operation Checks

CEPHADM_PAUSED

This indicates that cephadm background work has been paused with ceph orch pause. While paused, the background loop does not apply service specifications, run scheduled daemon actions, process OSD removals or continue an upgrade. It still checks hosts, refreshes daemon and device information, writes the client keyrings and ceph.conf files that it manages, and adjusts osd_memory_target where memory autotuning is enabled.

Resume cephadm work by running the following command:

ceph orch resume

CEPHADM_STRAY_HOST

This indicates that one or more hosts have Ceph daemons that are running, but are not registered as hosts managed by cephadm. This means that those services cannot currently be managed by cephadm (e.g., restarted, upgraded, included in ceph orch ps).

  • You can manage the host(s) by running the following command:

    ceph orch host add <hostname>

    Note

    You might need to configure SSH access to the remote host before this will work.

    See Adding Hosts for more information.

  • See Fully Qualified Domain Names vs Bare Host Names for more information about host names and domain names.

  • Alternatively, you can manually connect to the host and ensure that services on that host are removed or migrated to a host that is managed by cephadm.

  • This warning can be disabled entirely by running the following command:

    ceph config set mgr mgr/cephadm/warn_on_stray_hosts false

CEPHADM_STRAY_DAEMON

One or more Ceph daemons are running but are not managed by cephadm. This may be because they were deployed using a different tool, or because they were started manually. Those services cannot currently be managed by cephadm (e.g., restarted, upgraded, or included in ceph orch ps).

  • If the daemon is a stateful one (Monitor or OSD), it should be adopted by cephadm; see Converting an Existing Cluster to Cephadm. For stateless daemons, it is usually easiest to provision a new daemon with the ceph orch apply command and then stop the unmanaged daemon.

  • If the stray daemon(s) are running on hosts not managed by cephadm, you can manage the host(s) by running the following command:

    ceph orch host add <hostname>

    Note

    You might need to configure SSH access to the remote host before this will work.

    See Adding Hosts for more information.

  • See Fully Qualified Domain Names vs Bare Host Names for more information about host names and domain names.

  • This warning can be disabled entirely by running the following command:

    ceph config set mgr mgr/cephadm/warn_on_stray_daemons false

CEPHADM_HOST_CHECK_FAILED

One or more hosts have failed the basic cephadm host check, which verifies that (1) the host is reachable and cephadm can be executed there, and (2) that the host satisfies basic prerequisites, like a working container runtime (Podman or Docker) and working time synchronization. If this test fails, cephadm will not be able to manage services on that host.

You can manually run this check by running the following command:

ceph cephadm check-host <hostname>

You can remove a broken host from management by running the following command:

ceph orch host rm <hostname>

See Removing Hosts for more information.

You can disable this health warning by running the following command:

ceph config set mgr mgr/cephadm/warn_on_failed_host_check false

Cluster Configuration Checks

Cephadm periodically scans each host in the cluster in order to understand the state of the OS, disks, network interfaces etc. This information can then be analyzed for consistency across the hosts in the cluster to identify any configuration anomalies.

Enabling Cluster Configuration Checks

These configuration checks are an optional feature, and are enabled by running the following command:

ceph config set mgr mgr/cephadm/config_checks_enabled true

States Returned by Cluster Configuration Checks

Configuration checks are run after each host scan. Their outcome is reported through the CEPHADM_CHECK_* health checks described below, which you can see with ceph health detail.

Managing Configuration Checks (Subcommands)

The configuration checks themselves are managed through several cephadm subcommands.

To determine whether the configuration checks are enabled, run the following command:

ceph cephadm config-check status

This command returns the status of the configuration checker as either “Enabled” or “Disabled”.

To list all the configuration checks and their current states, run the following command:

ceph cephadm config-check ls

NAME             HEALTHCHECK                      STATUS   DESCRIPTION
kernel_security  CEPHADM_CHECK_KERNEL_LSM         enabled  checks SELINUX/Apparmor profiles are consistent across cluster hosts
os_subscription  CEPHADM_CHECK_SUBSCRIPTION       enabled  checks subscription states are consistent for all cluster hosts
public_network   CEPHADM_CHECK_PUBLIC_MEMBERSHIP  enabled  check that all hosts have a NIC on the Ceph public_network
osd_mtu_size     CEPHADM_CHECK_MTU                enabled  check that OSD hosts share a common MTU setting
osd_linkspeed    CEPHADM_CHECK_LINKSPEED          enabled  check that OSD hosts share a common linkspeed
network_missing  CEPHADM_CHECK_NETWORK_MISSING    enabled  checks that the cluster/public networks defined exist on the Ceph hosts
ceph_release     CEPHADM_CHECK_CEPH_RELEASE       enabled  check for Ceph version consistency - ceph daemons should be on the same release (unless upgrade is active)
kernel_version   CEPHADM_CHECK_KERNEL_VERSION     enabled  checks that the MAJ.MIN of the kernel on Ceph hosts is consistent

The name of each configuration check can be used to enable or disable a specific check by running a command of the following form:

ceph cephadm config-check [enable|disable] <name>

For example:

ceph cephadm config-check disable kernel_security

CEPHADM_CHECK_KERNEL_LSM

Each host within the cluster is expected to operate within the same Linux Security Module (LSM) state. For example, if the majority of the hosts are running with SELinux in enforcing mode, any host not running in this mode is flagged as an anomaly and a health check warning is raised.

CEPHADM_CHECK_SUBSCRIPTION

This check relates to the status of OS vendor subscription. This check is performed only for hosts using RHEL and helps to confirm that all hosts are covered by an active subscription, which ensures that patches and updates are available.

CEPHADM_CHECK_PUBLIC_MEMBERSHIP

All members of the cluster should have a network interface configured on at least one of the public network subnets. Hosts that are not on the public network will rely on routing, which may affect performance.

CEPHADM_CHECK_MTU

The MTU of the network interfaces on OSD hosts can be a key factor in consistent performance. This check examines hosts that are running OSD services to ensure that the MTU is configured consistently within the cluster. This is determined by determining the MTU setting that the majority of hosts is using. Any anomalies result in a health check warning.

CEPHADM_CHECK_LINKSPEED

This check is similar to the MTU check. Link speed consistency is a factor in consistent cluster performance, as is the MTU of the OSD node network interfaces. This check determines the link speed shared by the majority of OSD hosts, and a health check warning is raised for any hosts that are set at a lower link speed rate.

CEPHADM_CHECK_NETWORK_MISSING

The public_network and cluster_network settings support subnet definitions for IPv4 and IPv6. If these settings are not found on any host in the cluster, a health check warning is raised.

CEPHADM_CHECK_CEPH_RELEASE

Under normal operations, the Ceph cluster runs daemons that are of the same Ceph release (for example, Tentacle). This check determines the active release for each daemon, and reports any anomalies as a health check warning. This check is bypassed if an upgrade is in process.

CEPHADM_CHECK_KERNEL_VERSION

The OS kernel version (maj.min) is checked for consistency across hosts. The kernel version of the majority of the hosts is used as the basis for identifying anomalies.

Management of Client Keyrings and Configs

Cephadm can distribute copies of the ceph.conf file and client keyring files to hosts. In addition to the default location /etc/ceph, cephadm also stores config and keyring files in the /var/lib/ceph/<fsid>/config directory. It is usually a good idea to store a copy of the config and client.admin keyring on any host used to administer the cluster via the CLI. By default, cephadm does this for any nodes that have the _admin label (which normally includes the bootstrap host).

Note

Clients such as the ceph command still read the files in /etc/ceph. Ceph daemons that cephadm deploys do not: each daemon’s container gets the daemon’s own file, /var/lib/ceph/<fsid>/<daemon-name>/config, mounted as /etc/ceph/ceph.conf. The /var/lib/ceph/<fsid>/config directory is used by cephadm only. Having this config directory under the fsid helps cephadm to load the configuration associated with the cluster.

When a client keyring is placed under management, cephadm will:

  • build a list of target hosts based on the specified placement spec (see Daemon Placement)

  • store a copy of the /etc/ceph/ceph.conf file on the specified host(s)

  • store a copy of the ceph.conf file at /var/lib/ceph/<fsid>/config/ceph.conf on the specified host(s)

  • store a copy of the keyring file at /var/lib/ceph/<fsid>/config/ceph.<entity>.keyring on the specified host(s)

  • store a copy of the keyring file on the specified host(s)

  • update the ceph.conf file as needed (e.g., due to a change in the cluster Monitors)

  • update the keyring file if the entity’s key is changed (e.g., via ceph auth ... commands)

  • ensure that the keyring file has the specified ownership and specified mode

  • remove the keyring file when client keyring management is disabled

  • remove the keyring file from old hosts if the keyring placement spec is updated (as needed)

Listing Client Keyrings

To see the list of client keyrings are currently under management, run the following command:

ceph orch client-keyring ls

Enabling Management of a Keyring File

To enable management of a keyring file, run a command of the following form:

ceph orch client-keyring set <entity> <placement> [--mode=<mode>] [--owner=<uid>:<gid>] [--no-ceph-conf]

  • The keyring is written to /etc/ceph/ceph.<entity>.keyring (for example, /etc/ceph/ceph.client.rbd.keyring), which is where Ceph looks by default. The path cannot be changed. An existing file at that path is replaced.

  • The entity name must start with client..

  • A placement of * (all hosts) is common.

  • The mode defaults to 0600 and ownership to 0:0 (user root, group root).

For example, to create a client.rbd key and deploy it to hosts with the rbd-client label and make it group readable by uid/gid 107 (qemu), run the following commands:

ceph auth get-or-create-key client.rbd mon 'profile rbd' mgr 'profile rbd' osd 'profile rbd pool=my_rbd_pool'
ceph orch client-keyring set client.rbd label:rbd-client --owner 107:107 --mode 640

The resulting keyring file is:

-rw-r-----. 1 qemu qemu 156 Apr 21 08:47 /etc/ceph/ceph.client.rbd.keyring

By default, cephadm will also manage /etc/ceph/ceph.conf on hosts where it writes the keyrings. This feature can be suppressed by passing --no-ceph-conf when setting the keyring.

ceph orch client-keyring set client.foo label:foo 0:0 --no-ceph-conf

Disabling Management of a Keyring File

To disable management of a keyring file, run a command of the following form:

ceph orch client-keyring rm <entity>

Note

This deletes any keyring files for this entity that were previously written to cluster nodes.

Management of /etc/ceph/ceph.conf

Distributing ceph.conf to Hosts that Have no Keyrings

It might be useful to distribute ceph.conf files to hosts without an associated client keyring file. By default, cephadm deploys only a ceph.conf file to hosts where a client keyring is also distributed (see Management of Client Keyrings and Configs above). To write config files to hosts without client keyrings, run the following command:

ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true

Using Placement Specs to Specify which Hosts Get ceph.conf

By default, the configs are written to all hosts (i.e., those listed by ceph orch host ls). To specify which hosts get a ceph.conf, run a command of the following form:

ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf_hosts <placement spec>

For example, to distribute configs to hosts with the bare_config label, run the following command:

ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf_hosts label:bare_config

See Daemon Placement for more information about placement specs.

Limiting Passwordless Sudo Access

By default, the cephadm install guide recommends enabling passwordless sudo for the cephadm user. This option is the most flexible and future-proof but may not be preferred in all environments. An administrator can restrict sudo to only running an exact list of commands without password access. Note that this list may change between Ceph versions and administrators choosing this option should read the release notes and review this list in the destination version of the Ceph documentation. If the list differs one must extend the list of passwordless sudo commands prior to upgrade.

Commands requiring passwordless sudo support:

  • chmod

  • chown

  • ls

  • mkdir

  • mv

  • rm

  • sysctl

  • touch

  • true

  • which (see note)

  • /usr/bin/cephadm or python executable (see note)

Note

Typically cephadm will execute which to determine which python3 command is available and then use the command returned by which in subsequent commands. Before configuring sudo run which python3 to determine what python command to add to the sudo configuration. In some rare configurations /usr/bin/cephadm will be used instead.

Note

This list is correct for Umbrella and earlier releases. In later releases, the Manager no longer runs which, python3, rm, sysctl or true over SSH. It runs the cephadm binary directly, as sudo /var/lib/ceph/<fsid>/cephadm.<sha256 of the binary>, and uses ls, mkdir, touch, chown, chmod and mv to install that binary. The example sudoers line below does not allow that path.

Configuring the sudoers file can be performed using a tool like visudo and adding or replacing a user configuration line such as the following:

# assuming the cephadm user is named "ceph"
ceph ALL=(ALL) NOPASSWD:/usr/bin/chmod,/usr/bin/chown,/usr/bin/ls,/usr/bin/mkdir,/usr/bin/mv,/usr/bin/rm,/usr/sbin/sysctl,/usr/bin/touch,/usr/bin/true,/usr/bin/which,/usr/bin/cephadm,/usr/bin/python3

Purging a cluster

Danger

THIS OPERATION WILL DESTROY ALL DATA STORED IN THIS CLUSTER!

In order to destroy a cluster and delete all data stored in this cluster, disable cephadm to stop all orchestration operations (so we avoid deploying new daemons):

ceph mgr module disable cephadm

Then verify the FSID of the cluster:

ceph fsid

Purge Ceph daemons from all hosts in the cluster by running the following command on each host:

cephadm rm-cluster --force --zap-osds --fsid <fsid>

Replacing a Device

The ceph orch device replace command automates the process of replacing the underlying device of an OSD. Previously, this process required manual intervention at various stages. With this new command, all necessary operations are performed automatically, streamlining the replacement process and improving the overall user experience.

Note

This supports only LVM-based deployed OSD(s).

ceph orch device replace <host> <device-path>

If the device being replaced is shared by multiple OSDs (e.g., DB/WAL device shared by multiple OSDs), the orchestrator will warn you:

ceph orch device replace osd-1 /dev/vdd
Error EINVAL: /dev/vdd is a shared device.
Replacing /dev/vdd implies destroying OSD(s): ['0', '1'].
Please, *be very careful*, this can be a very dangerous operation.
If you know what you are doing, pass --yes-i-really-mean-it

If you know what you are doing, you can go ahead and pass --yes-i-really-mean-it:

ceph orch device replace osd-1 /dev/vdd --yes-i-really-mean-it
Scheduled to destroy osds: ['0', '1'] and mark /dev/vdd as being replaced.

cephadm will execute ceph-volume to zap and destroy all related devices and mark the corresponding OSD as destroyed so the different OSD(s) ID(s) will be preserved:

ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS     REWEIGHT  PRI-AFF
-1         0.97659  root default
-3         0.97659      host osd-1
 0    hdd  0.29300          osd.0     destroyed   1.00000  1.00000
 1    hdd  0.29300          osd.1     destroyed   1.00000  1.00000
 2    hdd  0.19530          osd.2            up   1.00000  1.00000
 3    hdd  0.19530          osd.3            up   1.00000  1.00000

The device being replaced is finally seen as being replaced preventing cephadm from redeploying the OSDs too fast:

ceph orch device ls
HOST     PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REFRESHED  REJECT REASONS
osd-1  /dev/vdb  hdd               200G  Yes        13s ago
osd-1  /dev/vdc  hdd               200G  Yes        13s ago
osd-1  /dev/vdd  hdd               200G  No         13s ago    Is being replaced
osd-1  /dev/vde  hdd               200G  No         13s ago    Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
osd-1  /dev/vdf  hdd               200G  No         13s ago    Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected

If for any reason you need to clear the ‘device replace header’ on a device, run a command of the following form:

ceph orch device replace <host> <device> --clear

For example:

ceph orch device replace devel-1 /dev/vdk --clear
Replacement header cleared on /dev/vdk

Cephadm will then redeploy the OSD service spec within a few minutes (unless the service is set to unmanaged).

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.