Notice

This document is for a development version of Ceph.

Upgrading Ceph

Cephadm can safely upgrade Ceph from one point release to the next. For example, you can upgrade from v15.2.0 (the first Octopus release) to the next point release, v15.2.1.

The automated upgrade process follows Ceph best practices. For example:

  • The upgrade order starts with managers, monitors, then other daemons.

  • Each daemon is restarted only after Ceph indicates that the cluster will remain available.

Note

The Ceph cluster health status is likely to switch to HEALTH_WARN during the upgrade.

Note

If a host is offline or in maintenance mode, ceph orch upgrade start refuses to start the upgrade. If a host goes offline during the upgrade, cephadm pauses the upgrade and raises the UPGRADE_OFFLINE_HOST health alert. The upgrade stays paused after the host returns. To continue, run ceph orch upgrade resume.

Note

When the PG autoscaler mode for any pool is set to on, we recommend disabling the autoscaler for the duration of the upgrade. This is so that PG splitting or merging in the middle of an upgrade does not unduly delay upgrade progress. In a very large cluster this could easily increase the time to complete by a day or more, especially if the upgrade happens to change PG autoscaler behavior by e.g. changing the default value of mon_target_pg_per_osd.

ceph osd pool set noautoscale
# Perform the upgrade
ceph osd pool unset noautoscale

When pausing autoscaler activity in this fashion, the existing values for each pool’s mode, off, on, or warn, are expected to remain. If the new release changes the above target value, there may be splitting or merging of PGs when unsetting after the upgrade.

Cephadm will automatically pause and resume the PG autoscaler activity during upgrade unless opted-in by setting:

ceph config set mgr mgr/cephadm/pg_autoscale_during_upgrade true

To view the current value:

ceph config get mgr mgr/cephadm/pg_autoscale_during_upgrade

If autoscaling was already off before the upgrade, it is still off after the upgrade. Setting pg_autoscale_during_upgrade to true tells cephadm to leave the autoscaler as it is. It does not turn autoscaling on.

Starting the Upgrade

Note

Note

Staggered Upgrade of the Monitors and Managers may be necessary to use the below CephFS upgrade feature.

Cephadm by default reduces max_mds to 1. This can be disruptive for large-scale CephFS deployments because the cluster cannot quickly reduce active MDS(s) to 1 and a single active MDS cannot easily handle the load of all clients even for a short time. Therefore, to upgrade MDS(s) without reducing max_mds, the fail_fs option can be set to true (default value is false) prior to initiating the upgrade:

ceph config set mgr mgr/orchestrator/fail_fs true

This would:

  1. Fail CephFS filesystems, bringing active MDS daemon(s) to up:standby state.

  2. Upgrade MDS daemons safely.

  3. Bring CephFS filesystems back up, bringing the state of active MDS daemon(s) from up:standby to up:active.

Before you use cephadm to upgrade Ceph, verify that all hosts are currently online and that your cluster is healthy by running the following command:

ceph -s

To upgrade to a specific release, run a command of the following form:

ceph orch upgrade start --ceph-version <version>

For example, to upgrade to v20.2.3, run the following command:

ceph orch upgrade start --ceph-version 20.2.3

Note

Since version v16.2.6 the Docker Hub registry is no longer used, so if you use Docker you have to point it to the image in the quay.io registry:

ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.3

CRUSH bucket-scoped OSD upgrades (osd ok-to-upgrade)

When performing OSD upgrades as part of a staggered Ceph upgrade, one may constrain the set of OSDs on which cephadm will operate. This ability is available in the Ceph Umbrella and later releases. As cephadm progresses through the specified CRUSH bucket, it asks the Monitors which OSDs may safely move to the target release. This process uses the ceph osd ok-to-upgrade command.

Requirements:

  • For OSD-only upgrades, pass --crush_bucket_type and --daemon-types osd. Supported bucket types today are host, rack, and chassis.

  • --crush_bucket_name is optional:

    • Type and name both given: cephadm upgrades OSDs in that one named bucket only.

    • Type given, name omitted: cephadm discovers every CRUSH bucket of that type (via osd crush tree), then upgrades them one at a time, in CRUSH tree order, automatically advancing to the next bucket once the current one finishes. No further operator action is needed between buckets; see Auto-iterating across all buckets of a type below.

  • The Monitor’s osd ok-to-upgrade expects the target short Ceph version (same shape as ceph_version_short in ceph osd metadata).

  • If the Monitors indicate to cephadm that no OSDs in the current CRUSH bucket are okay to upgrade, cephadm will log details and then retry the operation on that same bucket; it does not skip ahead to the next bucket.

  • If the bucket parameters for a ceph osd ok-to-upgrade upgrade are not provided, cephadm will fall back to the default ceph osd ok-to-stop gate for OSD upgrades.

  • Bucket-scope upgrades apply only to OSDs. CRUSH buckets do not influence upgrades of other daemon types, for example Monitors, Managers, and MDSes.

  • The set of buckets (and their membership) is fixed when the upgrade starts. Do not add, remove, or rename CRUSH buckets of the type being upgraded while an upgrade is in progress.

Single-bucket example (type and name):

ceph orch upgrade start --image quay.io/ceph/ceph:v21.2.1 \
  --daemon-types osd \
  --crush_bucket_type rack --crush_bucket_name rack-a

When performing OSD upgrades within this failure domain, cephadm calls ceph osd ok-to-upgrade with the specified bucket name, and max set to mgr/cephadm/max_parallel_osd_upgrades

Auto-iterating across all buckets of a type

If an operator provides --crush_bucket_type without --crush_bucket_name, cephadm looks up every bucket of that type (for example every rack, or every host) from the current CRUSH map and upgrades them one at a time instead of requiring a separate ceph orch upgrade start invocation per bucket:

ceph orch upgrade start --image quay.io/ceph/ceph:v21.2.1 \
  --daemon-types osd \
  --crush_bucket_type rack

For each bucket in turn, cephadm:

  1. Calls osd ok-to-upgrade for the current bucket and upgrades the OSDs that the Monitors report as safe to move, the same way as the single-bucket case above.

  2. Once every OSD in the current bucket is on the target image, advances to the next bucket in the (frozen) list and repeats.

  3. After the last bucket finishes, completes the upgrade the same way a normal OSD upgrade does (for example setting require_osd_release once every OSD in the cluster is on the target version).

The list of buckets is captured once, when the upgrade starts, so later changes to the CRUSH map are not picked up mid-upgrade. This also means the upgrade resumes against the same bucket list after a Manager failover, continuing from the bucket that was in progress rather than restarting.

ceph orch upgrade status reports which bucket is currently being upgraded along with how many buckets in the list are already complete and how many remain, for example:

Warning

Do not change the cluster’s topology during an OSD upgrade phase. This includes the name or type of any CRUSH bucket.

Limiting the Gateway Upgrade Batch Size

The number of gateway daemons (rgw, iscsi, nfs, nvmeof and smb) that cephadm upgrades in one pass is controlled by mgr/cephadm/max_gateway_upgrades_per_pass. The remaining daemons are upgraded by the subsequent passes, so the upgrade still completes without operator intervention. Because cephadm returns to the serve loop between passes, a lower value keeps the serve loop more responsive while an upgrade is in progress, so other work such as deploying new daemons and refreshing host metadata is delayed less. Raise the value to upgrade large numbers of gateway daemons in fewer passes:

ceph config set mgr mgr/cephadm/max_gateway_upgrades_per_pass 8

For rgw, iscsi, nvmeof and smb the value is a plain per-daemon cap. A pass upgrades at most that many daemons, and a service with more daemons than the limit is split across passes, so the service keeps serving clients while it is upgraded.

nfs is the exception: restarting one NFS daemon sends the rest of its service into a grace period, so a service that is split across passes pays for one grace period per pass instead of one for the whole service. NFS daemons are therefore grouped by service and whole services are added to a pass.

Setting the option to 0 disables the limit and restores the previous behavior of upgrading all daemons of the type in one pass.

Monitoring the Upgrade

Determine (1) whether an upgrade is in progress and (2) which container image the cluster is upgrading to by running the following command:

ceph orch upgrade status

Watching the Progress Bar During a Ceph Upgrade

During the upgrade, a progress bar is visible in the ceph status output. It looks like this:

ceph -s

[...]
  progress:
    Upgrade to 20.2.3 (20m)
      [=======.....................] (remaining: 60m)

Watching the Cephadm Log During an Upgrade

Watch the cephadm log by running the following command:

ceph -W cephadm

Canceling an Upgrade

You can stop the upgrade process at any time by running the following command:

ceph orch upgrade stop

Downgrading

Ceph does not support downgrading a cluster to an earlier major release. Cephadm refuses to start such a downgrade, and it also refuses to move a cluster from a stable release to a release candidate or a development release. When cephadm has upgraded every OSD to a new major release, it raises the require_osd_release flag in the OSD map to that release, so no manual step is needed. This is a one-way step: the flag can’t be lowered again, and OSDs from an earlier release can no longer start.

Cephadm does not prevent a downgrade to an earlier point release of the same major release, for example from 20.2.1 to 20.2.0, but such downgrades are not supported. They may work in some situations and fail in others. They are not recommended, and they are done only at the administrator’s risk. If you are tempted to downgrade to an earlier point release, consider instead a configuration workaround, or an upgrade to a later point release or a later major release. You can also build a local custom container image of the release you are running with the code fixes you need, for example by cherry-picking commits from a later release, and upgrade to that image as described in Using Customized Container Images.

To return to an earlier major release, deploy a new cluster at that release and migrate or restore the data into it.

Post-upgrade Actions

In case the new version is based on cephadm, once done with the upgrade the user has to update the cephadm package (or ceph-common package in case the user doesn’t use cephadm shell) to a version compatible with the new version.

Potential Problems

Error ENOENT: Module not found

The message Error ENOENT: Module not found appears in response to the command ceph orch upgrade status if the orchestrator has crashed:

ceph orch upgrade status
Error ENOENT: Module not found

This is possibly caused by invalid JSON in a mgr config-key. One known cause on releases before the fix: the OSD removal queue stored under the mgr/cephadm/osd_remove_queue config-key could contain a field (original_weight) that the cephadm module was unable to load back, which crashed the module whenever the Manager restarted. The workaround was to edit the stored JSON to remove the offending field and then restart ceph-mgr.

UPGRADE_NO_STANDBY_MGR

This alert (UPGRADE_NO_STANDBY_MGR) means that Ceph does not detect an active standby Manager daemon. In order to proceed with the upgrade, Ceph requires an active standby Manager daemon (which you can think of in this context as “a second manager”).

You can ensure that Cephadm is configured to run two (or more) Managers by running the following command:

ceph orch apply mgr 2  # or more

You can check the status of existing Manager daemons by running the following command:

ceph orch ps --daemon-type mgr

If an existing Manager daemon has stopped, you can try to restart it by running the following command:

ceph orch daemon restart <name>

UPGRADE_FAILED_PULL

This alert (UPGRADE_FAILED_PULL) means that Ceph was unable to pull the container image for the target version. This can happen if you specify a version or container image that does not exist (e.g. quay.io/ceph/ceph:v20.2.99), or if the container registry cannot be reached by one or more hosts in the cluster.

To cancel the existing upgrade and to specify a different target version, run the following commands:

ceph orch upgrade stop
ceph orch upgrade start --ceph-version <version>

UPGRADE_INCOMPATIBLE_HOST_CPU

This alert (UPGRADE_INCOMPATIBLE_HOST_CPU) means that one or more hosts in the cluster have a CPU that does not support the x86-64 microarchitecture level that the target release’s official builds are compiled for. For example, official builds of Umbrella (21.x) and later require CPUs that support x86-64-v3. Running the target release’s binaries on the listed hosts would crash with an illegal instruction error (SIGILL), so cephadm refuses to proceed.

A host’s detected microarchitecture level can be checked by running the following command on the host itself:

cephadm gather-facts | grep cpu_isa_level

To proceed with the upgrade, replace or remove the listed hosts. If you are using custom-built container images compiled for an older CPU generation, you can disable this check by running the following command:

ceph config set mgr mgr/cephadm/upgrade_cpu_isa_check false

Using Customized Container Images

For most users, upgrading requires nothing more complicated than specifying the Ceph version to which to upgrade. In such cases, cephadm locates the specific Ceph container image to use by combining the container_image_base configuration option (default: quay.io/ceph/ceph) with a tag of vX.Y.Z.

But it is possible to upgrade to an arbitrary container image, if that’s what you need. For example, the following command upgrades to a development build:

ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:recent-git-branch-name

For more information about available container images, see Ceph Container Images.

Staggered Upgrade

Some users may prefer to upgrade components in phases rather than all at once. The upgrade command, starting in 16.2.11 and 17.2.1 allows parameters to limit which daemons are upgraded by a single upgrade command. The options include daemon_types, services, hosts and limit.

  • daemon_types takes a comma-separated list of daemon types and will only upgrade daemons of those types.

  • services will only upgrade daemons belonging to those services.

    It is mutually exclusive with daemon_types and only takes services of one type at a time (e.g. can’t provide an OSD and RGW service at the same time).

  • hosts parameter follows the same format as the command line options for Daemon Placement.

    It can be combined with daemon_types or services or provided on its own.

  • limit takes an integer > 0 and provides a numerical limit on the number of daemons cephadm will upgrade.

    It can be combined with any of the other parameters.

For example, if you specify to upgrade daemons of type osd on host host1 with limit set to 3, cephadm will upgrade (up to) 3 OSD daemons on host1.

Example: specifying daemon types and hosts:

ceph orch upgrade start --image <image-name> --daemon-types mgr,mon --hosts host1,host2

Example: specifying services and using limit:

ceph orch upgrade start --image <image-name> --services rgw.example1,rgw.example2 --limit 2

Note

Cephadm strictly enforces an order to the upgrade of daemons that is still present in staggered upgrade scenarios. The current upgrade ordering is:

  • mgr

  • mon

  • crash

  • osd

  • mds

  • rgw

  • rbd-mirror

  • cephfs-mirror

  • ceph-exporter

  • iscsi

  • nfs

  • nvmeof

  • smb

  • node-exporter

  • prometheus

  • alertmanager

  • grafana

  • loki

  • promtail

  • alloy

  • pushgateway

  • mgmt-gateway

  • oauth2-proxy

If you specify parameters that would upgrade daemons out of order, the upgrade command fails with Cannot start upgrade. and names the daemons that you must upgrade first.

Note

Upgrade commands with limiting parameters will validate the options before beginning the upgrade, which may require pulling the new container image. Do not be surprised if the upgrade start command takes a while to return when limiting parameters are provided.

Note

In staggered upgrade scenarios (when a limiting parameter is provided) monitoring stack daemons including Prometheus and Node Exporter are refreshed after the Manager daemons have been upgraded. Do not be surprised if Manager upgrades thus take longer than expected. Note that the versions of monitoring stack daemons may not change between Ceph releases, in which case they are only redeployed.

Upgrading to a Version that Supports Staggered Upgrade from One that Doesn’t

While upgrading from a version that already supports staggered upgrades, the process simply requires providing the necessary arguments. However, if you wish to upgrade to a version that supports staggered upgrade from one that does not, there is a workaround. It requires first manually upgrading the Manager daemons and then passing the limiting parameters as usual.

Warning

Make sure you have multiple running Manager daemons before attempting this procedure.

To start with, determine which Manager is your active one and which are standby. This can be done in a variety of ways such as looking at the ceph -s output. Then, manually upgrade each standby Manager daemon with:

ceph orch daemon redeploy mgr.example1.abcdef --image <new-image-name>

Note

If you are on a very early version of cephadm (early Octopus), the orch daemon redeploy command may not have the --image flag. In that case, you must manually set the Manager container image and then redeploy the Manager:

ceph config set mgr container_image <new-image-name>
ceph orch daemon redeploy mgr.example1.abcdef

At this point, a Manager failover should allow us to have the active Manager be the one running the new version.

ceph mgr fail

Verify the active Manager is now the one running the new version. To complete the Manager upgrade:

ceph orch upgrade start --image <new-image-name> --daemon-types mgr

You should now have all your Manager daemons on the new version and be able to specify the limiting parameters for the rest of the upgrade.

Updating a non-Ceph Image Service with a Custom Image

To update a non-Ceph image service, run a command of the following form:

ceph orch update service <service_type> <image>

For example:

ceph orch update service prometheus quay.io/prometheus/prometheus:v3.6.0

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.