Upgrading Ceph

Each release of Ceph may have additional steps. Refer to the release notes document of your release to identify release-specific procedures for your cluster before using the upgrade procedures.

Summary

You can upgrade daemons in your Ceph cluster while the cluster is online and in service! Certain types of daemons depend upon others. For example, Ceph Metadata Servers and Ceph Object Gateways depend upon Ceph Monitors and Ceph OSD Daemons. We recommend upgrading in this order:

  1. Ceph Deploy
  2. Ceph Monitors
  3. Ceph OSD Daemons
  4. Ceph Metadata Servers
  5. Ceph Object Gateways

As a general rule, we recommend upgrading all the daemons of a specific type (e.g., all ceph-mon daemons, all ceph-osd daemons, etc.) to ensure that they are all on the same release. We also recommend that you upgrade all the daemons in your cluster before you try to exercise new functionality in a release.

The Upgrade Procedures are relatively simple, but do look at the release notes document of your release before upgrading. The basic process involves three steps:

  1. Use ceph-deploy on your admin node to upgrade the packages for multiple hosts (using the ceph-deploy install command), or login to each host and upgrade the Ceph package using your distro’s package manager. For example, when Upgrading Monitors, the ceph-deploy syntax might look like this:

    ceph-deploy install --release {release-name} ceph-node1[ ceph-node2]
    ceph-deploy install --release firefly mon1 mon2 mon3
    

    Note: The ceph-deploy install command will upgrade the packages in the specified node(s) from the old release to the release you specify. There is no ceph-deploy upgrade command.

  2. Login in to each Ceph node and restart each Ceph daemon. See Operating a Cluster for details.

  3. Ensure your cluster is healthy. See Monitoring a Cluster for details.

Important

Once you upgrade a daemon, you cannot downgrade it.

Ceph Deploy

Before upgrading Ceph daemons, upgrade the ceph-deploy tool.

sudo pip install -U ceph-deploy

Or:

sudo apt-get install ceph-deploy

Or:

sudo yum install ceph-deploy python-pushy

Upgrade Procedures

The following sections describe the upgrade process.

Important

Each release of Ceph may have some additional steps. Refer to the release notes document of your release for details BEFORE you begin upgrading daemons.

Upgrading Monitors

To upgrade monitors, perform the following steps:

  1. Upgrade the Ceph package for each daemon instance.

    You may use ceph-deploy to address all monitor nodes at once. For example:

    ceph-deploy install --release {release-name} ceph-node1[ ceph-node2]
    ceph-deploy install --release hammer mon1 mon2 mon3
    

    You may also use the package manager for your Linux distribution on each individual node. To upgrade packages manually on each Debian/Ubuntu host, perform the following steps:

    ssh {mon-host}
    sudo apt-get update && sudo apt-get install ceph
    

    On CentOS/Red Hat hosts, perform the following steps:

    ssh {mon-host}
    sudo yum update && sudo yum install ceph
    
  2. Restart each monitor. For Ubuntu distributions, use:

    sudo restart ceph-mon id={hostname}
    

    For CentOS/Red Hat/Debian distributions, use:

    sudo /etc/init.d/ceph restart {mon-id}
    

    For CentOS/Red Hat distributions deployed with ceph-deploy, the monitor ID is usually mon.{hostname}.

  3. Ensure each monitor has rejoined the quorum:

    ceph mon stat
    

Ensure that you have completed the upgrade cycle for all of your Ceph Monitors.

Upgrading an OSD

To upgrade a Ceph OSD Daemon, perform the following steps:

  1. Upgrade the Ceph OSD Daemon package.

    You may use ceph-deploy to address all Ceph OSD Daemon nodes at once. For example:

    ceph-deploy install --release {release-name} ceph-node1[ ceph-node2]
    ceph-deploy install --release hammer osd1 osd2 osd3
    

    You may also use the package manager on each node to upgrade packages using your distro’s package manager. For Debian/Ubuntu hosts, perform the following steps on each host:

    ssh {osd-host}
    sudo apt-get update && sudo apt-get install ceph
    

    For CentOS/Red Hat hosts, perform the following steps:

    ssh {osd-host}
    sudo yum update && sudo yum install ceph
    
  2. Restart the OSD, where N is the OSD number. For Ubuntu, use:

    sudo restart ceph-osd id=N
    

    For multiple OSDs on a host, you may restart all of them with Upstart.

    sudo restart ceph-osd-all
    

    For CentOS/Red Hat/Debian distributions, use:

    sudo /etc/init.d/ceph restart N
    
  3. Ensure each upgraded Ceph OSD Daemon has rejoined the cluster:

    ceph osd stat
    

Ensure that you have completed the upgrade cycle for all of your Ceph OSD Daemons.

Upgrading a Metadata Server

To upgrade a Ceph Metadata Server, perform the following steps:

  1. Upgrade the Ceph Metadata Server package. You may use ceph-deploy to address all Ceph Metadata Server nodes at once, or use the package manager on each node. For example:

    ceph-deploy install --release {release-name} ceph-node1
    ceph-deploy install --release hammer mds1
    

    To upgrade packages manually, perform the following steps on each Debian/Ubuntu host:

    ssh {mon-host}
    sudo apt-get update && sudo apt-get install ceph-mds
    

    Or the following steps on CentOS/Red Hat hosts:

    ssh {mon-host}
    sudo yum update && sudo yum install ceph-mds
    
  2. Restart the metadata server. For Ubuntu, use:

    sudo restart ceph-mds id={hostname}
    

    For CentOS/Red Hat/Debian distributions, use:

    sudo /etc/init.d/ceph restart mds.{hostname}
    

    For clusters deployed with ceph-deploy, the name is usually either the name you specified on creation or the hostname.

  3. Ensure the metadata server is up and running:

    ceph mds stat
    

Upgrading a Client

Once you have upgraded the packages and restarted daemons on your Ceph cluster, we recommend upgrading ceph-common and client libraries (librbd1 , librados3 and libradospp1) on your client nodes too.

  1. Upgrade the package:

    ssh {client-host}
    apt-get update && sudo apt-get install ceph-common librados3 libradospp1 librbd1 python-rados python-rbd
    
  2. Ensure that you have the latest version:

    ceph --version
    

If you do not have the latest version, you may need to uninstall, auto remove dependencies and reinstall.