Overview

The ceph-volume tool aims to be a single purpose command line tool to deploy logical volumes as OSDs, trying to maintain a similar API to ceph-disk when preparing, activating, and creating OSDs.

It deviates from ceph-disk by not interacting or relying on the udev rules that come installed for Ceph. These rules allow automatic detection of previously setup devices that are in turn fed into ceph-disk to activate them.

Replacing ceph-disk

The ceph-disk tool was created at a time were the project was required to support many different types of init systems (upstart, sysvinit, etc…) while being able to discover devices. This caused the tool to concentrate initially (and exclusively afterwards) on GPT partitions. Specifically on GPT GUIDs, which were used to label devices in a unique way to answer questions like:

  • is this device a Journal?
  • an encrypted data partition?
  • was the device left partially prepared?

To solve these, it used UDEV rules to match the GUIDs, that would call ceph-disk, and end up in a back and forth between the ceph-disk systemd unit and the ceph-disk executable. The process was very unreliable and time consuming (a timeout of close to three hours per OSD had to be put in place), and would cause OSDs to not come up at all during the boot process of a node.

It was hard to debug, or even replicate these problems given the asynchronous behavior of UDEV.

Since the world-view of ceph-disk had to be GPT partitions exclusively, it meant that it couldn’t work with other technologies like LVM, or similar device mapper devices. It was ultimately decided to create something modular, starting with LVM support, and the ability to expand on other technologies as needed.

GPT partitions are simple?

Although partitions in general are simple to reason about, ceph-disk partitions were not simple by any means. It required a tremendous amount of special flags in order to get them to work correctly with the device discovery workflow. Here is an example call to create a data partition:

/sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:f0fc39fd-eeb2-49f1-b922-a11939cf8a0f --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdb

Not only creating these was hard, but these partitions required devices to be exclusively owned by Ceph. For example, in some cases a special partition would be created when devices were encrypted, which would contain unencrypted keys. This was ceph-disk domain knowledge, which would not translate to a “GPT partitions are simple” understanding. Here is an example of that special partition being created:

/sbin/sgdisk --new=5:0:+10M --change-name=5:ceph lockbox --partition-guid=5:None --typecode=5:fb3aabf9-d25f-47cc-bf5e-721d181642be --mbrtogpt -- /dev/sdad

Modularity

ceph-volume was designed to be a modular tool because we anticipate that there are going to be lots of ways that people provision the hardware devices that we need to consider. There are already two: legacy ceph-disk devices that are still in use and have GPT partitions (handled by simple), and lvm. SPDK devices where we manage NVMe devices directly from userspace are on the immediate horizon, where LVM won’t work there since the kernel isn’t involved at all.

ceph-volume lvm

By making use of LVM tags, the lvm sub-command is able to store and later re-discover and query devices associated with OSDs so that they can later activated. This includes support for lvm-based technologies like dm-cache as well.

For ceph-volume, the use of dm-cache is transparent, there is no difference for the tool, and it treats dm-cache like a plain logical volume.

LVM performance penalty

In short: we haven’t been able to notice any significant performance penalties associated with the change to LVM. By being able to work closely with LVM, the ability to work with other device mapper technologies (for example dmcache) was a given: there is no technical difficulty in working with anything that can sit below a Logical Volume.