FS volumes and subvolumes

A single source of truth for CephFS exports is implemented in the volumes module of the Ceph Manager daemon (ceph-mgr). The OpenStack shared file system service (manila), Ceph Containter Storage Interface (CSI), storage administrators among others can use the common CLI provided by the ceph-mgr volumes module to manage the CephFS exports.

The ceph-mgr volumes module implements the following file system export abstactions:

  • FS volumes, an abstraction for CephFS file systems

  • FS subvolumes, an abstraction for independent CephFS directory trees

  • FS subvolume groups, an abstraction for a directory level higher than FS subvolumes to effect policies (e.g., File layouts) across a set of subvolumes

Some possible use-cases for the export abstractions:

  • FS subvolumes used as manila shares or CSI volumes

  • FS subvolume groups used as manila share groups

Requirements

  • Nautilus (14.2.x) or a later version of Ceph

  • Cephx client user (see User Management) with the following minimum capabilities:

    mon 'allow r'
    mgr 'allow rw'
    

FS Volumes

Create a volume using:

$ ceph fs volume create <vol_name>

This creates a CephFS file sytem and its data and metadata pools. It also tries to create MDSes for the filesytem using the enabled ceph-mgr orchestrator module (see Orchestrator CLI) , e.g., rook.

Remove a volume using:

$ ceph fs volume rm <vol_name> [--yes-i-really-mean-it]

This removes a file system and its data and metadata pools. It also tries to remove MDSes using the enabled ceph-mgr orchestrator module.

List volumes using:

$ ceph fs volume ls

FS Subvolume groups

Create a subvolume group using:

$ ceph fs subvolumegroup create <vol_name> <group_name> [--pool_layout <data_pool_name> --uid <uid> --gid <gid> --mode <octal_mode>]

The command succeeds even if the subvolume group already exists.

When creating a subvolume group you can specify its data pool layout (see File layouts), uid, gid, and file mode in octal numerals. By default, the subvolume group is created with an octal file mode ‘755’, uid ‘0’, gid ‘0’ and data pool layout of its parent directory.

Remove a subvolume group using:

$ ceph fs subvolumegroup rm <vol_name> <group_name> [--force]

The removal of a subvolume group fails if it is not empty or non-existent. ‘–force’ flag allows the non-existent subvolume group remove command to succeed.

Fetch the absolute path of a subvolume group using:

$ ceph fs subvolumegroup getpath <vol_name> <group_name>

List subvolume groups using:

$ ceph fs subvolumegroup ls <vol_name>

Create a snapshot (see Experimental Features) of a subvolume group using:

$ ceph fs subvolumegroup snapshot create <vol_name> <group_name> <snap_name>

This implicitly snapshots all the subvolumes under the subvolume group.

Remove a snapshot of a subvolume group using:

$ ceph fs subvolumegroup snapshot rm <vol_name> <group_name> <snap_name> [--force]

Using the ‘–force’ flag allows the command to succeed that would otherwise fail if the snapshot did not exist.

List snapshots of a subvolume group using:

$ ceph fs subvolumegroup snapshot ls <vol_name> <group_name>

FS Subvolumes

Create a subvolume using:

$ ceph fs subvolume create <vol_name> <subvol_name> [--size <size_in_bytes> --group_name <subvol_group_name> --pool_layout <data_pool_name> --uid <uid> --gid <gid> --mode <octal_mode>]

The command succeeds even if the subvolume already exists.

When creating a subvolume you can specify its subvolume group, data pool layout, uid, gid, file mode in octal numerals, and size in bytes. The size of the subvolume is specified by setting a quota on it (see Quotas). By default a subvolume is created within the default subvolume group, and with an octal file mode ‘755’, uid of its subvolume group, gid of its subvolume group, data pool layout of its parent directory and no size limit.

Remove a subvolume using:

$ ceph fs subvolume rm <vol_name> <subvol_name> [--group_name <subvol_group_name> --force]

The command removes the subvolume and its contents. It does this in two steps. First, it move the subvolume to a trash folder, and then asynchronously purges its contents.

The removal of a subvolume fails if it has snapshots, or is non-existent. ‘–force’ flag allows the non-existent subvolume remove command to succeed.

Resize a subvolume using:

$ ceph fs subvolume resize <vol_name> <subvol_name> <new_size> [--group_name <subvol_group_name>] [--no_shrink]

The command resizes the subvolume quota using the size specified by ‘new_size’. ‘–no_shrink’ flag prevents the subvolume to shrink below the current used size of the subvolume.

The subvolume can be resized to an infinite size by passing ‘inf’ or ‘infinite’ as the new_size.

Fetch the absolute path of a subvolume using:

$ ceph fs subvolume getpath <vol_name> <subvol_name> [--group_name <subvol_group_name>]

List subvolumes using:

$ ceph fs subvolume ls <vol_name> [--group_name <subvol_group_name>]

Create a snapshot of a subvolume using:

$ ceph fs subvolume snapshot create <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]

Remove a snapshot of a subvolume using:

$ ceph fs subvolume snapshot rm <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name> --force]

Using the ‘–force’ flag allows the command to succeed that would otherwise fail if the snapshot did not exist.

List snapshots of a subvolume using:

$ ceph fs subvolume snapshot ls <vol_name> <subvol_name> [--group_name <subvol_group_name>]

Cloning Snapshots

Subvolumes can be created by cloning subvolume snapshots. Cloning is an asynchronous operation involving copying data from a snapshot to a subvolume. Due to this bulk copy nature, cloning is currently inefficient for very huge data sets.

Before starting a clone operation, the snapshot should be protected. Protecting a snapshot ensures that the snapshot cannot be deleted when a clone operation is in progress. Snapshots can be protected using:

$ ceph fs subvolume snapshot protect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]

To initiate a clone operation use:

$ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name>

If a snapshot (source subvolume) is a part of non-default group, the group name needs to be specified as per:

$ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --group_name <subvol_group_name>

Cloned subvolumes can be a part of a different group than the source snapshot (by default, cloned subvolumes are created in default group). To clone to a particular group use:

$ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --target_group_name <subvol_group_name>

Similar to specifying a pool layout when creating a subvolume, pool layout can be specified when creating a cloned subvolume. To create a cloned subvolume with a specific pool layout use:

$ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --pool_layout <pool_layout>

To check the status of a clone operation use:

$ ceph fs clone status <vol_name> <clone_name> [--group_name <group_name>]

A clone can be in one of the following states:

  1. pending : Clone operation has not started

  2. in-progress : Clone operation is in progress

  3. complete : Clone operation has sucessfully finished

  4. failed : Clone operation has failed

Sample output from an in-progress clone operation:

$ ceph fs subvolume snapshot protect cephfs subvol1 snap1
$ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
$ ceph fs clone status cephfs clone1
{
  "status": {
    "state": "in-progress",
    "source": {
      "volume": "cephfs",
      "subvolume": "subvol1",
      "snapshot": "snap1"
    }
  }
}

(NOTE: since subvol1 is in default group, source section in clone status does not include group name)

Note

Cloned subvolumes are accessible only after the clone operation has successfully completed.

For a successsful clone operation, clone status would look like so:

$ ceph fs clone status cephfs clone1
{
  "status": {
    "state": "complete"
  }
}

or failed state when clone is unsuccessful.

On failure of a clone operation, the partial clone needs to be deleted and the clone operation needs to be retriggered. To delete a partial clone use:

$ ceph fs subvolume rm <vol_name> <clone_name> [--group_name <group_name>] --force

When no clone operations are in progress or scheduled, the snaphot can be unprotected. To unprotect a snapshot use:

$ ceph fs subvolume snapshot unprotect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]

Note that unprotecting a snapshot would fail if there are pending or in progress clone operations. Also note that, only unprotected snapshots can be removed. This guarantees that a snapshot cannot be deleted when clones are pending (or in progress).

Note

Cloning only synchronizes directories, regular files and symbolic links. Also, inode timestamps (access and modification times) are synchronized upto seconds granularity.

An in-progress or a pending clone operation can be canceled. To cancel a clone operation use the clone cancel command:

$ ceph fs clone cancel <vol_name> <clone_name> [--group_name <group_name>]

On successful cancelation, the cloned subvolume is moved to canceled state:

$ ceph fs subvolume snapshot protect cephfs subvol1 snap1
$ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
$ ceph fs clone cancel cephfs clone1
$ ceph fs clone status cephfs clone1
{
  "status": {
    "state": "canceled",
    "source": {
      "volume": "cephfs",
      "subvolume": "subvol1",
      "snapshot": "snap1"
    }
  }
}

Note

The canceled cloned can be deleted by using –force option in fs subvolume rm command.