Notice

This document is for a development version of Ceph.

CephFS Client Capabilities

Ceph authentication capabilities are used to restrict CephFS clients to the lowest level of authority necessary.

Note

Path restriction and layout-modification restriction were introduced in the Jewel release of Ceph.

Note

Using Erasure Coded (EC) pools with CephFS is supported only with BlueStore. Erasure-coded pools cannot be used as metadata pools. Overwrites must be enabled on erasure-coded data pools.

Path restriction

By default, clients are not restricted in the paths that they are allowed to mount. When clients mount a subdirectory (for example /home/user), the MDS does not by default verify that subsequent operations are “locked” within that directory.

To restrict clients so that they mount and work only within a certain directory, use path-based MDS authentication capabilities.

This restriction impacts only the filesystem hierarchy, or, in other words, the metadata tree that is managed by the MDS. Clients will still be able to access the underlying file data in RADOS directly. To segregate clients fully, isolate untrusted clients in their own RADOS namespace. You can place a client’s filesystem subtree in a particular namespace using file layouts and then restrict their RADOS access to that namespace using OSD capabilities.

Syntax

To grant rw access to the specified directory only, mention the specified directory while creating key for a client. Use a command of the following form:

ceph fs authorize <fs_name> client.<client_id> <path-in-cephfs> rw

For example, to restrict a client named foo so that it can write only in the bar directory of file system cephfs_a, run the following command:

ceph fs authorize cephfs_a client.foo / r /bar rw

This results in:

client.foo
  key: *key*
  caps: [mds] allow r, allow rw path=/bar
  caps: [mon] allow r
  caps: [osd] allow rw tag cephfs data=cephfs_a

To completely restrict the client to the bar directory, omit the root directory :

ceph fs authorize cephfs_a client.foo /bar rw

If a client’s read access is restricted to a path, the client will be able to mount the file system only by specifying a readable path in the mount command (see below).

Supplying all or * as the file system name grants access to every file system. It is usually necessary to quote * to protect it from the shell.

See User Management - Add a User to a Keyring for more on user management.

To restrict a client to only the specified sub-directory, mention the specified directory while mounting. Use a command of the following form:

ceph-fuse -n client.<client_id> <mount-path> -r *directory_to_be_mounted*

For example, to restrict client foo to mnt/bar directory, use the following command:

ceph-fuse -n client.foo mnt -r /bar

Reporting free space

When a client has mounted a sub-directory, the used space (df) is calculated from the quota on that sub-directory rather than from the overall amount of space used on the CephFS file system.

To make the client report the overall usage of the file system and not only the quota usage on the mounted sub-directory, set the following config option on the client:

client quota df = false

If quotas are not enabled or if no quota is set on the mounted sub-directory, then the overall usage of the file system will be reported irrespective of the value of this setting.

Layout and Quota restriction (the ‘p’ flag)

To set layouts or quotas, clients require the p flag in addition to rw. Using the p flag with rw restricts all the attributes that are set by special extended attributes by using a ceph. prefix, and restricts other means of setting these fields (such as openc operations with layouts).

For example, in the following snippet client.0 can modify layouts and quotas on the file system cephfs_a, but client.1 cannot:

client.0
    key: AQAz7EVWygILFRAAdIcuJ12opU/JKyfFmxhuaw==
    caps: [mds] allow rwp
    caps: [mon] allow r
    caps: [osd] allow rw tag cephfs data=cephfs_a

client.1
    key: AQAz7EVWygILFRAAdIcuJ12opU/JKyfFmxhuaw==
    caps: [mds] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw tag cephfs data=cephfs_a

Snapshot restriction (the ‘s’ flag)

To create or delete snapshots, clients require the s flag in addition to rw. Note that when capability string also contains the p flag, the s flag must appear after it (all flags except rw must be specified in alphabetical order).

For example, in the following snippet client.0 can create or delete snapshots in the bar directory of file system cephfs_a:

client.0
    key: AQAz7EVWygILFRAAdIcuJ12opU/JKyfFmxhuaw==
    caps: [mds] allow rw, allow rws path=/bar
    caps: [mon] allow r
    caps: [osd] allow rw tag cephfs data=cephfs_a

Network restriction

client.foo
  key: *key*
  caps: [mds] allow r network 10.0.0.0/8, allow rw path=/bar network 10.0.0.0/8
  caps: [mon] allow r network 10.0.0.0/8
  caps: [osd] allow rw tag cephfs data=cephfs_a network 10.0.0.0/8

The optional {network/prefix} is a standard network-name-and-prefix length in CIDR notation (for example, 10.3.0.0/16). If {network/prefix}} is present, the use of this capability is restricted to clients connecting from this network.

File system Information Restriction

The monitor cluster can present a limited view of the available file systems. In this case, the monitor cluster informs clients only about file systems specified by the administrator. Other file systems are not reported and commands affecting them fail as though the file systems do not exist.

Consider following example. The Ceph cluster has 2 file systems:

ceph fs ls

name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
name: cephfs2, metadata pool: cephfs2_metadata, data pools: [cephfs2_data ]

We authorize client someuser for only one file system:

ceph fs authorize cephfs client.someuser / rw

[client.someuser]
    key = AQAmthpf89M+JhAAiHDYQkMiCq3x+J0n9e8REQ==

cat ceph.client.someuser.keyring

[client.someuser]
    key = AQAmthpf89M+JhAAiHDYQkMiCq3x+J0n9e8REQ==
    caps mds = "allow rw fsname=cephfs"
    caps mon = "allow r fsname=cephfs"
    caps osd = "allow rw tag cephfs data=cephfs"

The client can see only the file system that it is authorized to see:

ceph fs ls -n client.someuser -k ceph.client.someuser.keyring

name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

Standby MDS daemons are always displayed. Information about restricted MDS daemons and file systems may become available by other means, such as by running ceph health detail.

MDS communication restriction

By default, user applications may communicate with any MDS, regardless of whether they are allowed to modify data on an associated file system (see Path restriction above). Client communication can be restricted to MDS daemons associated with particular file system(s) by adding MDS caps for that particular file system. Consider the following example where the Ceph cluster has two file systems:

ceph fs ls

name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
name: cephfs2, metadata pool: cephfs2_metadata, data pools: [cephfs2_data ]

Client someuser is authorized for only one file system:

ceph fs authorize cephfs client.someuser / rw

[client.someuser]
    key = AQBPSARfg8hCJRAAEegIxjlm7VkHuiuntm6wsA==

ceph auth get client.someuser > ceph.client.someuser.keyring

exported keyring for client.someuser

cat ceph.client.someuser.keyring

[client.someuser]
    key = AQBPSARfg8hCJRAAEegIxjlm7VkHuiuntm6wsA==
    caps mds = "allow rw fsname=cephfs"
    caps mon = "allow r"
    caps osd = "allow rw tag cephfs data=cephfs"

Mounting cephfs1 on the already-created mount point /mnt/cephfs1 with someuser works:

sudo ceph-fuse /mnt/cephfs1 -n client.someuser -k ceph.client.someuser.keyring --client-fs=cephfs

Note

If /mnt/cephfs does not exist prior to running the above command, create it by running mkdir /mnt/cephfs1.

ceph-fuse[96634]: starting ceph client
ceph-fuse[96634]: starting fuse

mount | grep ceph-fuse

ceph-fuse on /mnt/cephfs1 type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

Mounting cephfs2 with someuser does not work:

sudo ceph-fuse /mnt/cephfs2 -n client.someuser -k ceph.client.someuser.keyring --client-fs=cephfs2

ceph-fuse[96599]: starting ceph client
ceph-fuse[96599]: ceph mount failed with (1) Operation not permitted

Root squash

The root squash feature is implemented as a safety measure to prevent scenarios such as an accidental forced removal of a path (for example, sudo rm -rf /path). Enable root_squash mode in MDS caps to disallow clients with uid=0 or gid=0 to perform write access operations (for example rm, rmdir, rmsnap, mkdir, and mksnap). This mode permits the read operations on a root client, unlike the behavior of other file systems.

Here is an example of enabling root_squash in a filesystem, except within the /volumes directory tree in the filesystem:

ceph fs authorize a client.test_a / rw root_squash /volumes rw
ceph auth get client.test_a

[client.test_a]
    key = AQBZcDpfEbEUKxAADk14VflBXt71rL9D966mYA==
    caps mds = "allow rw fsname=a root_squash, allow rw fsname=a path=/volumes"
    caps mon = "allow r fsname=a"
    caps osd = "allow rw tag cephfs data=a"

Updating Capabilities using fs authorize

Beginning with the Reef release of Ceph, fs authorize can be used to add new caps to an existing client (for another CephFS or another path in the same file system).

The following example demonstrates the behavior that results from running the command ceph fs authorize a client.x / rw twice.

  1. Create a new client:

    ceph fs authorize a client.x / rw

    [client.x]
    key = AQAOtSVk9WWtIhAAJ3gSpsjwfIQ0gQ6vfSx/0w==

  2. Get the client capabilities:

    ceph auth get client.x

    [client.x]
      key = AQAOtSVk9WWtIhAAJ3gSpsjwfIQ0gQ6vfSx/0w==
      caps mds = "allow rw fsname=a"
      caps mon = "allow r fsname=a"
      caps osd = "allow rw tag cephfs data=a"

  3. Previously, running fs authorize a client.x / rw a second time printed an error message. In the Reef release and in later releases, this command prints a message reporting that the capabilities did not get updated:

    ./bin/ceph fs authorize a client.x / rw

    no update for caps of client.x

Adding New Caps Using fs authorize

Add capabilities for another path in same CephFS:

ceph fs authorize a client.x /dir1 rw

updated caps for client.x

ceph auth get client.x

[client.x]
        key = AQAOtSVk9WWtIhAAJ3gSpsjwfIQ0gQ6vfSx/0w==
        caps mds = "allow r fsname=a, allow rw fsname=a path=some/dir"
        caps mon = "allow r fsname=a"
        caps osd = "allow rw tag cephfs data=a"

Add capabilities for another CephFS on the Ceph cluster:

ceph fs authorize b client.x / rw

updated caps for client.x

ceph auth get client.x

[client.x]
        key = AQD6tiVk0uJdARAABMaQuLRotxTi3Qdj47FkBA==
        caps mds = "allow rw fsname=a, allow rw fsname=b"
        caps mon = "allow r fsname=a, allow r fsname=b"
        caps osd = "allow rw tag cephfs data=a, allow rw tag cephfs data=b"

Changing rw permissions in caps

Capabilities can be modified by running fs authorize only in the case when read/write permissions must be changed. This is because the command fs authorize becomes ambiguous. For example, a user runs fs authorize cephfs1 client.x /dir1 rw to create a client and then runs fs authorize cephfs1 client.x /dir2 rw (notice that /dir1 has been changed to /dir2). Running the second command could be interpreted to change /dir1 to /dir2 with current capabilities or could be interpreted to authorize the client with a new capability for the path /dir2. As shown previously, the second interpretation is chosen and it is therefore impossible to update a part of the capabilities granted except rw permissions. The following shows how read/write permissions for client.x can be changed:

ceph fs authorize a client.x / r
 [client.x]
     key = AQBBKjBkIFhBDBAA6q5PmDDWaZtYjd+jafeVUQ==

ceph auth get client.x

[client.x]
        key = AQBBKjBkIFhBDBAA6q5PmDDWaZtYjd+jafeVUQ==
        caps mds = "allow r fsname=a"
        caps mon = "allow r fsname=a"
        caps osd = "allow r tag cephfs data=a"

fs authorize never deducts any part of caps

Capabilities that have been issued to a client can not be removed by running fs authorize again. For example, if a client capability has root_squash applied on a certain CephFS, running fs authorize again for the same CephFS but without root_squash will not lead to any update and the client caps will remain unchanged:

ceph fs authorize a client.x / rw root_squash

[client.x]
        key = AQD61CVkcA1QCRAAd0XYqPbHvcc+lpUAuc6Vcw==

ceph auth get client.x

[client.x]
        key = AQD61CVkcA1QCRAAd0XYqPbHvcc+lpUAuc6Vcw==
        caps mds = "allow rw fsname=a root_squash"
        caps mon = "allow r fsname=a"
        caps osd = "allow rw tag cephfs data=a"

ceph fs authorize a client.x / rw

[client.x]
        key = AQD61CVkcA1QCRAAd0XYqPbHvcc+lpUAuc6Vcw==
no update was performed for caps of client.x. caps of client.x remains unchanged.

If a client already has a capability for file-system name a and path dir1, running fs authorize again for FS name a but path dir2, instead of modifying the capabilities client already holds, a new cap for dir2 will be granted:

ceph fs authorize a client.x /dir1 rw
ceph auth get client.x

[client.x]
        key = AQC1tyVknMt+JxAAp0pVnbZGbSr/nJrmkMNKqA==
        caps mds = "allow rw fsname=a path=/dir1"
        caps mon = "allow r fsname=a"
        caps osd = "allow rw tag cephfs data=a"

ceph fs authorize a client.x /dir2 rw

updated caps for client.x

ceph auth get client.x

[client.x]
        key = AQC1tyVknMt+JxAAp0pVnbZGbSr/nJrmkMNKqA==
        caps mds = "allow rw fsname=a path=dir1, allow rw fsname=a path=dir2"
        caps mon = "allow r fsname=a"
        caps osd = "allow rw tag cephfs data=a"

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.