Notice

This document is for a development version of Ceph.

CephFS Exports over NFS

CephFS namespaces can be exported over NFS protocol using the NFS-Ganesha NFS server

Requirements

  • Latest Ceph file system with mgr enabled

  • nfs-ganesha, nfs-ganesha-ceph, nfs-ganesha-rados-grace and nfs-ganesha-rados-urls packages (version 3.3 and above)

Note

From Pacific, the nfs mgr module must be enabled prior to use.

Ganesha Configuration Hierarchy

Cephadm and rook starts nfs-ganesha daemon with bootstrap configuration containing minimal ganesha configuration, creates empty rados common config object in nfs-ganesha pool and watches this config object. The mgr/nfs module adds rados export object urls to the common config object. If cluster config is set, it creates user config object containing custom ganesha configuration and adds it url to common config object.

../../_images/86401b17a202ce3640efd170f50140324c51977c3dde60c0af829ff78bb2557c.png

Create NFS Ganesha Cluster

$ ceph nfs cluster create <clusterid> [<placement>] [--ingress --virtual-ip <ip>]

This creates a common recovery pool for all NFS Ganesha daemons, new user based on clusterid, and a common NFS Ganesha config RADOS object.

Note

Since this command also brings up NFS Ganesha daemons using a ceph-mgr orchestrator module (see Orchestrator CLI) such as “mgr/cephadm”, at least one such module must be enabled for it to work.

Currently, NFS Ganesha daemon deployed by cephadm listens on the standard port. So only one daemon will be deployed on a host.

<clusterid> is an arbitrary string by which this NFS Ganesha cluster will be known.

<placement> is an optional string signifying which hosts should have NFS Ganesha daemon containers running on them and, optionally, the total number of NFS Ganesha daemons on the cluster (should you want to have more than one NFS Ganesha daemon running per node). For example, the following placement string means “deploy NFS Ganesha daemons on nodes host1 and host2 (one daemon per host):

"host1,host2"

and this placement specification says to deploy single NFS Ganesha daemon each on nodes host1 and host2 (for a total of two NFS Ganesha daemons in the cluster):

"2 host1,host2"

To deploy NFS with an HA front-end (virtual IP and load balancer), add the --ingress flag and specify a virtual IP address. This will deploy a combination of keepalived and haproxy to provide an high-availability NFS frontend for the NFS service.

For more details, refer Placement Specification but keep in mind that specifying the placement via a YAML file is not supported.

Delete NFS Ganesha Cluster

$ ceph nfs cluster rm <clusterid>

This deletes the deployed cluster.

List NFS Ganesha Cluster

$ ceph nfs cluster ls

This lists deployed clusters.

Show NFS Ganesha Cluster Information

$ ceph nfs cluster info [<clusterid>]

This displays ip and port of deployed cluster.

Note

This will not work with rook backend. Instead expose port with kubectl patch command and fetch the port details with kubectl get services command:

$ kubectl patch service -n rook-ceph -p '{"spec":{"type": "NodePort"}}' rook-ceph-nfs-<cluster-name>-<node-id>
$ kubectl get services -n rook-ceph rook-ceph-nfs-<cluster-name>-<node-id>

Set Customized NFS Ganesha Configuration

$ ceph nfs cluster config set <clusterid> -i <config_file>

With this the nfs cluster will use the specified config and it will have precedence over default config blocks.

Example use cases

  1. Changing log level

It can be done by adding LOG block in the following way:

LOG {
 COMPONENTS {
     ALL = FULL_DEBUG;
 }
}
  1. Adding custom export block

The following sample block creates a single export. This export will not be managed by ceph nfs export interface:

EXPORT {
  Export_Id = 100;
  Transports = TCP;
  Path = /;
  Pseudo = /ceph/;
  Protocols = 4;
  Access_Type = RW;
  Attr_Expiration_Time = 0;
  Squash = None;
  FSAL {
    Name = CEPH;
    Filesystem = "filesystem name";
    User_Id = "user id";
    Secret_Access_Key = "secret key";
  }
}

Note

User specified in FSAL block should have proper caps for NFS-Ganesha daemons to access ceph cluster. User can be created in following way using auth get-or-create:

# ceph auth get-or-create client.<user_id> mon 'allow r' osd 'allow rw pool=nfs-ganesha namespace=<nfs_cluster_name>, allow rw tag cephfs data=<fs_name>' mds 'allow rw path=<export_path>'

Reset NFS Ganesha Configuration

$ ceph nfs cluster config reset <clusterid>

This removes the user defined configuration.

Note

With a rook deployment, ganesha pods must be explicitly restarted for the new config blocks to be effective.

Create CephFS Export

Warning

Currently, the nfs interface is not integrated with dashboard. Both dashboard and nfs interface have different export requirements and create exports differently. Management of dashboard created exports is not supported.

$ ceph nfs export create cephfs <fsname> <clusterid> <binding> [--readonly] [--path=/path/in/cephfs]

This creates export RADOS objects containing the export block, where

<fsname> is the name of the FS volume used by the NFS Ganesha cluster that will serve this export.

<clusterid> is the NFS Ganesha cluster ID.

<binding> is the pseudo root path (must be an absolute path and unique). It specifies the export position within the NFS v4 Pseudo Filesystem.

<path> is the path within cephfs. Valid path should be given and default path is ‘/’. It need not be unique. Subvolume path can be fetched using:

$ ceph fs subvolume getpath <vol_name> <subvol_name> [--group_name <subvol_group_name>]

Note

Export creation is supported only for NFS Ganesha clusters deployed using nfs interface.

Delete CephFS Export

$ ceph nfs export rm <clusterid> <binding>

This deletes an export in an NFS Ganesha cluster, where:

<clusterid> is the NFS Ganesha cluster ID.

<binding> is the pseudo root path (must be an absolute path).

List CephFS Exports

$ ceph nfs export ls <clusterid> [--detailed]

It lists exports for a cluster, where:

<clusterid> is the NFS Ganesha cluster ID.

With the --detailed option enabled it shows entire export block.

Get CephFS Export

$ ceph nfs export get <clusterid> <binding>

This displays export block for a cluster based on pseudo root name (binding), where:

<clusterid> is the NFS Ganesha cluster ID.

<binding> is the pseudo root path (must be an absolute path).

Update CephFS Export

$ ceph nfs export update -i <json_file>

This updates the cephfs export specified in the json file. Export in json format can be fetched with above get command. For example:

$ ceph nfs export get vstart /cephfs > update_cephfs_export.json
$ cat update_cephfs_export.json
{
  "export_id": 1,
  "path": "/",
  "cluster_id": "vstart",
  "pseudo": "/cephfs",
  "access_type": "RW",
  "squash": "no_root_squash",
  "security_label": true,
  "protocols": [
    4
  ],
  "transports": [
    "TCP"
  ],
  "fsal": {
    "name": "CEPH",
    "user_id": "vstart1",
    "fs_name": "a",
    "sec_label_xattr": ""
  },
  "clients": []
}
# Here in the fetched export, pseudo and access_type is modified. Then the modified file is passed to update interface
$ ceph nfs export update -i update_cephfs_export.json
$ cat update_cephfs_export.json
{
  "export_id": 1,
  "path": "/",
  "cluster_id": "vstart",
  "pseudo": "/cephfs_testing",
  "access_type": "RO",
  "squash": "no_root_squash",
  "security_label": true,
  "protocols": [
    4
  ],
  "transports": [
    "TCP"
  ],
  "fsal": {
    "name": "CEPH",
    "user_id": "vstart1",
    "fs_name": "a",
    "sec_label_xattr": ""
  },
  "clients": []
}

Configuring NFS Ganesha to export CephFS with vstart

  1. Using cephadm

    $ MDS=1 MON=1 OSD=3 NFS=1 ../src/vstart.sh -n -d --cephadm
    

    This will deploy a single NFS Ganesha daemon using vstart.sh, where the daemon will listen on the default NFS Ganesha port.

  2. Using test orchestrator

    $ MDS=1 MON=1 OSD=3 NFS=1 ../src/vstart.sh -n -d
    

    Environment variable NFS is the number of NFS Ganesha daemons to be deployed, each listening on a random port.

    Note

    NFS Ganesha packages must be pre-installed for this to work.

Mount

After the exports are successfully created and NFS Ganesha daemons are no longer in grace period. The exports can be mounted by

$ mount -t nfs -o port=<ganesha-port> <ganesha-host-name>:<ganesha-pseudo-path> <mount-point>

Note

Only NFS v4.0+ is supported.

Troubleshooting

Checking NFS-Ganesha logs with

  1. cephadm

    $ cephadm logs --fsid <fsid> --name nfs.<cluster_id>.hostname
    
  2. rook

    $ kubectl logs -n rook-ceph rook-ceph-nfs-<cluster_id>-<node_id> nfs-ganesha
    

Log level can be changed using nfs cluster config set command.