Installation (Kubernetes + Helm)

The ceph-helm project enables you to deploy Ceph in a Kubernetes environment. This documentation assumes a Kubernetes environment is available.

Current limitations

  • The public and cluster networks must be the same

  • If the storage class user id is not admin, you will have to manually create the user in your Ceph cluster and create its secret in Kubernetes

  • ceph-mgr can only run with 1 replica

Install and start Helm

Helm can be installed by following these instructions.

Helm finds the Kubernetes cluster by reading from the local Kubernetes config file; make sure this is downloaded and accessible to the helm client.

A Tiller server must be configured and running for your Kubernetes cluster, and the local Helm client must be connected to it. It may be helpful to look at the Helm documentation for init. To run Tiller locally and connect Helm to it, run:

$ helm init

The ceph-helm project uses a local Helm repo by default to store charts. To start a local Helm repo server, run:

$ helm serve &
$ helm repo add local http://localhost:8879/charts

Add ceph-helm to Helm local repos

$ git clone
$ cd ceph-helm/ceph
$ make

Configure your Ceph cluster

Create a ceph-overrides.yaml that will contain your Ceph configuration. This file may exist anywhere, but for this document will be assumed to reside in the user’s home directory.:

$ cat ~/ceph-overrides.yaml

  - name: dev-sdd
    device: /dev/sdd
    zap: "1"
  - name: dev-sde
    device: /dev/sde
    zap: "1"

  name: ceph-rbd
  pool: rbd
  user_id: k8s


If journal is not set it will be colocated with device


The ceph-helm/ceph/ceph/values.yaml file contains the full list of option that can be set

Create the Ceph cluster namespace

By default, ceph-helm components assume they are to be run in the ceph Kubernetes namespace. To create the namespace, run:

$ kubectl create namespace ceph

Configure RBAC permissions

Kubernetes >=v1.6 makes RBAC the default admission controller. ceph-helm provides RBAC roles and permissions for each component:

$ kubectl create -f ~/ceph-helm/ceph/rbac.yaml

The rbac.yaml file assumes that the Ceph cluster will be deployed in the ceph namespace.

Label kubelets

The following labels need to be set to deploy a Ceph cluster:
  • ceph-mon=enabled

  • ceph-mgr=enabled

  • ceph-osd=enabled

  • ceph-osd-device-<name>=enabled

The ceph-osd-device-<name> label is created based on the osd_devices name value defined in our ceph-overrides.yaml. From our example above we will have the two following label: ceph-osd-device-dev-sdb and ceph-osd-device-dev-sdc.

For each Ceph Monitor:

$ kubectl label node <nodename> ceph-mon=enabled ceph-mgr=enabled

For each OSD node:

$ kubectl label node <nodename> ceph-osd=enabled ceph-osd-device-dev-sdb=enabled ceph-osd-device-dev-sdc=enabled

Ceph Deployment

Run the helm install command to deploy Ceph:

$ helm install --name=ceph local/ceph --namespace=ceph -f ~/ceph-overrides.yaml
NAME:   ceph
LAST DEPLOYED: Wed Oct 18 22:25:06 2017

==> v1/Secret
NAME                    TYPE    DATA  AGE
ceph-keystone-user-rgw  Opaque  7     1s

==> v1/ConfigMap
NAME              DATA  AGE
ceph-bin-clients  2     1s
ceph-bin          24    1s
ceph-etc          1     1s
ceph-templates    5     1s

==> v1/Service
ceph-mon  None            <none>       6789/TCP  1s
ceph-rgw  <none>       8088/TCP  1s

==> v1beta1/DaemonSet
NAME              DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE-SELECTOR                                     AGE
ceph-mon          3        3        0      3           0          ceph-mon=enabled                                  1s
ceph-osd-dev-sde  3        3        0      3           0          ceph-osd-device-dev-sde=enabled,ceph-osd=enabled  1s
ceph-osd-dev-sdd  3        3        0      3           0          ceph-osd-device-dev-sdd=enabled,ceph-osd=enabled  1s

==> v1beta1/Deployment
ceph-mds              1        1        1           0          1s
ceph-mgr              1        1        1           0          1s
ceph-mon-check        1        1        1           0          1s
ceph-rbd-provisioner  2        2        2           0          1s
ceph-rgw              1        1        1           0          1s

==> v1/Job
NAME                                 DESIRED  SUCCESSFUL  AGE
ceph-mgr-keyring-generator           1        0           1s
ceph-mds-keyring-generator           1        0           1s
ceph-osd-keyring-generator           1        0           1s
ceph-rgw-keyring-generator           1        0           1s
ceph-mon-keyring-generator           1        0           1s
ceph-namespace-client-key-generator  1        0           1s
ceph-storage-keys-generator          1        0           1s

==> v1/StorageClass

The output from helm install shows us the different types of resources that will be deployed.

A StorageClass named ceph-rbd of type will be created with ceph-rbd-provisioner Pods. These will allow a RBD to be automatically provisioned upon creation of a PVC. RBDs will also be formatted when mapped for the first time. All RBDs will use the ext4 filesystem. does not support the fsType option. By default, RBDs will use image format 2 and layering. You can overwrite the following storageclass’ defaults in your values file:

  name: ceph-rbd
  pool: rbd
  user_id: k8s
  user_secret_name: pvc-ceph-client-key
  image_format: "2"
  image_features: layering

Check that all Pods are running with the command below. This might take a few minutes:

$ kubectl -n ceph get pods
NAME                                    READY     STATUS    RESTARTS   AGE
ceph-mds-3804776627-976z9               0/1       Pending   0          1m
ceph-mgr-3367933990-b368c               1/1       Running   0          1m
ceph-mon-check-1818208419-0vkb7         1/1       Running   0          1m
ceph-mon-cppdk                          3/3       Running   0          1m
ceph-mon-t4stn                          3/3       Running   0          1m
ceph-mon-vqzl0                          3/3       Running   0          1m
ceph-osd-dev-sdd-6dphp                  1/1       Running   0          1m
ceph-osd-dev-sdd-6w7ng                  1/1       Running   0          1m
ceph-osd-dev-sdd-l80vv                  1/1       Running   0          1m
ceph-osd-dev-sde-6dq6w                  1/1       Running   0          1m
ceph-osd-dev-sde-kqt0r                  1/1       Running   0          1m
ceph-osd-dev-sde-lp2pf                  1/1       Running   0          1m
ceph-rbd-provisioner-2099367036-4prvt   1/1       Running   0          1m
ceph-rbd-provisioner-2099367036-h9kw7   1/1       Running   0          1m
ceph-rgw-3375847861-4wr74               0/1       Pending   0          1m


The MDS and RGW Pods are pending since we did not label any nodes with ceph-rgw=enabled or ceph-mds=enabled

Once all Pods are running, check the status of the Ceph cluster from one Mon:

$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph -s
  id:     e8f9da03-c2d2-4ad3-b807-2a13d0775504
  health: HEALTH_OK

  mon: 3 daemons, quorum mira115,mira110,mira109
  mgr: mira109(active)
  osd: 6 osds: 6 up, 6 in

  pools:   0 pools, 0 pgs
  objects: 0 objects, 0 bytes
  usage:   644 MB used, 5555 GB / 5556 GB avail

Configure a Pod to use a PersistentVolume from Ceph

Create a keyring for the k8s user defined in the ~/ceph-overwrite.yaml and convert it to base64:

$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- bash
# ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx pool=rbd'  | base64
# exit

Edit the user secret present in the ceph namespace:

$ kubectl -n ceph edit secrets/pvc-ceph-client-key

Add the base64 value to the key value with your own and save:

apiVersion: v1
kind: Secret
  creationTimestamp: 2017-10-19T17:34:04Z
  name: pvc-ceph-client-key
  namespace: ceph
  resourceVersion: "8665522"
  selfLink: /api/v1/namespaces/ceph/secrets/pvc-ceph-client-key
  uid: b4085944-b4f3-11e7-add7-002590347682

We are going to create a Pod that consumes a RBD in the default namespace. Copy the user secret from the ceph namespace to default:

$ kubectl -n ceph get secrets/pvc-ceph-client-key -o json | jq '.metadata.namespace = "default"' | kubectl create -f -
secret "pvc-ceph-client-key" created
$ kubectl get secrets
NAME                  TYPE                                  DATA      AGE
default-token-r43wl   3         61d
pvc-ceph-client-key                     1         20s

Create and initialize the RBD pool:

$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph osd pool create rbd 256
pool 'rbd' created
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd pool init rbd


Kubernetes uses the RBD kernel module to map RBDs to hosts. Luminous requires CRUSH_TUNABLES 5 (Jewel). The minimal kernel version for these tunables is 4.5. If your kernel does not support these tunables, run ceph osd crush tunables hammer


Since RBDs are mapped on the host system. Hosts need to be able to resolve the ceph-mon.ceph.svc.cluster.local name managed by the kube-dns service. To get the IP address of the kube-dns service, run kubectl -n kube-system get svc/kube-dns

Create a PVC:

$ cat pvc-rbd.yaml
kind: PersistentVolumeClaim
apiVersion: v1
  name: ceph-pvc
   - ReadWriteOnce
       storage: 20Gi
  storageClassName: ceph-rbd

$ kubectl create -f pvc-rbd.yaml
persistentvolumeclaim "ceph-pvc" created
$ kubectl get pvc
NAME       STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
ceph-pvc   Bound     pvc-1c2ada50-b456-11e7-add7-002590347682   20Gi       RWO           ceph-rbd        3s

You can check that the RBD has been created on your cluster:

$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd ls
$ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd info kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915
rbd image 'kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915':
    size 20480 MB in 5120 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.10762ae8944a
    format: 2
    features: layering
    create_timestamp: Wed Oct 18 22:45:59 2017

Create a Pod that will use the PVC:

$ cat pod-with-rbd.yaml
kind: Pod
apiVersion: v1
  name: mypod
    - name: busybox
      image: busybox
        - sleep
        - "3600"
      - mountPath: "/mnt/rbd"
        name: vol1
    - name: vol1
        claimName: ceph-pvc

$ kubectl create -f pod-with-rbd.yaml
pod "mypod" created

Check the Pod:

$ kubectl get pods
mypod     1/1       Running   0          17s
$ kubectl exec mypod -- mount | grep rbd
/dev/rbd0 on /mnt/rbd type ext4 (rw,relatime,stripe=1024,data=ordered)


OSDs and Monitor logs can be accessed via the kubectl logs [-f] command. Monitors have multiple stream of logging, each stream is accessible from a container running in the ceph-mon Pod.

There are 3 containers running in the ceph-mon Pod:
  • ceph-mon, equivalent of ceph-mon.hostname.log on baremetal

  • cluster-audit-log-tailer, equivalent of ceph.audit.log on baremetal

  • cluster-log-tailer, equivalent of ceph.log on baremetal or ceph -w

Each container is accessible via the --container or -c option. For instance, to access the cluster-tail-log, one can run:

$ kubectl -n ceph logs ceph-mon-cppdk -c cluster-log-tailer