Introduction

cephadm-ansible is a collection of Ansible modules and playbooks to simplify workflows that are not covered by [cephadm]. They are covered by the following playbooks:

  • cephadm-preflight.yml: Initial setup of hosts before bootstrapping the cluster

  • cephadm-clients.yml: Setting up client hosts

  • cephadm-purge-cluster.yml: Remove a Ceph cluster

  • cephadm-distribute-ssh-key.yml: Distribute a SSH public key to all hosts

  • cephadm-set-container-insecure-registries.yml: Add a block in /etc/containers/registries.conf to add an insecure registry

Additionnally, several ansible modules are provided in order to let people writing their own playbooks.

cephadm

https://docs.ceph.com/en/latest/cephadm/

Terminology

admin host

A host where the admin keyring and ceph config file are present. Although the admin host and the bootstrap host are usually the same host, it is possible to have multiple admin hosts later. cephadm will make a host become ‘admin’ when the label _admin is added to that host. (ie: ceph orch host label add <host> _admin). This hosts should be present in the group [admin] in the ansible inventory. If for some reason you decide a host shouldn’t be a ‘admin host’ anymore, you have to :

  • remove it from the group [admin] in the ansible inventory,

  • remove the admin keyring,

  • remove the ceph config file,

  • remove the _admin label. (ie ceph orch host label rm <host> _admin)

ansible host

The host where any cephadm-ansible playbook is run.

bootstrap host

The host where the ceph cluster will start. Unless you pass the --skip-admin-label option to the ceph bootstram command, this host will receive the admin keyring and the ceph config file and should be considered as an ‘admin host’. This host should be present in the group [admin] in the ansible inventory.

Ansible inventory

The ansible inventory is a file where all the hosts intended to be part of the ceph cluster will be listed. The most common format are INI or YAML.

Although you probably want to keep it as simple as possible, you can organize your inventory and create groups, cephadm-ansible won’t differentiate except for the following requirements:

  • Client hosts must be defined in a dedicated group [clients].

  • Both cephadm-purge-cluster.yml and cephadm-clients.yml playbooks require a group [admin] with at least one admin host (usually it will be the bootstrap node).

Note

the name of the client group can be changed. In that case you have to set the variable client_group.

Otherwise, you can create groups such as [monitors], [osds], [rgws], that might help you keep clarity in your inventory file and ease the --limit usage if you plan to use it to target specific a group of hosts only.

A basic inventory would look like following:

# cat hosts
ceph-mon1
ceph-mon2
ceph-mon3
ceph-osd1
ceph-osd2
ceph-osd3
ceph-mds1
ceph-mds2
ceph-rgw1
ceph-rgw2

[clients]
ceph-client1
ceph-client2
ceph-client3

[admin]
ceph-mon1

Playbooks

cephadm-preflight

This playbook configures the Ceph repository. It also installs some prerequisites (podman, lvm2, chronyd, cephadm, …)

Usage:

ansible-playbook -i <inventory host file> cephadm-preflight.yml

You can limit the execution to a set of hosts by using --limit option:

ansible-playbook -i <inventory host file> cephadm-preflight.yml --limit <my_osd_group|my_node_name>

You can override variables using --extra-vars parameter:

ansible-playbook -i <inventory host file> cephadm-preflight.yml --extra-vars "ceph_origin=rhcs"

Options

ceph_origin

description

The source of Ceph repositories.

valid values

rhcs

Repository from Red Hat Ceph Storage.

community

Community repository (https://download.ceph.com)

custom

Custom repository. When ceph_origin: custom is defined, you have to set the variable custom_repo_url with the URL of your repository. Passing the extra-var -e custom_repo_state=absent allows you to remove this repository later.

It also supports deploying multiple repositories, in that case you must set the variable ceph_custom_repositories instead. ceph_custom_repositories is a dictionnary that should look like following:

ceph_custom_repositories:
  - name: ceph_custom_noarch
    state: present
    description: Ceph custom repo noarch
    gpgcheck: 'no'
    baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/noarch
    file: ceph_shaman_build_noarch
    priority: '2'
    enabled: 1
  - name: ceph_custom_x86_64
    state: present
    description: Ceph custom repo x86_64
    gpgcheck: 'no'
    baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/x86_64
    file: ceph_shaman_build_x86_64
    priority: '2'
    enabled: 1

Given that the definition is more complex, you might want to define it as a group_vars/host_vars rather than as an extra-var:

$ cat group_vars/all
---
ceph_custom_repositories:
  - name: ceph_custom_noarch
    state: present
    description: Ceph custom repo noarch
    gpgcheck: 'no'
    baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/noarch
    file: ceph_shaman_build_noarch
    priority: '2'
    enabled: 1
  - name: ceph_custom_x86_64
    state: present
    description: Ceph custom repo x86_64
    gpgcheck: 'no'
    baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/x86_64
    file: ceph_shaman_build_x86_64
    priority: '2'
    enabled: 1
shaman

Devel repository.

default

“community”

ceph_stable_key

description

URL to the gpg key.

default

https://download.ceph.com/keys/release.asc

ceph_release

description

The release of Ceph.

default

Corresponding Ceph release.

ceph_dev_branch

description

The development branch to be used in shaman when ceph_origin is ‘shaman’.

default

“main”

ceph_dev_sha1

description

The sha1 corresponding to the build to be used when ceph_origin is ‘shaman’.

default

“latest”

custom_repo_url

description

The url of the repository when ceph_origin is ‘custom’. Mutually exclusive with ceph_custom_repositories.

ceph_custom_repositories

This variable is a list. Mutually exclusive with custom_repo_url. The following options can be specified for each element that represents a repository to be set up:

name
description

The name of the repository.

gpgkey
description

The url of the gpg key corresponding to the repository being set up.

gpgcheck
description

Whether gpgcheck has to be performed.

state
description

Whether this repository has to be present or absent. (Default: present)

description
description

A short repository description

baseurl
description

The url of the repository pointing to the location where ‘repodata’ directory lives.

file
description

The filename Ansible will use to write the repository file.

priority
description

The priority of this repository.

components
description

This is a Debian OS-based family parameter only. The components of this ubuntu/debian repository.

Example:

ceph_custom_repositories:
  - name: ceph_custom_noarch
    state: present
    description: Ceph custom repo noarch
    gpgcheck: 'no'
    baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/noarch
    file: ceph_shaman_build_noarch
    priority: '2'
  - name: ceph_custom_x86_64
    state: present
    description: Ceph custom repo x86_64
    gpgcheck: 'no'
    baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/x86_64
    file: ceph_shaman_build_x86_64
    priority: '2'

set_insecure_registries

description

Whether cephadm-preflight.yml playbook will call cephadm-set-container-insecure-registries.yml to add an insecure registry in /etc/containers/registries.conf. insecure_registry option must be passed (-e insecure_registry=<registry url>)

default

false

cephadm-set-container-insecure-registries

This playbook adds a block in /etc/containers/registries.conf in order to allow an insecure registry to be used.

Usage:

ansible-playbook -i <inventory host file> cephadm-set-container-insecure-registries.yml -e insecure_registry=<registry url>

Options

insecure_registry

description

The address of the insecure registry to be added to /etc/containers/registries.conf.

default

No default.

cephadm-distribute-ssh-key

This playbook distributes an SSH public key over all hosts present in the inventory. The key to be copied will be read from a file specified at the path defined in cephadm_pubkey_path from the Ansible controller host. If cephadm_pubkey_path is unset, the playbook will assume it is supposed to get it from the command cephadm get-pub-key.

Usage:

ansible-playbook -i <inventory host file> cephadm-distribute-ssh-key.yml -e admin_node=ceph-node01 -e cephadm_pubkey_path=/home/cephadm/ceph.key

Options

fsid

description

The fsid of the Ceph cluster.

admin_node

description

The name of a node with enough privileges to call cephadm get-pub-key command. (usually the bootstrap node).

cephadm_ssh_user

description

The ssh username on remote hosts that will be used by cephadm.

cephadm_pubkey_path

description

Full path name of the ssh public key file on the ansible controller host.

cephadm-purge

This playbook purges a Ceph cluster managed with cephadm

You must define a group [admin] in your inventory with a node where the admin keyring is present at /etc/ceph/ceph.client.admin.keyring

Usage:

ansible-playbook -i <inventory host file> cephadm-purge-cluster.yml -e fsid=<your fsid>

Options

fsid

description

The fsid of the cluster.

cephadm-clients

If you plan to deploy client nodes, you must define a group called “clients” in your inventory:

$ cat hosts
node1
node2
node3

[clients]
client1
client2
client3
node123

This playbooks distribute keyring and conf files to a set of client hosts.

Usage:

ansible-playbook -i <inventory host file> cephadm-clients.yml -e fsid=<cluster fsid> -e keyring=<path to the keyring>

Options

keyring

description

The full path name of the keyring file on the host (which should be admin[0]) which holds the key for the client to use

conf

description

The full path name of the conf file on the (which should be admin[0]) host to use (undefined will trigger a minimal conf)

keyring_dest

description

The full path name of the destination where the keyring will be copied on the remote host. (default: /etc/ceph/ceph.keyring)

rocksdb-resharding

This playbook reshards the rocksDB database for a given OSD.

Usage:

ansible-playbook -i <inventory host file> rocksdb-resharding.yml -e osd_id=0 -e admin_node=ceph-mon0 -e rocksdb_sharding_parameters='m(3) p(3,0-12) O(3,0-13) L P'

Options

fsid

description

The fsid of the Ceph cluster.

osd_id

description

The id of the OSD where you want to reshard its corresponding rocksdb database.

admin_node

description

The name of a node with enough privileges to stop/start daemons via cephadm shell ceph orch daemon command. (Usually the bootstrap node)

rocksdb_sharding_parameters

description

The rocksdb sharding parameter to set. Default is ‘m(3) p(3,0-12) O(3,0-13) L P’.

docker

A boolean to be set in order to tell the playbook cephadm uses docker instead of podman as container engine. Default is False.

Modules

Introduction

cephadm-ansible provides several modules to make it easier to write playbooks around cephadm/ceph orch. The idea is to let you write your own playbooks, rather than providing a unique playbook that would try to cover anyone’s use case. This way you can have a solution that fits better with your needs.

At the moment only the most important tasks are supported. This means that any operation not covered would have to be done either with either the command or shell Ansible tasks in your playbook.

Module descriptions

cephadm_bootstrap

mon_ip

Ceph monitor IP address.

image

Ceph container image.

docker

Use docker instead of podman.

fsid

Ceph FSID.

pull

Pull the Ceph container image.

dashboard

Deploy the Ceph dashboard.

dashboard_user

Ceph dashboard user.

dashboard_password

Ceph dashboard password.

monitoring

Deploy the monitoring stack.

firewalld

Manage firewall rules with firewalld.

allow_overwrite

allow overwrite of existing -output-* config/keyring/ssh files.

registry_url

URL for custom registry.

registry_username

Username for custom registry.

registry_password

Password for custom registry.

registry_json

JSON file with custom registry login info (URL, username, password).

ssh_user

SSH user used for cephadm ssh to the hosts.

ssh_config

SSH config file path for cephadm ssh client.

allow_fqdn_hostname

Allow hostname that is fully-qualified.

cluster_network

Subnet to use for cluster replication, recovery and heartbeats.

ceph_orch_host

fsid

The fsid of the Ceph cluster to interact with.

image

Ceph container image.

name

name of the host to be added/removed/updated.

address

address of the host, required when state is present.

set_admin_label

enforce ‘_admin’ label on the host specified in ‘name’.

labels

list of labels to apply on the host.

state

If set to ‘present’, it will ensure the host specified in ‘name’ will be present along with the labels specified in labels. If set to ‘absent’, it will remove the host specified in ‘name’. If set to ‘drain’, it will schedule to remove all daemons from the host specified in ‘name’.

ceph_config

fsid

The fsid of the Ceph cluster to interact with.

image

Ceph container image.

action

Whether to get or set the parameter specified in ‘option’.

who

Which daemon the configuration should be set to.

option

Name of the parameter to be set.

value

Value of the parameter to set.

ceph_orch_apply

fsid

The fsid of the Ceph cluster to interact with.

image

Ceph container image.

spec

The service spec to apply.

ceph_orch_daemon

fsid

The fsid of the Ceph cluster to interact with.

image

Ceph container image.

state

The desired state of the service specified in ‘name’. If ‘started’, it ensures the service is started. If ‘stopped’, it ensures the service is stopped. If ‘restarted’, it will restart the service.

daemon_id

The id of the service.

daemon_type

The type of the service.

cephadm_registry_login

state

Whether the module should log in to the registry or log out.

registry_url

The container registry to log in or log out.

registry_username

The username to log in to the container registry.

registry_password

The corresponding password to be used with registry_username.

Samples

This shows how the supported modules can be used in a playbook. This doesn’t cover the pre-requisites steps (preflight, …) so it implies all requirements are satisfied (podman, lvm2,…). It assumes your “bootstrap host” (or “admin host”) can ssh to other hosts with root user without password.

Bootstrap and add some hosts:

# cat hosts
ceph-mon1 monitor_address=10.10.10.101 labels="['_admin', 'mon', 'mgr']"
ceph-mon2 labels="['mon', 'mgr']"
ceph-mon3 labels="['mon', 'mgr']"
ceph-osd1 labels="['osd']"
ceph-osd2 labels="['osd']"
ceph-osd3 labels="['osd']"
# cat site.yml
---
- name: bootstrap the cluster
  hosts: ceph-mon1
  become: true
  gather_facts: false
  tasks:
    - name: login to quay.io registry
      cephadm_registry_login:
        state: login
        registry_url: quay.io
        registry_username: foo
        registry_password: b4r

    - name: bootstrap initial cluster
      cephadm_bootstrap:
        mon_ip: "{{ monitor_address }}"

- name: add more hosts
  hosts: all
  become: true
  gather_facts: true
  tasks:
    - name: add hosts to the cluster
      ceph_orch_host:
        name: "{{ ansible_facts['hostname'] }}"
        address: "{{ ansible_facts['default_ipv4']['address'] }}"
        labels: "{{ labels }}"
      delegate_to: ceph-mon1

- name: deploy osd service
  hosts: ceph-mon1
  become: true
  gather_facts: false
  tasks:
    - name: apply osd spec
      ceph_orch_apply:
      spec: |
        service_type: osd
        service_id: osd
        placement:
          host_pattern: '*'
          label: osd
        spec:
          data_devices:
            all: true

- name: change osd_default_notify_timeout option
  hosts: ceph-mon1
  become: true
  gather_facts: false
  tasks:
    - name: decrease the value of osd_default_notify_timeout option
      ceph_config:
        action: set
        who: osd
        option: osd_default_notify_timeout
        value: 20

Note

You may have noticed that most of the time, the target node in the different plays in the playbook above is ceph-mon1, which is the bootstrap node.

Upcoming changes

Important

The name of the project might change in the next release.

Important

In the next release, this project will be distributed as an Ansible collection.