Introduction¶
cephadm-ansible is a collection of Ansible modules and playbooks to simplify workflows that are not covered by [cephadm]. They are covered by the following playbooks:
cephadm-preflight.yml: Initial setup of hosts before bootstrapping the cluster
cephadm-clients.yml: Setting up client hosts
cephadm-purge-cluster.yml: Remove a Ceph cluster
cephadm-distribute-ssh-key.yml: Distribute a SSH public key to all hosts
cephadm-set-container-insecure-registries.yml: Add a block in /etc/containers/registries.conf to add an insecure registry
Additionnally, several ansible modules are provided in order to let people writing their own playbooks.
Terminology¶
- admin host
A host where the admin keyring and ceph config file are present. Although the admin host and the bootstrap host are usually the same host, it is possible to have multiple admin hosts later.
cephadm
will make a host become ‘admin’ when the label_admin
is added to that host. (ie:ceph orch host label add <host> _admin
). This hosts should be present in the group[admin]
in the ansible inventory. If for some reason you decide a host shouldn’t be a ‘admin host’ anymore, you have to :remove it from the group
[admin]
in the ansible inventory,remove the admin keyring,
remove the ceph config file,
remove the
_admin
label. (ieceph orch host label rm <host> _admin
)
- ansible host
The host where any cephadm-ansible playbook is run.
- bootstrap host
The host where the ceph cluster will start. Unless you pass the
--skip-admin-label
option to theceph bootstram
command, this host will receive the admin keyring and the ceph config file and should be considered as an ‘admin host’. This host should be present in the group[admin]
in the ansible inventory.
Ansible inventory¶
The ansible inventory is a file where all the hosts intended to be part of the ceph cluster will be listed. The most common format are INI or YAML.
Although you probably want to keep it as simple as possible, you can organize your inventory and create groups, cephadm-ansible won’t differentiate except for the following requirements:
Client hosts must be defined in a dedicated group
[clients]
.Both
cephadm-purge-cluster.yml
andcephadm-clients.yml
playbooks require a group[admin]
with at least one admin host (usually it will be the bootstrap node).
Note
the name of the client group can be changed. In that case you have to set the variable client_group.
Otherwise, you can create groups such as [monitors]
, [osds]
, [rgws]
, that might help you keep clarity in your inventory file and ease the --limit
usage if you plan to use it to target specific a group of hosts only.
A basic inventory would look like following:
# cat hosts
ceph-mon1
ceph-mon2
ceph-mon3
ceph-osd1
ceph-osd2
ceph-osd3
ceph-mds1
ceph-mds2
ceph-rgw1
ceph-rgw2
[clients]
ceph-client1
ceph-client2
ceph-client3
[admin]
ceph-mon1
Playbooks¶
cephadm-preflight¶
This playbook configures the Ceph repository. It also installs some prerequisites (podman, lvm2, chronyd, cephadm, …)
Usage:
ansible-playbook -i <inventory host file> cephadm-preflight.yml
You can limit the execution to a set of hosts by using --limit
option:
ansible-playbook -i <inventory host file> cephadm-preflight.yml --limit <my_osd_group|my_node_name>
You can override variables using --extra-vars
parameter:
ansible-playbook -i <inventory host file> cephadm-preflight.yml --extra-vars "ceph_origin=rhcs"
Options¶
ceph_origin¶
- description
The source of Ceph repositories.
valid values
rhcs
Repository from Red Hat Ceph Storage.
community
Community repository (https://download.ceph.com)
custom
Custom repository. When
ceph_origin: custom
is defined, you have to set the variablecustom_repo_url
with the URL of your repository. Passing the extra-var-e custom_repo_state=absent
allows you to remove this repository later.It also supports deploying multiple repositories, in that case you must set the variable
ceph_custom_repositories
instead.ceph_custom_repositories
is a dictionnary that should look like following:ceph_custom_repositories: - name: ceph_custom_noarch state: present description: Ceph custom repo noarch gpgcheck: 'no' baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/noarch file: ceph_shaman_build_noarch priority: '2' enabled: 1 - name: ceph_custom_x86_64 state: present description: Ceph custom repo x86_64 gpgcheck: 'no' baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/x86_64 file: ceph_shaman_build_x86_64 priority: '2' enabled: 1
Given that the definition is more complex, you might want to define it as a group_vars/host_vars rather than as an extra-var:
$ cat group_vars/all --- ceph_custom_repositories: - name: ceph_custom_noarch state: present description: Ceph custom repo noarch gpgcheck: 'no' baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/noarch file: ceph_shaman_build_noarch priority: '2' enabled: 1 - name: ceph_custom_x86_64 state: present description: Ceph custom repo x86_64 gpgcheck: 'no' baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/x86_64 file: ceph_shaman_build_x86_64 priority: '2' enabled: 1
shaman
Devel repository.
- default
“community”
ceph_stable_key¶
- description
URL to the gpg key.
- default
ceph_release¶
- description
The release of Ceph.
- default
Corresponding Ceph release.
ceph_dev_branch¶
- description
The development branch to be used in shaman when ceph_origin is ‘shaman’.
- default
“main”
ceph_dev_sha1¶
- description
The sha1 corresponding to the build to be used when ceph_origin is ‘shaman’.
- default
“latest”
custom_repo_url¶
- description
The url of the repository when
ceph_origin
is ‘custom’. Mutually exclusive withceph_custom_repositories
.
ceph_custom_repositories¶
This variable is a list.
Mutually exclusive with custom_repo_url
.
The following options can be specified for each element that represents a repository to be set up:
name¶
- description
The name of the repository.
gpgkey¶
- description
The url of the gpg key corresponding to the repository being set up.
gpgcheck¶
- description
Whether gpgcheck has to be performed.
state¶
- description
Whether this repository has to be present or absent. (Default: present)
description¶
- description
A short repository description
baseurl¶
- description
The url of the repository pointing to the location where ‘repodata’ directory lives.
file¶
- description
The filename Ansible will use to write the repository file.
priority¶
- description
The priority of this repository.
components¶
- description
This is a Debian OS-based family parameter only. The components of this ubuntu/debian repository.
Example:
ceph_custom_repositories:
- name: ceph_custom_noarch
state: present
description: Ceph custom repo noarch
gpgcheck: 'no'
baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/noarch
file: ceph_shaman_build_noarch
priority: '2'
- name: ceph_custom_x86_64
state: present
description: Ceph custom repo x86_64
gpgcheck: 'no'
baseurl: https://4.chacra.ceph.com/r/ceph/main/cf17ed16c3964b635e9b6c22e607ea5672341c5c/centos/8/flavors/default/x86_64
file: ceph_shaman_build_x86_64
priority: '2'
set_insecure_registries¶
- description
Whether
cephadm-preflight.yml
playbook will callcephadm-set-container-insecure-registries.yml
to add an insecure registry in/etc/containers/registries.conf
.insecure_registry
option must be passed (-e insecure_registry=<registry url>)- default
false
cephadm-set-container-insecure-registries¶
This playbook adds a block in /etc/containers/registries.conf
in order to allow an insecure registry to be used.
Usage:
ansible-playbook -i <inventory host file> cephadm-set-container-insecure-registries.yml -e insecure_registry=<registry url>
Options¶
insecure_registry¶
- description
The address of the insecure registry to be added to
/etc/containers/registries.conf
.- default
No default.
cephadm-distribute-ssh-key¶
This playbook distributes an SSH public key over all hosts present in the inventory.
The key to be copied will be read from a file specified at the path defined in cephadm_pubkey_path
from the Ansible controller host.
If cephadm_pubkey_path
is unset, the playbook will assume it is supposed to get it from the command cephadm get-pub-key
.
Usage:
ansible-playbook -i <inventory host file> cephadm-distribute-ssh-key.yml -e admin_node=ceph-node01 -e cephadm_pubkey_path=/home/cephadm/ceph.key
Options¶
fsid¶
- description
The fsid of the Ceph cluster.
admin_node¶
- description
The name of a node with enough privileges to call cephadm get-pub-key command. (usually the bootstrap node).
cephadm_ssh_user¶
- description
The ssh username on remote hosts that will be used by
cephadm
.
cephadm_pubkey_path¶
- description
Full path name of the ssh public key file on the ansible controller host.
cephadm-purge¶
This playbook purges a Ceph cluster managed with cephadm
You must define a group [admin]
in your inventory with a node where
the admin keyring is present at /etc/ceph/ceph.client.admin.keyring
Usage:
ansible-playbook -i <inventory host file> cephadm-purge-cluster.yml -e fsid=<your fsid>
Options¶
fsid¶
- description
The fsid of the cluster.
cephadm-clients¶
If you plan to deploy client nodes, you must define a group called “clients” in your inventory:
$ cat hosts
node1
node2
node3
[clients]
client1
client2
client3
node123
This playbooks distribute keyring and conf files to a set of client hosts.
Usage:
ansible-playbook -i <inventory host file> cephadm-clients.yml -e fsid=<cluster fsid> -e keyring=<path to the keyring>
Options¶
keyring¶
- description
The full path name of the keyring file on the host (which should be admin[0]) which holds the key for the client to use
conf¶
- description
The full path name of the conf file on the (which should be admin[0]) host to use (undefined will trigger a minimal conf)
keyring_dest¶
- description
The full path name of the destination where the keyring will be copied on the remote host. (default: /etc/ceph/ceph.keyring)
rocksdb-resharding¶
This playbook reshards the rocksDB database for a given OSD.
Usage:
ansible-playbook -i <inventory host file> rocksdb-resharding.yml -e osd_id=0 -e admin_node=ceph-mon0 -e rocksdb_sharding_parameters='m(3) p(3,0-12) O(3,0-13) L P'
Options¶
fsid¶
- description
The fsid of the Ceph cluster.
osd_id¶
- description
The id of the OSD where you want to reshard its corresponding rocksdb database.
admin_node¶
- description
The name of a node with enough privileges to stop/start daemons via cephadm shell ceph orch daemon command. (Usually the bootstrap node)
rocksdb_sharding_parameters¶
- description
The rocksdb sharding parameter to set. Default is ‘m(3) p(3,0-12) O(3,0-13) L P’.
docker¶
A boolean to be set in order to tell the playbook cephadm uses
docker
instead ofpodman
as container engine. Default isFalse
.
Modules¶
Introduction¶
cephadm-ansible provides several modules to make it easier to write playbooks around cephadm/ceph orch. The idea is to let you write your own playbooks, rather than providing a unique playbook that would try to cover anyone’s use case. This way you can have a solution that fits better with your needs.
At the moment only the most important tasks are supported.
This means that any operation not covered would have to be done either with either the command
or shell
Ansible tasks in your playbook.
Module descriptions¶
cephadm_bootstrap¶
mon_ip
Ceph monitor IP address.
image
Ceph container image.
docker
Use docker instead of podman.
fsid
Ceph FSID.
pull
Pull the Ceph container image.
dashboard
Deploy the Ceph dashboard.
dashboard_user
Ceph dashboard user.
dashboard_password
Ceph dashboard password.
monitoring
Deploy the monitoring stack.
firewalld
Manage firewall rules with firewalld.
allow_overwrite
allow overwrite of existing -output-* config/keyring/ssh files.
registry_url
URL for custom registry.
registry_username
Username for custom registry.
registry_password
Password for custom registry.
registry_json
JSON file with custom registry login info (URL, username, password).
ssh_user
SSH user used for cephadm ssh to the hosts.
ssh_config
SSH config file path for cephadm ssh client.
allow_fqdn_hostname
Allow hostname that is fully-qualified.
cluster_network
Subnet to use for cluster replication, recovery and heartbeats.
ceph_orch_host¶
fsid
The fsid of the Ceph cluster to interact with.
image
Ceph container image.
name
name of the host to be added/removed/updated.
address
address of the host, required when
state
ispresent
.set_admin_label
enforce ‘_admin’ label on the host specified in ‘name’.
labels
list of labels to apply on the host.
state
If set to ‘present’, it will ensure the host specified in ‘name’ will be present along with the labels specified in
labels
. If set to ‘absent’, it will remove the host specified in ‘name’. If set to ‘drain’, it will schedule to remove all daemons from the host specified in ‘name’.
ceph_config¶
fsid
The fsid of the Ceph cluster to interact with.
image
Ceph container image.
action
Whether to get or set the parameter specified in ‘option’.
who
Which daemon the configuration should be set to.
option
Name of the parameter to be set.
value
Value of the parameter to set.
ceph_orch_apply¶
fsid
The fsid of the Ceph cluster to interact with.
image
Ceph container image.
spec
The service spec to apply.
ceph_orch_daemon¶
fsid
The fsid of the Ceph cluster to interact with.
image
Ceph container image.
state
The desired state of the service specified in ‘name’. If ‘started’, it ensures the service is started. If ‘stopped’, it ensures the service is stopped. If ‘restarted’, it will restart the service.
daemon_id
The id of the service.
daemon_type
The type of the service.
cephadm_registry_login¶
state
Whether the module should log in to the registry or log out.
registry_url
The container registry to log in or log out.
registry_username
The username to log in to the container registry.
registry_password
The corresponding password to be used with
registry_username
.
Samples¶
This shows how the supported modules can be used in a playbook. This doesn’t cover the pre-requisites steps (preflight, …) so it implies all requirements are satisfied (podman, lvm2,…). It assumes your “bootstrap host” (or “admin host”) can ssh to other hosts with root user without password.
Bootstrap and add some hosts:
# cat hosts
ceph-mon1 monitor_address=10.10.10.101 labels="['_admin', 'mon', 'mgr']"
ceph-mon2 labels="['mon', 'mgr']"
ceph-mon3 labels="['mon', 'mgr']"
ceph-osd1 labels="['osd']"
ceph-osd2 labels="['osd']"
ceph-osd3 labels="['osd']"
# cat site.yml
---
- name: bootstrap the cluster
hosts: ceph-mon1
become: true
gather_facts: false
tasks:
- name: login to quay.io registry
cephadm_registry_login:
state: login
registry_url: quay.io
registry_username: foo
registry_password: b4r
- name: bootstrap initial cluster
cephadm_bootstrap:
mon_ip: "{{ monitor_address }}"
- name: add more hosts
hosts: all
become: true
gather_facts: true
tasks:
- name: add hosts to the cluster
ceph_orch_host:
name: "{{ ansible_facts['hostname'] }}"
address: "{{ ansible_facts['default_ipv4']['address'] }}"
labels: "{{ labels }}"
delegate_to: ceph-mon1
- name: deploy osd service
hosts: ceph-mon1
become: true
gather_facts: false
tasks:
- name: apply osd spec
ceph_orch_apply:
spec: |
service_type: osd
service_id: osd
placement:
host_pattern: '*'
label: osd
spec:
data_devices:
all: true
- name: change osd_default_notify_timeout option
hosts: ceph-mon1
become: true
gather_facts: false
tasks:
- name: decrease the value of osd_default_notify_timeout option
ceph_config:
action: set
who: osd
option: osd_default_notify_timeout
value: 20
Note
You may have noticed that most of the time, the target node in the different plays in the playbook above is ceph-mon1
, which is the bootstrap node.
Upcoming changes¶
Important
The name of the project might change in the next release.
Important
In the next release, this project will be distributed as an Ansible collection.