This document is for a development version of Ceph.
Deploying a new Ceph cluster¶
Cephadm creates a new Ceph cluster by “bootstrapping” on a single host, expanding the cluster to encompass any additional hosts, and then deploying the needed services.
Podman or Docker for running containers
Time synchronization (such as chrony or NTP)
LVM2 for provisioning storage devices
Any modern Linux distribution should be sufficient. Dependencies are installed automatically by the bootstrap process below.
cephadm command can
bootstrap a new cluster
launch a containerized shell with a working Ceph CLI, and
aid in debugging containerized Ceph daemons.
There are a few ways to install cephadm:
curlto fetch the most recent version of the standalone script.
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
This script can be run directly from the current directory:
Although the standalone script is sufficient to get a cluster started, it is convenient to have the
cephadmcommand installed on the host. To install the packages that provide the
cephadmcommand for the Octopus release, run the following commands:
./cephadm add-repo --release octopus ./cephadm install
cephadmis now in your PATH by running
which cephadmcommand will return this:
Some commercial Linux distributions (e.g., RHEL, SLE) may already include up-to-date Ceph packages. In that case, you can install cephadm directly. For example:
dnf install -y cephadm
zypper install -y cephadm
Bootstrap a new cluster¶
You need to know which IP address to use for the cluster’s first monitor daemon. This is normally just the IP for the first host. If there are multiple networks and interfaces, be sure to choose one that will be accessible by any host accessing the Ceph cluster.
ceph bootstrap command:
cephadm bootstrap --mon-ip *<mon-ip>*
This command will:
Create a monitor and manager daemon for the new cluster on the local host.
Generate a new SSH key for the Ceph cluster and adds it to the root user’s
Write a minimal configuration file needed to communicate with the new cluster to
Write a copy of the
client.adminadministrative (privileged!) secret key to
Write a copy of the public key to
The default bootstrap behavior will work for the vast majority of
users. See below for a few options that may be useful for some users,
cephadm bootstrap -h to see all available options:
Bootstrap writes the files needed to access the new cluster to
/etc/ceph, so that any Ceph packages installed on the host itself (e.g., to access the command line interface) can easily find them.
Daemon containers deployed with cephadm, however, do not need
/etc/cephat all. Use the
--output-dir *<directory>*option to put them in a different directory (like
.), avoiding any potential conflicts with existing Ceph configuration (cephadm or otherwise) on the same host.
You can pass any initial Ceph configuration options to the new cluster by putting them in a standard ini-style configuration file and using the
You can choose the ssh user cephadm will use to connect to hosts by using the
--ssh-user *<user>*option. The ssh key will be added to
/home/*<user>*/.ssh/authorized_keys. This user will require passwordless sudo access.
If you are using a container on an authenticated registry that requires login you may add the three arguments
--registry-url <url of registry>,
--registry-username <username of account on registry>,
--registry-password <password of account on registry>OR
--registry-json <json file with login info>. Cephadm will attempt to login to this registry so it may pull your container and then store the login info in its config database so other hosts added to the cluster may also make use of the authenticated registry.
Enable Ceph CLI¶
Cephadm does not require any Ceph packages to be installed on the
host. However, we recommend enabling easy access to the
command. There are several ways to do this:
cephadm shellcommand launches a bash shell in a container with all of the Ceph packages installed. By default, if configuration and keyring files are found in
/etc/cephon the host, they are passed into the container environment so that the shell is fully functional. Note that when executed on a MON host,
cephadm shellwill infer the
configfrom the MON container instead of using the default configuration. If
--mount <path>is given, then the host
<path>(file or directory) will appear under
/mntinside the container:
cephcommands, you can also run commands like this:
cephadm shell -- ceph -s
You can install the
ceph-commonpackage, which contains all of the ceph commands, including
mount.ceph(for mounting CephFS file systems), etc.:
cephadm add-repo --release octopus cephadm install ceph-common
Confirm that the
ceph command is accessible with:
Confirm that the
ceph command can connect to the cluster and also
its status with:
Add hosts to the cluster¶
To add each new host to the cluster, perform two steps:
Install the cluster’s public SSH key in the new host’s root user’s
ssh-copy-id -f -i /etc/ceph/ceph.pub root@*<new-host>*
ssh-copy-id -f -i /etc/ceph/ceph.pub root@host2 ssh-copy-id -f -i /etc/ceph/ceph.pub root@host3
Tell Ceph that the new node is part of the cluster:
ceph orch host add *newhost*
ceph orch host add host2 ceph orch host add host3
Deploy additional monitors (optional)¶
A typical Ceph cluster has three or five monitor daemons spread across different hosts. We recommend deploying five monitors if there are five or more nodes in your cluster.
When Ceph knows what IP subnet the monitors should use it can automatically deploy and scale monitors as the cluster grows (or contracts). By default, Ceph assumes that other monitors should use the same subnet as the first monitor’s IP.
If your Ceph monitors (or the entire cluster) live on a single subnet, then by default cephadm automatically adds up to 5 monitors as you add new hosts to the cluster. No further steps are necessary.
If there is a specific IP subnet that should be used by monitors, you can configure that in CIDR format (e.g.,
ceph config set mon public_network *<mon-cidr-network>*
ceph config set mon public_network 10.1.2.0/24
Cephadm deploys new monitor daemons only on hosts that have IPs configured in the configured subnet.
If you want to adjust the default of 5 monitors, run this command:
ceph orch apply mon *<number-of-monitors>*
To deploy monitors on a specific set of hosts, run this command:
ceph orch apply mon *<host1,host2,host3,...>*
Be sure to include the first (bootstrap) host in this list.
You can control which hosts the monitors run on by making use of host labels. To set the
monlabel to the appropriate hosts, run this command:
ceph orch host label add *<hostname>* mon
To view the current hosts and labels, run this command:
ceph orch host ls
ceph orch host label add host1 mon ceph orch host label add host2 mon ceph orch host label add host3 mon ceph orch host ls
HOST ADDR LABELS STATUS host1 mon host2 mon host3 mon host4 host5
Tell cephadm to deploy monitors based on the label by running this command:
ceph orch apply mon label:mon
You can explicitly specify the IP address or CIDR network for each monitor and control where it is placed. To disable automated monitor deployment, run this command:
ceph orch apply mon --unmanaged
To deploy each additional monitor:
ceph orch daemon add mon *<host1:ip-or-network1> [<host1:ip-or-network-2>...]*
For example, to deploy a second monitor on
newhost1using an IP address
10.1.2.123and a third monitor on
10.1.2.0/24, run the following commands:
ceph orch apply mon --unmanaged ceph orch daemon add mon newhost1:10.1.2.123 ceph orch daemon add mon newhost2:10.1.2.0/24
The apply command can be confusing. For this reason, we recommend using YAML specifications.
ceph orch apply moncommand supersedes the one before it. This means that you must use the proper comma-separated list-based syntax when you want to apply monitors to more than one host. If you do not use the proper syntax, you will clobber your work as you go.
ceph orch apply mon host1 ceph orch apply mon host2 ceph orch apply mon host3
This results in only one host having a monitor applied to it: host 3.
(The first command creates a monitor on host1. Then the second command clobbers the monitor on host1 and creates a monitor on host2. Then the third command clobbers the monitor on host2 and creates a monitor on host3. In this scenario, at this point, there is a monitor ONLY on host3.)
To make certain that a monitor is applied to each of these three hosts, run a command like this:
ceph orch apply mon "host1,host2,host3"
There is another way to apply monitors to multiple hosts: a
yamlfile can be used. Instead of using the “ceph orch apply mon” commands, run a command of this form:
ceph orch apply -i file.yaml
Here is a sample file.yaml file:
service_type: mon placement: hosts: - host1 - host2 - host3
An inventory of storage devices on all cluster hosts can be displayed with:
ceph orch device ls
A storage device is considered available if all of the following conditions are met:
The device must have no partitions.
The device must not have any LVM state.
The device must not be mounted.
The device must not contain a file system.
The device must not contain a Ceph BlueStore OSD.
The device must be larger than 5 GB.
Ceph refuses to provision an OSD on a device that is not available.
There are a few ways to create new OSDs:
Tell Ceph to consume any available and unused storage device:
ceph orch apply osd --all-available-devices
Create an OSD from a specific device on a specific host:
ceph orch daemon add osd *<host>*:*<device-path>*
ceph orch daemon add osd host1:/dev/sdb
Use OSD Service Specification to describe device(s) to consume based on their properties, such device type (SSD or HDD), device model names, size, or the hosts on which the devices exist:
ceph orch apply osd -i spec.yml
One or more MDS daemons is required to use the CephFS file system.
These are created automatically if the newer
ceph fs volume
interface is used to create a new file system. For more information,
see FS volumes and subvolumes.
ceph fs volume create <fs_name> --placement=""<placement spec>""
See Stateless services (MDS/RGW/NFS/rbd-mirror/iSCSI) for manually deploying MDS daemons.
Cephadm deploys radosgw as a collection of daemons that manage a particular realm and zone. (For more information about realms and zones, see Multi-Site.)
Note that with cephadm, radosgw daemons are configured via the monitor
configuration database instead of via a ceph.conf or the command line. If
that configuration isn’t already in place (usually in the
client.rgw.<realmname>.<zonename> section), then the radosgw
daemons will start up with default settings (e.g., binding to port
To deploy a set of radosgw daemons for a particular realm and zone, run the following command:
ceph orch apply rgw *<realm-name>* *<zone-name>* --placement="*<num-daemons>* [*<host1>* ...]"
For example, to deploy 2 rgw daemons serving the myorg realm and the us-east-1 zone on myhost1 and myhost2:
ceph orch apply rgw myorg us-east-1 --placement="2 myhost1 myhost2"
Cephadm will wait for a healthy cluster and automatically create the supplied realm and zone if they do not exist before deploying the rgw daemon(s)
Alternatively, the realm, zonegroup, and zone can be manually created using
radosgw-admin realm create --rgw-realm=<realm-name> --default
radosgw-admin zonegroup create --rgw-zonegroup=<zonegroup-name> --master --default
radosgw-admin zone create --rgw-zonegroup=<zonegroup-name> --rgw-zone=<zone-name> --master --default
radosgw-admin period update --rgw-realm=<realm-name> --commit
See Placement Specification for details of the placement specification.
Deploying NFS ganesha¶
Cephadm deploys NFS Ganesha using a pre-defined RADOS pool and optional namespace
To deploy a NFS Ganesha gateway, run the following command:
ceph orch apply nfs *<svc_id>* *<pool>* *<namespace>* --placement="*<num-daemons>* [*<host1>* ...]"
For example, to deploy NFS with a service id of foo, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:
ceph orch apply nfs foo nfs-ganesha nfs-ns
Create the nfs-ganesha pool first if it doesn’t exist.
See Placement Specification for details of the placement specification.