.. _cephfs_metrics: Metrics ======= CephFS uses :ref:`Perf Counters` to track metrics. The counters can be labeled (:ref:`Labeled Perf Counters`). Client Metrics -------------- CephFS exports client metrics as :ref:`Labeled Perf Counters`, which could be used to monitor the client performance. CephFS exports the below client metrics. .. list-table:: Client Metrics :widths: 25 25 75 :header-rows: 1 * - Name - Type - Description * - num_clients - Gauge - Number of client sessions * - cap_hits - Gauge - Percentage of file capability hits over total number of caps * - cap_miss - Gauge - Percentage of file capability misses over total number of caps * - avg_read_latency - Gauge - Mean value of the read latencies * - avg_write_latency - Gauge - Mean value of the write latencies * - avg_metadata_latency - Gauge - Mean value of the metadata latencies * - dentry_lease_hits - Gauge - Percentage of dentry lease hits handed out over the total dentry lease requests * - dentry_lease_miss - Gauge - Percentage of dentry lease misses handed out over the total dentry lease requests * - opened_files - Gauge - Number of opened files * - opened_inodes - Gauge - Number of opened inodes * - pinned_icaps - Gauge - Number of pinned Inode Caps * - total_inodes - Gauge - Total number of Inodes * - total_read_ops - Gauge - Total number of read operations generated by all processes * - total_read_size - Gauge - Number of bytes read in input/output operations generated by all processes * - total_write_ops - Gauge - Total number of write operations generated by all processes * - total_write_size - Gauge - Number of bytes written in input/output operations generated by all processes MDS Rank Metrics ---------------- Per-MDS-rank metrics are also exported as :ref:`Labeled Perf Counters` with a ``rank`` label. These describe the MDS daemon process itself, not any particular client or subvolume. .. list-table:: MDS Rank Metrics :widths: 25 25 75 :header-rows: 1 * - Name - Type - Description * - ``cpu_usage`` - Gauge - Sum of per-core CPU utilisation for the MDS process (``100`` == one fully saturated core; values can exceed 100 on multi-core systems). * - ``open_requests`` - Gauge - Number of metadata requests currently in flight for this MDS rank. Subvolume Metrics ----------------- CephFS exports subvolume metrics as :ref:`Labeled Perf Counters`, which can be used to monitor subvolume performance and utilization. **I/O Performance Metrics** I/O performance metrics (IOPS, throughput, latency) are aggregated within a sliding window of 30 seconds by default. This interval is configurable via the ``subv_metrics_window_interval`` parameter (see :ref:`MDS config reference`). In large clusters with tens of thousands of subvolumes, this parameter also controls when stale metrics are evicted: once the sliding window becomes empty (no I/O activity), the metrics entry is removed rather than reporting zeros, reducing memory usage and computational overhead. .. important:: **Metadata operations do NOT trigger metric updates.** Only actual data I/O (reads and writes to file contents) updates the sliding window and keeps the subvolume metrics entry active. Metadata-only operations such as ``mkdir``, ``rmdir``, ``unlink``, ``rename``, ``chmod``, ``chown``, ``setxattr``, ``stat``, and ``ls`` do **not** generate I/O metrics. This means: - If a subvolume has only metadata activity (e.g., creating/deleting files without writing data), its I/O metrics will show zeros or the entry may be evicted after the window expires. - After deleting files, the ``used_bytes`` value will not immediately reflect the freed space until either new data I/O occurs or the MDS broadcasts updated quota information. **Utilization Metrics** In addition to I/O performance, subvolume metrics include utilization counters: - ``quota_bytes``: The configured quota limit for the subvolume (0 if unlimited). - ``used_bytes``: Current space usage based on the inode's recursive statistics (``rstat.rbytes``). These values are updated when the MDS broadcasts quota information to clients. The ``used_bytes`` reflects the recursive byte count of the subvolume root inode, which is maintained by the MDS as files are created, modified, or deleted. However, since metric reporting depends on I/O activity to keep entries alive, the utilization values are only reported while the subvolume has active I/O within the sliding window. .. list-table:: Subvolume Metrics :widths: 25 25 75 :header-rows: 1 * - Name - Type - Description * - ``avg_read_iops`` - Gauge - Average read IOPS (input/output operations per second) over the sliding window. * - ``avg_read_tp_Bps`` - Gauge - Average read throughput in bytes per second. * - ``avg_read_lat_msec`` - Gauge - Average read latency in milliseconds. * - ``avg_write_iops`` - Gauge - Average write IOPS over the sliding window. * - ``avg_write_tp_Bps`` - Gauge - Average write throughput in bytes per second. * - ``avg_write_lat_msec`` - Gauge - Average write latency in milliseconds. * - ``quota_bytes`` - Gauge - Configured quota limit in bytes (0 if no quota/unlimited). * - ``used_bytes`` - Gauge - Current space usage in bytes (recursive byte count of subvolume root). Getting Metrics =============== The metrics could be scraped from the MDS admin socket as well as using the tell interface. The ``mds_client_metrics-`` section in the output of ``counter dump`` command displays the metrics for each client as shown below:: "mds_client_metrics": [ { "labels": { "fs_name": "", "id": "14213" }, "counters": { "num_clients": 2 } } ], "mds_client_metrics-": [ { "labels": { "client": "client.0", "rank": "0" }, "counters": { "cap_hits": 5149, "cap_miss": 1, "avg_read_latency": 0.000000000, "avg_write_latency": 0.000000000, "avg_metadata_latency": 0.000000000, "dentry_lease_hits": 0, "dentry_lease_miss": 0, "opened_files": 1, "opened_inodes": 2, "pinned_icaps": 2, "total_inodes": 2, "total_read_ops": 0, "total_read_size": 0, "total_write_ops": 4836, "total_write_size": 633864192 } }, { "labels": { "client": "client.1", "rank": "0" }, "counters": { "cap_hits": 3375, "cap_miss": 8, "avg_read_latency": 0.000000000, "avg_write_latency": 0.000000000, "avg_metadata_latency": 0.000000000, "dentry_lease_hits": 0, "dentry_lease_miss": 0, "opened_files": 1, "opened_inodes": 2, "pinned_icaps": 2, "total_inodes": 2, "total_read_ops": 0, "total_read_size": 0, "total_write_ops": 3169, "total_write_size": 415367168 } } ] The subvolume metrics are dumped as a part of the same command. The ``mds_subvolume_metrics`` section in the output of ``counter dump`` command displays the metrics for each subvolume as shown below:: "mds_subvolume_metrics": [ { "labels": { "fs_name": "a", "subvolume_path": "/volumes/_nogroup/test_subvolume" }, "counters": { "avg_read_iops": 0, "avg_read_tp_Bps": 11, "avg_read_lat_msec": 0, "avg_write_iops": 1564, "avg_write_tp_Bps": 6408316, "avg_write_lat_msec": 338, "quota_bytes": 10737418240, "used_bytes": 1073741824 } }