Notice

This document is for a development version of Ceph.

Zoned Storage Support

http://zonedstorage.io

Zoned Storage is a class of storage devices that enables host and storage devices to cooperate to achieve higher storage capacities, increased throughput, and lower latencies. The zoned storage interface is available through the SCSI Zoned Block Commands (ZBC) and Zoned Device ATA Command Set (ZAC) standards on Shingled Magnetic Recording (SMR) hard disks today and is also being adopted for NVMe Solid State Disks with the upcoming NVMe Zoned Namespaces (ZNS) standard.

This project aims to enable Ceph to work on zoned storage drives and at the same time explore research problems related to adopting this new interface. The first target is to enable non-overwrite workloads (e.g. RGW) on host-managed SMR (HM-SMR) drives and explore cleaning (garbage collection) policies. HM-SMR drives are high capacity hard drives with the ZBC/ZAC interface. The longer term goal is to support ZNS SSDs, as they become available, as well as overwrite workloads.

The first patch in these series enabled writing data to HM-SMR drives. This patch introduces ZonedFreelistManger, a FreelistManager implementation that passes enough information to ZonedAllocator to correctly initialize state of zones by tracking the write pointer and the number of dead bytes per zone. We have to introduce a new FreelistManager implementation because with zoned devices a region of disk can be in three states (empty, used, and dead), whereas current BitmapFreelistManager tracks only two states (empty and used). It is not possible to accurately initialize the state of zones in ZonedAllocator by tracking only two states. The third planned patch will introduce a rudimentary cleaner to form a baseline for further research.

Currently we can perform basic RADOS benchmarks on an OSD running on an HM-SMR drives, restart the OSD, and read the written data, and write new data, as can be seen below.

Please contact Abutalib Aghayev <agayev@psu.edu> for questions.

$ sudo zbd report -i -n /dev/sdc
Device /dev/sdc:
    Vendor ID: ATA HGST HSH721414AL T240
    Zone model: host-managed
    Capacity: 14000.520 GB (27344764928 512-bytes sectors)
    Logical blocks: 3418095616 blocks of 4096 B
    Physical blocks: 3418095616 blocks of 4096 B
    Zones: 52156 zones of 256.0 MB
    Maximum number of open zones: no limit
    Maximum number of active zones: no limit
52156 / 52156 zones
$ MON=1 OSD=1 MDS=0 sudo ../src/vstart.sh --new --localhost --bluestore --bluestore-devs /dev/sdc --bluestore-zoned
<snipped verbose output>
$ sudo ./bin/ceph osd pool create bench 32 32
pool 'bench' created
$ sudo ./bin/rados bench -p bench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_h0.cc.journaling712.narwhal.p_29846
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        45        29   115.943       116    0.384175    0.407806
    2      16        86        70   139.949       164    0.259845    0.391488
    3      16       125       109   145.286       156     0.31727    0.404727
    4      16       162       146   145.953       148    0.826671    0.409003
    5      16       203       187   149.553       164     0.44815    0.404303
    6      16       242       226   150.621       156    0.227488    0.409872
    7      16       281       265   151.384       156    0.411896    0.408686
    8      16       320       304   151.956       156    0.435135    0.411473
    9      16       359       343   152.401       156    0.463699    0.408658
   10      15       396       381   152.356       152    0.409554    0.410851
Total time run:         10.3305
Total writes made:      396
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     153.333
Stddev Bandwidth:       13.6561
Max bandwidth (MB/sec): 164
Min bandwidth (MB/sec): 116
Average IOPS:           38
Stddev IOPS:            3.41402
Max IOPS:               41
Min IOPS:               29
Average Latency(s):     0.411226
Stddev Latency(s):      0.180238
Max latency(s):         1.00844
Min latency(s):         0.108616
$ sudo ../src/stop.sh
$ # Notice the lack of "--new" parameter to vstart.sh
$ MON=1 OSD=1 MDS=0 sudo ../src/vstart.sh --localhost --bluestore --bluestore-devs /dev/sdc --bluestore-zoned
<snipped verbose output>
$ sudo ./bin/rados bench -p bench 10 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        61        45   179.903       180    0.117329    0.244067
    2      16       116       100   199.918       220    0.144162    0.292305
    3      16       174       158   210.589       232    0.170941    0.285481
    4      16       251       235   234.918       308    0.241175    0.256543
    5      16       316       300   239.914       260    0.206044    0.255882
    6      15       392       377   251.206       308    0.137972    0.247426
    7      15       458       443   252.984       264   0.0800146    0.245138
    8      16       529       513   256.346       280    0.103529    0.239888
    9      16       587       571   253.634       232    0.145535      0.2453
   10      15       646       631   252.254       240    0.837727    0.246019
Total time run:       10.272
Total reads made:     646
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   251.558
Average IOPS:         62
Stddev IOPS:          10.005
Max IOPS:             77
Min IOPS:             45
Average Latency(s):   0.249385
Max latency(s):       0.888654
Min latency(s):       0.0103208
$ sudo ./bin/rados bench -p bench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_h0.aa.journaling712.narwhal.p_64416
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        46        30   119.949       120     0.52627    0.396166
    2      16        82        66   131.955       144     0.48087    0.427311
    3      16       123       107   142.627       164      0.3287    0.420614
    4      16       158       142   141.964       140    0.405177    0.425993
    5      16       192       176   140.766       136    0.514565    0.425175
    6      16       224       208   138.635       128     0.69184    0.436672
    7      16       261       245   139.967       148    0.459929    0.439502
    8      16       301       285   142.468       160    0.250846    0.434799
    9      16       336       320   142.189       140    0.621686    0.435457
   10      16       374       358   143.166       152    0.460593    0.436384

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.