ceph_test_rados — Model-Based RADOS Stress Test
================================================

``ceph_test_rados`` is a model-based integration test that verifies the
data correctness of the RADOS layer under stress. It maintains an in-memory
model of expected object data and metadata, and compares it against the
actual object data returned by RADOS after every read, detecting data
corruption, snapshot inconsistencies, and attribute mismatches.

.. note::

   This is **not** a performance benchmark. For throughput and latency
   measurement, use ``rados bench``. ``ceph_test_rados`` is a
   *correctness verifier*.

How It Works
------------

1. **Initialization**: Creates ``--objects`` initial objects via write
   (or append for EC pools).
2. **Stress loop**: Generates a randomized stream of up to ``--max-ops``
   operations, each selected by weighted probability from the
   ``--op`` arguments.
3. **Verification**: Every read dispatches 3 pipelined reads and
   compares data, xattrs, and omap entries against the in-memory model.
4. **Completion**: Prints the error count and per-operation-type
   statistics to stderr.

Architecture
~~~~~~~~~~~~

The tool is built from several components:

- ``TestRados.cc`` — CLI parsing, ``main()``, and the
  ``WeightedTestGenerator`` which selects operations by weight.
- ``RadosModel.h`` — The ``RadosTestContext`` (in-memory model) and all
  26 ``TestOp`` subclasses (``ReadOp``, ``WriteOp``, ``SnapCreateOp``,
  etc.).
- ``Object.h`` / ``Object.cc`` — Content generators
  (``VarLenGenerator``, ``AppendGenerator``) and the ``ObjectDesc``
  model that tracks layered object contents across snapshots.
- ``TestOpStat.h`` — Per-operation-type latency statistics collector.

Synopsis
--------

::

    ceph_test_rados
        --op <read|write|write_excl|writesame|delete|snap_create|snap_remove|
              rollback|setattr|rmattr|watch|copy_from|hit_set_list|is_dirty|
              undirty|cache_flush|cache_try_flush|cache_evict|append|append_excl|
              set_redirect|unset_redirect|chunk_read|tier_promote|tier_flush|
              set_chunk|tier_evict> <weight>
        [--op <operation_type> <weight> ...]
        [--pool <pool_name>]
        [--max-ops <op_count>]
        [--objects <object_count>]
        [--max-in-flight <max_concurrent>]
        [--size <max_size_bytes>]
        [--min-stride-size <bytes>]
        [--max-stride-size <bytes>]
        [--max-seconds <seconds>]
        [--ec-pool]
        [--no-omap]
        [--no-sparse]
        [--pool-snaps]
        [--balance-reads]
        [--localize-reads]
        [--offlen_randomization_ratio <0-100>]
        [--write-fadvise-dontneed]
        [--max-attr-len <bytes>]
        [--set_redirect]
        [--set_chunk]
        [--low_tier_pool <pool_name>]
        [--enable_dedup]
        [--dedup_chunk_algo <fastcdc|fixcdc>]
        [--dedup_chunk_size <bytes>]
        [--timestamps]

At least one ``--op`` with a positive weight is required.

Core Parameters
---------------

``--pool <name>``
    Target RADOS pool (must already exist). Default: ``rbd``.

``--max-ops <n>``
    Maximum number of operations to execute (including initial object
    writes). Default: ``1000``.

``--objects <n>``
    Number of distinct objects to create and test against. Must satisfy
    ``max_in_flight * 2 <= objects``. Default: ``50``.

``--max-in-flight <n>``
    Maximum concurrent asynchronous operations. Default: ``16``.

``--max-seconds <n>``
    Wall-clock time limit in seconds. ``0`` means unlimited (run until
    ``--max-ops`` is exhausted). Default: ``0``.

Object Geometry
---------------

``--size <n>``
    Maximum object size in bytes. Actual sizes are randomized within
    approximately ``[size/2, size]``. Default: ``4000000`` (~3.8 MiB).

``--min-stride-size <n>``
    Minimum write stride in bytes. Must be < ``--max-stride-size``
    and <= ``--size``. Default: ``size / 10``.

``--max-stride-size <n>``
    Maximum write stride in bytes. Must be > ``--min-stride-size``
    and <= ``--size``. Default: ``size / 5``.

Pool Type and Behavior
----------------------

``--ec-pool``
    Indicates that the target is an erasure-coded pool **that does not support overwrites**.
    **Must appear before any** ``--op`` **arguments.**
    
    .. note::

       This is largely a legacy parameter. When Ceph originally introduced
       EC pools, they did not support partial overwrites or sparse reads. Today,
       if an EC pool supports overwrites (e.g., via BlueStore), you should *not*
       use this flag, so that ``ceph_test_rados`` can test partial overwrites.
       In the Teuthology QA suite, setting ``erasure_code_use_overwrites: true``
       prevents the test runner from passing this flag.

    Using this flag has the following effects:
    
    1. Implicitly sets ``--no-sparse``.
    2. Initial object creation writes use ``append`` mode instead of ``write``.
    3. Overwrite operations (``write``, ``write_excl``, ``writesame``) are
       disallowed and will cause startup validation to fail.

``--no-omap``
    Disable omap operations. Automatically set if the pool does not
    support omap (auto-detected at startup).

``--no-sparse``
    Disable sparse reads (use full reads only). Automatically set when
    ``--ec-pool`` is used.

``--pool-snaps``
    Use pool-level snapshots instead of self-managed snapshots.

Read Routing
------------

``--balance-reads``
    Set ``LIBRADOS_OPERATION_BALANCE_READS`` on read operations,
    allowing reads from any replica.

``--localize-reads``
    Set ``LIBRADOS_OPERATION_LOCALIZE_READS`` on read operations,
    preferring the closest replica.

``--offlen_randomization_ratio <n>``
    Percentage chance (0–100) that a read uses a randomized offset
    instead of reading from offset 0. Default: ``50``.

Write Behavior
--------------

``--write-fadvise-dontneed``
    Set the ``write_fadvise_dontneed`` flag on the pool, advising the
    OSD backend not to cache written data.

``--max-attr-len <n>``
    Maximum generated xattr length in bytes. Default: ``20000``.

Manifest and Tiering
--------------------

``--set_redirect``
    Enable redirect manifest testing. Requires ``--low_tier_pool``.

``--set_chunk``
    Enable chunk-based manifest testing. Requires ``--low_tier_pool``.

``--low_tier_pool <name>``
    Low-tier pool for redirect/chunk/dedup operations. Must be a
    different pool from ``--pool`` to avoid a known race condition.
    Required when ``--set_redirect`` or ``--set_chunk`` is set.

Deduplication
-------------

``--enable_dedup``
    Enable deduplication testing. Requires ``--dedup_chunk_algo`` and
    ``--dedup_chunk_size``. Configures the pool with SHA-256
    fingerprinting and the specified chunking algorithm.

``--dedup_chunk_algo <algorithm>``
    Chunking algorithm: ``fastcdc`` or ``fixcdc``.

``--dedup_chunk_size <size>``
    Chunk size for content-defined chunking (e.g., ``131072``).

Output
------

``--timestamps``
    Prefix each output line with a coarse timestamp.

Operation Types
---------------

Operations are specified via ``--op <name> <weight>``. Weights are
relative: an operation with weight 100 is twice as likely as one with
weight 50.

.. list-table::
   :header-rows: 1
   :widths: 20 10 70

   * - Name
     - Valid with --ec-pool
     - Description
   * - ``read``
     - Yes
     - Read and verify object data, xattrs, and omap against the model.
   * - ``write``
     - No
     - Random-offset partial write.
   * - ``write_excl``
     - No
     - Random-offset partial write that asserts the object already exists
       (``assert_exists()``) as part of the transaction.
   * - ``writesame``
     - No
     - Write same data pattern across an extent.
   * - ``delete``
     - Yes
     - Delete an object.
   * - ``snap_create``
     - Yes
     - Create a snapshot (quiesces in-flight ops first).
   * - ``snap_remove``
     - Yes
     - Remove a snapshot.
   * - ``rollback``
     - Yes
     - Roll back an object to a previous snapshot.
   * - ``setattr``
     - Yes
     - Set random xattrs (and omap if supported).
   * - ``rmattr``
     - Yes
     - Remove random xattrs (and omap if supported).
   * - ``watch``
     - Yes
     - Establish a watch, self-notify, wait for callback.
   * - ``copy_from``
     - Yes
     - Server-side copy between objects in the pool.
   * - ``hit_set_list``
     - Yes
     - List HitSet entries.
   * - ``is_dirty``
     - Yes
     - Check object dirty state (cache tier).
   * - ``undirty``
     - Yes
     - Mark object clean (cache tier).
   * - ``cache_flush``
     - Yes
     - Flush object from cache tier (blocking).
   * - ``cache_try_flush``
     - Yes
     - Try to flush object from cache tier (non-blocking).
   * - ``cache_evict``
     - Yes
     - Evict object from cache tier.
   * - ``append``
     - Yes
     - Append data to an object.
   * - ``append_excl``
     - Yes
     - Append data that asserts the object already exists.
   * - ``set_redirect``
     - Yes
     - Set redirect manifest to low-tier pool.
   * - ``unset_redirect``
     - Yes
     - Remove redirect manifest.
   * - ``chunk_read``
     - Yes
     - Read and verify a chunk from a manifest object.
   * - ``tier_promote``
     - Yes
     - Promote object from lower tier.
   * - ``tier_flush``
     - Yes
     - Flush object to backing tier.
   * - ``set_chunk``
     - Yes
     - Set chunk manifest (requires ``--enable_dedup``).
   * - ``tier_evict``
     - Yes
     - Evict object to backing tier.

Environment Variables
---------------------

``CEPH_CLIENT_ID``
    Client ID for the librados connection. If unset, connects as the
    default client.

Standard Ceph environment variables (``CEPH_CONF``, ``CEPH_KEYRING``,
etc.) are respected.

Teuthology Integration
----------------------

The tool is typically invoked via the ``rados`` Teuthology task defined
in ``qa/tasks/rados.py``. The task creates pools, translates YAML
configuration into CLI arguments, and manages the process lifecycle.

Example YAML configuration::

    tasks:
    - rados:
        clients: [client.0]
        ops: 400000
        max_seconds: 600
        objects: 1024
        size: 16384
        op_weights:
          read: 100
          write: 100
          delete: 50
          snap_create: 50
          snap_remove: 50
          rollback: 50

Workload examples are in ``qa/suites/rados/thrash*/workloads/``.

.. note::

   The Teuthology wrapper automatically splits ``write`` and ``append``
   weights into regular and ``_excl`` halves. This does not happen at
   the CLI level: specify both variants explicitly when invoking the
   binary directly.

Examples
--------

Basic replicated pool test::

    ceph_test_rados \
      --pool testpool \
      --max-ops 10000 \
      --objects 500 \
      --max-in-flight 16 \
      --size 4000000 \
      --op read 100 \
      --op write 100 \
      --op delete 10

EC pool (without allow_ec_overwrites) with snapshots::

    ceph_test_rados \
      --ec-pool \
      --pool my-ec-pool \
      --max-ops 4000 \
      --objects 50 \
      --pool-snaps \
      --op read 100 \
      --op append 100 \
      --op delete 50 \
      --op snap_create 50 \
      --op snap_remove 50 \
      --op rollback 50

Deduplication test::

    ceph_test_rados \
      --pool testpool \
      --low_tier_pool low_tier \
      --set_chunk \
      --enable_dedup \
      --dedup_chunk_algo fastcdc \
      --dedup_chunk_size 131072 \
      --max-ops 1500 \
      --objects 50 \
      --op read 100 \
      --op write 50 \
      --op set_chunk 30 \
      --op tier_promote 10

Exit Status
-----------

The tool will immediately panic (via ``ceph_abort()``) and dump core
if any data verification errors (e.g., mismatching object content,
corrupt metadata) are detected during reads.

If no bugs are hit and the execution time/op count is exhausted, the
tool will exit cleanly with status **0**.

Exit status **1** indicates a startup validation failure (such as
incompatible arguments).

Source Files
------------

- ``src/test/osd/TestRados.cc`` — CLI parsing and main loop
- ``src/test/osd/RadosModel.h`` — Test context and operation classes
- ``src/test/osd/Object.h`` — Content generation and verification model
- ``src/test/osd/TestOpStat.h`` — Operation statistics
- ``qa/tasks/rados.py`` — Teuthology task wrapper