Notice

This document is for a development version of Ceph.

PG Merge Synchronization in Crimson

This document describes the infrastructure used to coordinate the merging of Placement Groups (PGs) in Crimson.

Background

Crimson utilizes a shared-nothing memory model where each PG is owned by a specific CPU core. Merging requires cross-core coordination while maintaining memory safety and epoch consistency.

Core Concepts

Migration Safety (Memory Ownership)

PGs are managed by their birth_shard. To prevent cross-shard deallocation crashes:

  1. Extraction: ShardServices::extract_pg() removes the source PG from its shard map so it stops receiving messages.

  2. Transportation: The PG crosses cores in a seastar::foreign_ptr.

  3. Storage: The target shard stores each source in crimson::local_shared_foreign_ptr<Ref<PG>> inside the target PG’s rendezvous map. When the target drops the last reference after PG::merge_from(), destruction is routed back to the source birth_shard.

Target and Source PG Synchronization

Merging involves a race between source PGs checking in and the target PG reaching the merge point in PGAdvanceMap.

Synchronization state lives on the target PG, not in a per-shard registry:

  • ``merge_rendezvous_t``: A map of arrived sources plus a seastar::semaphore counting first-time registrations.

  • ``PG::add_merge_source()``: Called on the target’s shard when ShardServices::register_merge_source() delivers a source PG. Duplicate source pgids are ignored (idempotent under replay).

  • ``PG::collect_merge_sources(n)``: Waits on the rendezvous semaphore for n arrivals (one signal per first insert), asserts the map has n entries, then returns sources and clears the rendezvous. If reset_merge_rendezvous() breaks the wait, returns an empty map.

  • ``PG::reset_merge_rendezvous()``: Clears in-flight handoffs and resets the semaphore. Called on PG::stop() or after a Seastore cross-shard abort so a failed try cannot leave stale sources for a later epoch.

Cross-shard handoff: register_merge_source() resolves the target shard, extracts the source locally, and uses invoke_on to call add_merge_source() on the target PG.

Map Advancement and Pipeline Stalling

To ensure the merge occurs between PGs at the same logical time:

  • PGAdvanceMap::check_for_merges() detects pg_num shrink between map epochs and returns a merge_result_t (source, target, or none).

  • merge_pg() performs merge-specific work only (Seastore eligibility, rendezvous collection, PG::merge_from() on the target).

  • A single PGAdvanceMap operation may batch multiple map epochs (from through to). Only a merge source stops the epoch loop early: finish_merge_source() commits rctx, stops the PG, and calls register_merge_source(). A merge target runs merge_from() at the merge epoch, then continues advancing through the remaining epochs up to to (matching classic OSD behavior). Normal completion at the end of start() activates the final map and commits rctx (also covering a completed merge target).

  • The target does not call merge_from() until sources frozen at the merge epoch have arrived on its rendezvous.

Seastore Cross-Shard Merges

Seastore does not support merging collections across reactor shards. Before a source hands off or a target collects sources, merge_pg() calls ShardServices::seastore_merge_shards_ok() with the full sibling set. If the target and any source map to different shards, the OSD aborts the attempt: clears ready-to-merge state, sends MOSDPGReadyToMerge{ ready=false } for the source so the monitor does not commit the unsafe pg_num decrement, resets the target rendezvous, and sends MOSDPGStopMerge to the monitor (once per pool per OSD). The monitor clamps pg_num_target to pg_num and clears FLAG_CRIMSON_ALLOW_PG_MERGE for that pool until operators re-enable merging (crimson_allow_pg_merge).

Cleanup

After complete_rctx(), the target releases its local_shared_foreign_ptr references; source PG objects are destroyed on their birth_shard.

When the OSD finishes consuming a new OSDMap in OSD::committed_osd_maps(), OSDSingletonState::prune_sent_ready_to_merge() drops sent_ready_to_merge_source entries (and stale pools_merge_stopped_reported pool ids) for PGs that no longer exist in the committed map, mirroring classic OSD::consume_map().

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.