Notice

This document is for a development version of Ceph.

Report a Documentation Bug

OSD developer documentation

Contents

• Asynchronous Recovery
• Backfill Reservation
• ceph_test_rados — Model-Based RADOS Stress Test
  • How It Works
    • Architecture
  • Synopsis
  • Core Parameters
  • Object Geometry
  • Pool Type and Behavior
  • Read Routing
  • Write Behavior
  • Manifest and Tiering
  • Deduplication
  • Output
  • Operation Types
  • Environment Variables
  • Teuthology Integration
  • Examples
  • Exit Status
  • Source Files
• Erasure Coded Placement Groups
  • Glossary
  • Table of contents
    • Developer notes
      • Introduction
      • Reading and writing encoded chunks from and to OSDs
      • Erasure code library
      • Notes
    • Jerasure plugin
      • Introduction
    • ECBackend Implementation Strategy
      • Miscellaneous initial design notes
        • PGTemp and Crush
        • Distinguished acting set positions
        • Object Classes
        • Scrub
        • Crush
    • ECBackend
      • MAIN OPERATION OVERVIEW
        • WHOLE STRIPE WRITE
        • READ-MODIFY-WRITE
        • OSD Object Write and Consistency
        • ExtentCache
        • Pipeline
    • Erasure coding enhancements design document
      • Objectives
        • Current Read Implementation
        • Partial Reads
        • Current Overwrite Implementation
        • Partial Overwrites
        • Parity-delta-write
        • Direct Read I/O
        • Distributed processing of writes
        • Direct Write I/O
        • Larger chunk size
      • Design
        • Test tools
        • I/O path changes
          • Avoid unnecessary reads and writes
          • Parity-delta-write
          • Direct reads
          • Direct writes
            • I/O serialization, recovery/backfill and other error scenarios
          • Stripe cache
          • Automatically choose chunk size
          • Deep scrub support
        • Metadata changes
          • How many copies of metadata are required?
          • M+1 arbitrarily distributed copies
          • M+1 copies on known shards
          • Partial Writes and the PG log
          • Backfill
            • Code references
          • Version number and last modification time for clients
          • Direct Write with Metadata updates
        • Upgrade and backwards compatibility
        • Not under consideration
      • Stories
        • Test tools - enhanced I/O generator for testing erasure coding
        • Test tools - offline consistency checking tool
        • Test tools - online consistency checking tool
        • Switch for JErasure to ISA-L
        • Sub Stripe Reads
        • Simple Optimizations to Overwrite
        • Eliminate unnecessary chunk writes but keep metadata transactions
        • Avoid zero padding objects to a full stripe
        • Erasure coding plugin changes to support distributed partial writes
        • Erasure coding interface to allow RADOS clients to direct I/Os to OSD storing the data
        • Changes to object_info_t
        • Changes to PGLog and Peering to support updating a subset of OSDs
        • Change to selection of (acting) primary
        • Implement parity-delta-write with all computation on the primary
        • Upgrades and backwards compatibility
        • Changes to Backfill to use the vector in object_info_t
        • Test tools - offline metadata validation tool
        • Eliminate transactions on OSDs not updating data chunks
        • Direct reads to OSDs (single chunk only)
        • Direct reads to OSDs (multiple chunks)
        • Implement distributed parity-delta-write
        • Test tools - EC error injection thrasher
        • Implement prefetch message MOSDEcSubOpPrefetch and modify extent cache
        • Implement sequencing message MOSDEcSubOpSequence
        • Direct writes to OSD (single chunk only)
        • Direct writes to OSD (multiple chunks)
        • Deep scrub / CRC
        • Variable chunk size erasure coding
        • CLAY Erasure Codes
    • Direct reads design document
      • Basic design
        • Small reads
        • Medium sized reads
        • Large size reads
      • Objecter
      • EC Backend
      • Missing objects
      • Torn Writes
      • Changes to default chunk sizes
      • Read-write ordering
      • Read Instability
      • Kernel Client Modifications
      • Performance, QoS and throttling
        • CPU Load
        • Throttling Considerations
      • Replica Balanced Reads
        • Splitting Reads
      • Plugin Support
      • Supported Ops
        • Sparse Reads
      • Object (user) Version
      • Testing
        • ceph_test_rados_io_sequence
          • Small vs Medium vs Large Reads
          • Objecter
          • Stale Maps/Fallback path
          • Missing Objects
          • Read/Write Ordering
          • Kernel level testing
          • Miscellaneous
        • ceph_test_rados
    • EC Stretch Cluster design document
      • 1. Intent & High-Level Summary
      • 2. Proposed Configuration (CLI) Changes
        • 2.1 ceph osd pool create
          • 2.2.1 Full command syntax
          • 2.2.2 Legacy positional syntax
          • 2.2.1 Basic Parameters
          • 2.2.2 Advanced
          • 2.2.3 Deprecated
        • 2.3 Examples
      • 1. Approaches Considered but Rejected
        • 3.1 Multi-layered Backends
        • 3.2 LRC (Locally Repairable Codes) Plugin
      • 4. Configuration Logic (Preliminary)
      • 5. Multi-Zone & Topology Considerations
        • 5.1 Single-Zone Pools
      • 6. Topologies and Terminology
        • 6.1 Logical Diagram (2-Zone HA)
        • 6.2 Key Definitions
      • 7. Read Path & Recovery Strategies
        • 7.1 Read from Primary — R1
          • 7.1.1 Zone-local Recovery
          • 7.1.2 Zone-Local Recovery
        • 7.2 Direct Reads — Primary Zone — R1
        • 7.3 Direct Reads — Zone-Local — R1
        • 7.4 Read from Zone Primary — Later Release
          • 7.4.1 Safe Shard Identification & Synchronous Recovery — Later Release
      • 8. Write Path & Transaction Handling
        • 8.1 Direct-to-OSD Writes (Primary Encodes All Shards) — R1
        • 8.2 Replicate Transaction (Preferred Path) — Later Release
        • 8.3 Client Writes Direct to Zone Primary — Later Release
        • 8.4 Hybrid Approach — Later Release
      • 9. Recovery Logic
        • 9.1 Primary-Centric Recovery — R1
        • 9.2 Cost-Based Recovery Planning — Later Release
        • 9.3 Delegated Remote-zone Recovery — Later Release
        • 9.4 Fallback Strategies — Later Release
          • 9.4.1 Remote-zone Fallback (Push Recovery)
          • 9.4.2 Primary Domain Fallback
      • 10. PG Log Handling — R1
        • 10.1 Primary-Capable vs. Non-Primary Shards
        • 10.2 Independent Log Generation at Zone Primary — Later Release
        • 10.3 Log Entry Compatibility & Upgrade Safety — Later Release
      • 11. Peering & Stretch Mode — R1
        • 11.1 Pool Lifecycle
        • 11.2 Minimum PG Size Semantics
          • 11.2.1 Per-Pool Min-Size Interpretations — R1
          • 11.2.2 Per-Zone Min-Size Interpretations — Later Release
        • 11.3 Stretch Mode State Machine
        • 11.4 OSDMonitor Changes
        • 11.5 Peering State Changes
        • 11.6 Relationship to peering_crush_bucket_* Fields
        • 11.7 Network Partition Handling
        • 11.8 Online OSDs in Offline Zones
      • 12. Scrub Behavior — R1
      • 13. Migration Path
        • 13.1 Pool Migration — R1
        • 13.2 In-Place Replica Count Modification — Later Release
      • 14. Implementation Order
        • 14.1 R1 — Read-Optimized Stretch Cluster
        • 14.2 Later Release — Recovery Bandwidth Optimization
        • 14.3 Later Release — Write Bandwidth Optimization
        • 14.4 Later Release — Additional Enhancements
      • 15. Upgrade & Backward Compatibility
      • 16. Kernel Changes
      • 17. Inter-Zone Statistics — R1
      • 18. Non-Redundant Pool Zone Affinity
        • Implementation Details
• last_epoch_started
• Log Based PG
  • Background
    • Why PrimaryLogPG?
    • Primary log-based replication
  • ReplicatedBackend/ECBackend unification strategy
    • PGBackend
  • PGBackend Interface Explanation
    • Readable vs Degraded
    • Client Reads
    • Scrubs
    • Recovery
• Manifest
  • Introduction
  • Intended Usage Model
    • RBD
    • RadosGW
  • Status and Future Work
    • Cleanups
    • Testing
    • Snapshots
  • Data Structures
  • Request Handling
  • RADOS Interface
  • ceph-dedup-tool
• Map and PG Message handling
  • Overview
  • MOSDMap
  • MOSDPGOp/MOSDPGSubOp
  • CEPH_MSG_OSD_OP processing
  • Peering Messages
• QoS Study with mClock and WPQ Schedulers
  • Introduction
  • Overview
  • Test Environment
  • Test Methodology
  • Establish Baseline Client Throughput (IOPS)
  • MClock Profile Allocations
    • high_client_ops(default)
    • balanced
    • high_recovery_ops
    • custom
  • Recovery Test Steps
    • Test Steps(Using cbt)
    • Non-Default Ceph Recovery Options
  • Test Results
    • Test Results With NVMe SSDs
      • Client Throughput Comparison
      • Client Latency Comparison
      • Recovery Statistics Comparison
    • Test Results With HDDs (WAL and dB configured)
      • Client Throughput & latency Comparison
      • Recovery Statistics Comparison
    • Test Results With HDDs (No WAL and dB configured)
      • Client Throughput & latency Comparison
      • Recovery Statistics Comparison
  • Key Takeaways and Conclusion
• OSD
  • Concepts
  • Overview
• Partial Object Recovery
  • Procedures for Partial Object Recovery
• OSDMap Trimming and PastIntervals
  • PastIntervals
  • Maintenance and Trimming
  • OSDMap Trimming
  • Back to PastIntervals
  • OSDSuperblock::maps
• PG
  • Concepts
  • Peering Details and Gotchas
• PG Removal
• PGPool
• Recovery Reservation
  • Dump Reservations
  • Things to Note
  • See Also
• Refcount
  • Introduction
  • Class Interface
• Scrub internals and diagnostics
  • Scrubbing Behavior Table
  • State variables
  • Scrub Reservations
• Snaps
  • Overview
  • Ondisk Structures
  • Snap Removal
  • Recovery
  • SnapMapper
  • Split
  • clone_overlap
• Preventing Stale Reads
  • readable_until
  • Prior Intervals
  • PG “laggy” state
  • PG “wait” state
  • Dead OSDs
• Watch Notify
  • Overview
  • Watch Lifecycle
  • Notify Lifecycle
• Writeback Throttle

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.