Ceph Cheatsheet

Avoid using ceph osd reweight with values other than 0 (out) or 1 (in) unless you have a specific reason.

The Problem: ceph osd reweight does not affect the bucket (e.g., host) weight. This means the probability of placing a Placement Group (PG) on a host remains the same, causing other OSDs on that same host to receive more PGs than their size would normally dictate.

The Solution: To artificially shrink the capacity of a device without affecting bucket probabilities, use ceph osd crush reweight instead.

When balancing PGs (Placement Groups) across OSDs, it is recommended that a PG shard be around 10-50 GiB. This represents the amount of data that can be moved from one OSD to another during balancing.

To calculate the shard size manually:

• For replicated pools: shardsize = pool_size / pg_num
• For erasure coded (n+m) pools: shardsize = pool_size / (pg_num * n)

Ceph is a distributed storage cluster composed of several key services. Understanding these roles is essential for planning deployment and troubleshooting:

• Client: Connects to the cluster to access data (e.g., Linux kernel, QEMU, Samba, or CLI tools like ceph, rbd, rados).
• Cluster (RADOS): The core distributed storage layer.
• Object Store Device (OSD): The actual storage provider. Each OSD stores data on a single drive using a key-value database.
• Monitor (MON): The cluster coordinator. It maintains cluster state and requires an odd number of instances (e.g., 3 or 5) to ensure consensus and tolerate outages.
• Metadata Server (MDS): Specifically for CephFS. It handles filesystem inode transactions and stores its state on OSDs.
• Manager (MGR): Collects cluster statistics, hosts the WebUI, manages balancing, and collects crash reports.

While MDS path restrictions control metadata access, they do not prevent direct OSD access. To truly restrict access, use Namespaces. Namespaces are pool-independent and allow OSDs to check access via namespace restriction.

Workflow:

1. Set the namespace on a CephFS directory: setfattr -n ceph.dir.layout.pool_namespace -v $namespacename <path>.
2. Update client auth caps to include the namespace: osd 'allow rw namespace=$namespacename tag cephfs data=lolfs'.

# Set namespace on a directory
sudo setfattr -n ceph.dir.layout.pool_namespace -v $namespacename /mnt/cephfs/directory/name

# Update auth caps for client to use that namespace
ceph auth caps client.somename mds 'allow rw path=/directory/name' mon 'allow r' osd 'allow rw namespace=$namespacename tag cephfs data=lolfs'

To ensure data safety, always set min_size to be at least one more than the minimum required for reachability. This ensures that even when the cluster is at its min_size threshold, every write is still backed by at least one redundant OSD. If another disk fails while at min_size without redundancy, all data is lost.

Recommended combinations:

• Replica: For n >= 3, use size = n and min_size = 2.
• Erasure Coding: For n >= 2, m >= 2, i >= 1, use size = n + m and min_size = n + i.

Check your current pool settings using: ceph osd pool ls detail

Ceph organizes data using a hierarchy of abstractions:

• Object: The fundamental unit of storage (typically 4MiB). Files and block devices are split into these chunks.
• Pool: A logical grouping of objects. Pools allow you to define specific redundancy (e.g., replication vs. erasure coding) and placement rules for a set of data.
• Namespace: A prefix used to create access realms for objects.
• Placement Group (PG): A partition of a pool created by hashing object keys. Objects are assigned to PGs, and PGs are distributed across OSDs. This partitioning allows Ceph to scale and manage data distribution efficiently.

To run CephFS on an erasure coded (EC) pool, you must enable allow_ec_overwrites. Typically, CephFS uses replicated pools for metadata and EC pools for data.

Pool Setup Example:

1. Create EC profile.
2. Create EC data pool with allow_ec_overwrites true.
3. Create replicated metadata/root pools.
4. Set size (total replicas) and min_size (minimum OSDs required for a PG to be online).

# erasure coding pool (for data)
ceph osd pool create lol_data 32 32 erasure standard_8_2
ceph osd pool set lol_data allow_ec_overwrites true

# replicated pools (for metadata)
ceph osd pool create lol_root 32 replicated
ceph osd pool create lol_metadata 32 replicated

# min_size configuration
ceph osd pool set lol_root size 3
ceph osd pool set lol_root min_size 2
ceph osd pool set lol_metadata size 3
ceph osd pool set lol_metadata min_size 2

Since RBD objects are typically 4 MiB, you can optimize ext4 performance by aligning its stripe width to match the RADOS object size. This ensures allocations are distributed across RADOS objects effectively.

1. Filesystem Creation: Use mkfs.ext4 with stride and stripe_width set to 1024 (since $1024 \times 4096\text{ bytes} = 4\text{ MiB}$).
2. Mounting: You can also specify the stripe width during the mount command using the stripe option.

To verify the current stripe width, use dumpe2fs /dev/rbd/your/rbd and look for RAID stripe width.

# Create ext4 with optimized striping
mkfs -E nodiscard,stride=1024,stripe_width=1024 /dev/rbd/your/rbd

# Mount with stripe option
mount -o defaults,noauto,stripe=1024 /dev/rbd/your/rbd /your/mountpoint

If you have an inode number (e.g., from ls -li) and need to find its file name and path, you can decode the parent extended attribute (xattr) which contains an inode_backtrace_t structure.

Note on Pools:

• For files, use the <cephfs-root-pool>.
• For directories, use the <cephfs-metadata-pool>.

Procedure:

1. Convert the inode number to hex.
2. Use rados to get the parent xattr.
3. Pipe the output to ceph-dencoder to decode the inode_backtrace_t JSON.

# have a inode number of a file with e.g. `ls -li <file>`
rados -p <cephfs-root-pool> getxattr $(printf %x <inodenumnber>).00000000 parent | ceph-dencoder type inode_backtrace_t import - decode dump_json

You can use lvmcache to cache a Ceph RBD on local SSD storage (ensure the SSD is part of a RAID or otherwise secured). This is particularly effective for speeding up HDD-based Ceph pools by:

1. Drastically reducing on-RBD filesystem commit latency.
2. Improving read performance by avoiding the ~15ms latency of fetching from HDDs.

Note: Since Linux 5.7, krbd uses per-cpu queue processing, so grouping multiple RBDs into a single device via LVM/MD may no longer provide significant benefits.