openstack-helm-infra/doc/source/testing/ceph-resiliency/failure-domain.rst

Failure Domains in CRUSH Map

Overview

The CRUSH Map in a Ceph cluster is best visualized as an inverted tree. The hierarchical layout describes the physical topology of the Ceph cluster. Through the physical topology, failure domains are conceptualized from the different branches in the inverted tree. CRUSH rules are created and map to failure domains with data placement policy to distribute the data.

The internal nodes (non-leaves and non-root) in the hierarchy are identified as buckets. Each bucket is a hierarchical aggregation of storage locations and their assigned weights. These are the types defined by CRUSH as the supported buckets.

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

This guide describes the host and rack buckets and their role in constructing a CRUSH Map with separate failure domains. Once a Ceph cluster is configured with the expected CRUSh Map and Rule, the PGs of the designated pool are verified with a script (utils-checkPGs.py) to ensure that the OSDs in all the PGs reside in separate failure domains.

Ceph Environment

The ceph commands and scripts described in this write-up are executed as Linux user root on one of orchestration nodes and one of the ceph monitors deployed as kubernetes pods. The root user has the credential to execute all the ceph commands.

On a kubernetes cluster, a separate namespace named ceph is configured for the ceph cluster. Include the ceph namespace in kubectl when executing this command.

A kubernetes pod is a collection of docker containers sharing a network and mount namespace. It is the basic unit of deployment in the kubernetes cluster. The node in the kubernetes cluster where the orchestration operations are performed needs access to the kubectl command. In this guide, this node is referred to as the orchestration node. On this node, you can list all the pods that are deployed. To execute a command in a given pod, use kubectl to locate the name of the pod and switch to it to execute the command.

Orchestration Node

To gain access to the kubernetes orchestration node, use your login credential and the authentication procedure assigned to you. For environments setup with SSH key-based access, your id_rsa.pub (generated through the ssh-keygen) public key should be in your ~/.ssh/authorized_keys file on the orchestration node.

The kubernetes and ceph commands require the root login credential to execute. Your Linux login requires the sudo privilege to execute commands as user root. On the orchestration node, acquire the root's privilege with your Linux login through the sudo command.

[orchestration]$ sudo -i
<Your Linux login's password>:
[orchestration]#

Kubernetes Pods

On the orchestration node, execute the kubectl command to list the specific set of pods with the --selector option. This kubectl command lists all the ceph monitor pods.

[orchestration]# kubectl get pods -n ceph --selector component=mon
NAME             READY     STATUS    RESTARTS   AGE
ceph-mon-85mlt   2/2       Running   0          9d
ceph-mon-9mpnb   2/2       Running   0          9d
ceph-mon-rzzqr   2/2       Running   0          9d
ceph-mon-snds8   2/2       Running   0          9d
ceph-mon-snzwx   2/2       Running   0          9d

The following kubectl command lists the Ceph OSD pods.

[orchestration]# kubectl get pods -n ceph --selector component=osd
NAME                              READY     STATUS    RESTARTS   AGE
ceph-osd-default-166a1044-95s74   2/2       Running   0          9d
ceph-osd-default-166a1044-bglnm   2/2       Running   0          9d
ceph-osd-default-166a1044-lq5qq   2/2       Running   0          9d
ceph-osd-default-166a1044-lz6x6   2/2       Running   0          9d
. . .

To list all the pods in all the namespaces, execute this kubectl command.

[orchestration]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS      RESTARTS   AGE
ceph          ceph-bootstrap-rpzld                       0/1       Completed   0          10d
ceph          ceph-cephfs-client-key-generator-pvzs6     0/1       Completed   0          10d
ceph          ceph-cephfs-provisioner-796668cd7-bn6mn    1/1       Running     0          10d

Execute Commands in Pods

To execute multiple commands in a pod, you can switch to the execution context of the pod with a /bin/bash session.

[orchestration]# kubectl exec -it ceph-mon-85mlt -n ceph -- /bin/bash
[ceph-mon]# ceph status
  cluster:
    id:     07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8
    health: HEALTH_OK

  services:
    mon: 5 daemons, quorum host1,host2,host3,host4,host5
    mgr: host6(active), standbys: host1
    mds: cephfs-1/1/1 up  {0=mds-ceph-mds-7cb4f57cc-prh87=up:active}, 1 up:standby
    osd: 72 osds: 72 up, 72 in
    rgw: 2 daemons active

  data:
    pools:   20 pools, 3944 pgs
    objects: 86970 objects, 323 GB
    usage:   1350 GB used, 79077 GB / 80428 GB avail
    pgs:     3944 active+clean

  io:
    client:   981 kB/s wr, 0 op/s rd, 84 op/s wr

To verify that you are executing within the context of a pod. Display the content of the /proc/self/cgroup control group file. The kubepods output in the cgroup file shows that you're executing in a docker container of a pod.

[ceph-mon]# cat /proc/self/cgroup
11:hugetlb:/kubepods/besteffort/podafb3689c-8c5b-11e8-be6a-246e96290f14/ff6cbc58348a44722ee6a493845b9c2903fabdce80d0902d217cc4d6962d7b53
. . .

To exit the pod and resume the orchestration node's execution context.

[ceph-mon]# exit
[orchestration]#

To verify that you are executing on the orchestration node's context, display the /proc/self/cgroup control group file. You would not see the kubepods docker container in the output.

[orchestration]# cat /proc/self/cgroup
11:blkio:/user.slice
10:freezer:/
9:hugetlb:/
. . .

It is also possible to run the ceph commands via the kubectl exec without switching to a pod's container.

[orchestration]# kubectl exec ceph-mon-9mpnb -n ceph -- ceph status
  cluster:
    id:     07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8
    health: HEALTH_OK
. . .

Failure Domains

A failure domain provides the fault isolation for the data and it corresponds to a branch on the hierarchical topology. To protect against data loss, OSDs that are allocated to PGs should be chosen from different failure domains. Losing a branch takes down all the OSDs in that branch only and OSDs in the other branches are not effected.

In a data center, baremetal hosts are typically installed in a rack (refrigerator size cabinet). Multiple racks with hosts in each rack are used to provision the OSDs running on each host. A rack is envisioned as a branch in the CRUSH topology.

To provide data redundancy, ceph maintains multiple copies of the data. The total number of copies to store for each piece of data is determined by the ceph osd_pool_default_size ceph.conf parameter. With this parameter set to 3, each piece of the data has 3 copies that gets stored in a pool. Each copy is stored on different OSDs allocated from different failure domains.

Host

Choosing host as the failure domain lacks all the protections against data loss.

To illustrate, a Ceph cluster has been provisioned with six hosts and four OSDs on each host. The hosts are enclosed in respective racks where each rack contains two hosts.

In the configuration of the Ceph cluster, without explicit instructions on where the host and rack buckets should be placed, Ceph would create a CRUSH map without the rack bucket. A CRUSH rule that get created uses the host as the failure domain. With the size (replica) of a pool set to 3, the OSDs in all the PGs are allocated from different hosts.

root=default
├── host1
│   ├── osd.1
│   ├── osd.2
│   ├── osd.3
│   └── osd.4
├── host2
│   ├── osd.5
│   ├── osd.6
│   ├── osd.7
│   └── osd.8
├── host3
│   ├── osd.9
│   ├── osd.10
│   ├── osd.11
│   └── osd.12
├── host4
│   ├── osd.13
│   ├── osd.14
│   ├── osd.15
│   └── osd.16
├── host5
│   ├── osd.17
│   ├── osd.18
│   ├── osd.19
│   └── osd.20
└── host6
    ├── osd.21
    ├── osd.22
    ├── osd.23
    └── osd.24

On this ceph cluster, it has a CRUSH rule that uses the host as the failure domain.

# ceph osd crush rule ls
replicated_host
# ceph osd crush rule dump replicated_host
{
    "rule_id": 0,
    "rule_name": "replicated_host",
    "ruleset": 0,
    "type": 1,
    "min_size": 1,
    "max_size": 10,
    "steps": [
        {
            "op": "take",
            "item": -1,
            "item_name": "default"
        },
        {
            "op": "chooseleaf_firstn",
            "num": 0,
            "type": "host" },
        {
            "op": "emit"
        }
    ]
}

Verify the CRUSH rule that is assigned to the ceph pool. In this example, the rbd pool is used.

# ceph osd pool get rbd crush_rule
crush_rule: replicated_host
# ceph osd pool get rbd size
size: 3
# ceph osd pool get rbd pg_num
pg_num: 1024

To verify that the OSDs in all the PGs are allocated from different hosts, invoke the utils-checkPGs.py utility on the ceph pool. The offending PGs are printed to stdout.

# /tmp/utils-checkPGs.py rbd
Checking PGs in pool rbd ... Passed

With host as the failure domain, quite possibly, some of the PGs might have OSDs allocated from different hosts that are located in the same rack. For example, one PG might have OSD numbers [1, 8, 13]. OSDs 1 and 8 are found on hosts located in rack1. When rack1 suffers a catastrophe failure, PGs with OSDs allocated from the hosts in rack1 would be severely degraded.

Rack

Choosing rack as the failure domain provides better protection against data loss.

To prevent PGs with OSDs allocated from hosts that are located in the same rack, configure the CRUSH hierarchy with the rack buckets. In each rack bucket, it contains the hosts that reside in the same physical rack. A CRUSH Rule is configured with rack as the failure domain.

In the following hierarchical topology, the Ceph cluster was configured with three rack buckets. Each bucket has two hosts. In pools that were created with the CRUSH rule set to rack, the OSDs in all the PGs are allocated from the distinct rack.

root=default
├── rack1
│   ├── host1
│   │   ├── osd.1
│   │   ├── osd.2
│   │   ├── osd.3
│   │   └── osd.4
│   └── host2
│       ├── osd.5
│       ├── osd.6
│       ├── osd.7
│       └── osd.8
├── rack2
│   ├── host3
│   │   ├── osd.9
│   │   ├── osd.10
│   │   ├── osd.11
│   │   └── osd.12
│   └── host4
│       ├── osd.13
│       ├── osd.14
│       ├── osd.15
│       └── osd.16
└── rack3
    ├── host5
    │   ├── osd.17
    │   ├── osd.18
    │   ├── osd.19
    │   └── osd.20
    └── host6
        ├── osd.21
        ├── osd.22
        ├── osd.23
        └── osd.24

Verify the Ceph cluster has a CRUSH rule with rack as the failure domain.

# ceph osd crush rule ls
rack_replicated_rule
# ceph osd crush rule dump rack_replicated_rule
{
    "rule_id": 2,
    "rule_name": "rack_replicated_rule",
    "ruleset": 2,
    "type": 1,
    "min_size": 1,
    "max_size": 10,
    "steps": [
        {
            "op": "take",
            "item": -1,
            "item_name": "default"
        },
        {
            "op": "chooseleaf_firstn",
            "num": 0,
            "type": "rack"
        },
        {
            "op": "emit"
        }
    ]
}

Create a ceph pool with its CRUSH rule set to the rack's rule.

# ceph osd pool create rbd 2048 2048 replicated rack_replicated_rule
pool 'rbd' created
# ceph osd pool get rbd crush_rule
crush_rule: rack_replicated_rule
# ceph osd pool get rbd size
size: 3
# ceph osd pool get rbd pg_num
pg_num: 2048

Invoke the utils-checkPGs.py script on the pool to verify that there are no PGs with OSDs allocated from the same rack. The offending PGs are printed to stdout.

# /tmp/utils-checkPGs.py rbd
Checking PGs in pool rbd ... Passed

CRUSH Map and Rule

On a properly configured Ceph cluster, there are different ways to view the CRUSH hierarchy.

ceph CLI

Print to stdout the CRUSH hierarchy with the ceph CLI.

root@host5:/# ceph osd crush tree
ID  CLASS WEIGHT   TYPE NAME
 -1       78.47974 root default
-15       26.15991     rack rack1
 -2       13.07996         host host1
  0   hdd  1.09000             osd.0
  1   hdd  1.09000             osd.1
  2   hdd  1.09000             osd.2
  3   hdd  1.09000             osd.3
  4   hdd  1.09000             osd.4
  5   hdd  1.09000             osd.5
  6   hdd  1.09000             osd.6
  7   hdd  1.09000             osd.7
  8   hdd  1.09000             osd.8
  9   hdd  1.09000             osd.9
 10   hdd  1.09000             osd.10
 11   hdd  1.09000             osd.11
 -5       13.07996         host host2
 12   hdd  1.09000             osd.12
 13   hdd  1.09000             osd.13
 14   hdd  1.09000             osd.14
 15   hdd  1.09000             osd.15
 16   hdd  1.09000             osd.16
 17   hdd  1.09000             osd.17
 18   hdd  1.09000             osd.18
 19   hdd  1.09000             osd.19
 20   hdd  1.09000             osd.20
 21   hdd  1.09000             osd.21
 22   hdd  1.09000             osd.22
 23   hdd  1.09000             osd.23
-16       26.15991     rack rack2
-13       13.07996         host host3
 53   hdd  1.09000             osd.53
 54   hdd  1.09000             osd.54
 58   hdd  1.09000             osd.58
 59   hdd  1.09000             osd.59
 64   hdd  1.09000             osd.64
 65   hdd  1.09000             osd.65
 66   hdd  1.09000             osd.66
 67   hdd  1.09000             osd.67
 68   hdd  1.09000             osd.68
 69   hdd  1.09000             osd.69
 70   hdd  1.09000             osd.70
 71   hdd  1.09000             osd.71
 -9       13.07996         host host4
 36   hdd  1.09000             osd.36
 37   hdd  1.09000             osd.37
 38   hdd  1.09000             osd.38
 39   hdd  1.09000             osd.39
 40   hdd  1.09000             osd.40
 41   hdd  1.09000             osd.41
 42   hdd  1.09000             osd.42
 43   hdd  1.09000             osd.43
 44   hdd  1.09000             osd.44
 45   hdd  1.09000             osd.45
 46   hdd  1.09000             osd.46
 47   hdd  1.09000             osd.47
-17       26.15991     rack rack3
-11       13.07996         host host5
 48   hdd  1.09000             osd.48
 49   hdd  1.09000             osd.49
 50   hdd  1.09000             osd.50
 51   hdd  1.09000             osd.51
 52   hdd  1.09000             osd.52
 55   hdd  1.09000             osd.55
 56   hdd  1.09000             osd.56
 57   hdd  1.09000             osd.57
 60   hdd  1.09000             osd.60
 61   hdd  1.09000             osd.61
 62   hdd  1.09000             osd.62
 63   hdd  1.09000             osd.63
 -7       13.07996         host host6
 24   hdd  1.09000             osd.24
 25   hdd  1.09000             osd.25
 26   hdd  1.09000             osd.26
 27   hdd  1.09000             osd.27
 28   hdd  1.09000             osd.28
 29   hdd  1.09000             osd.29
 30   hdd  1.09000             osd.30
 31   hdd  1.09000             osd.31
 32   hdd  1.09000             osd.32
 33   hdd  1.09000             osd.33
 34   hdd  1.09000             osd.34
 35   hdd  1.09000             osd.35
root@host5:/#

To see weight and affinity of each OSD.

root@host5:/# ceph osd tree
ID  CLASS WEIGHT   TYPE NAME                 STATUS REWEIGHT PRI-AFF
 -1       78.47974 root default
-15       26.15991     rack rack1
 -2       13.07996         host host1
  0   hdd  1.09000             osd.0             up  1.00000 1.00000
  1   hdd  1.09000             osd.1             up  1.00000 1.00000
  2   hdd  1.09000             osd.2             up  1.00000 1.00000
  3   hdd  1.09000             osd.3             up  1.00000 1.00000
  4   hdd  1.09000             osd.4             up  1.00000 1.00000
  5   hdd  1.09000             osd.5             up  1.00000 1.00000
  6   hdd  1.09000             osd.6             up  1.00000 1.00000
  7   hdd  1.09000             osd.7             up  1.00000 1.00000
  8   hdd  1.09000             osd.8             up  1.00000 1.00000
  9   hdd  1.09000             osd.9             up  1.00000 1.00000
 10   hdd  1.09000             osd.10            up  1.00000 1.00000
 11   hdd  1.09000             osd.11            up  1.00000 1.00000
 -5       13.07996         host host2
 12   hdd  1.09000             osd.12            up  1.00000 1.00000
 13   hdd  1.09000             osd.13            up  1.00000 1.00000
 14   hdd  1.09000             osd.14            up  1.00000 1.00000
 15   hdd  1.09000             osd.15            up  1.00000 1.00000
 16   hdd  1.09000             osd.16            up  1.00000 1.00000
 17   hdd  1.09000             osd.17            up  1.00000 1.00000
 18   hdd  1.09000             osd.18            up  1.00000 1.00000
 19   hdd  1.09000             osd.19            up  1.00000 1.00000
 20   hdd  1.09000             osd.20            up  1.00000 1.00000
 21   hdd  1.09000             osd.21            up  1.00000 1.00000
 22   hdd  1.09000             osd.22            up  1.00000 1.00000
 23   hdd  1.09000             osd.23            up  1.00000 1.00000

crushtool CLI

To extract the CRUSH Map from a running cluster and convert it into ascii text.

# ceph osd getcrushmap -o /tmp/cm.bin
100
# crushtool -d /tmp/cm.bin -o /tmp/cm.rack.ascii
# cat /tmp/cm.rack.ascii
. . .
# buckets
host host1 {
      id -2           # do not change unnecessarily
      id -3 class hdd         # do not change unnecessarily
      # weight 13.080
      alg straw2
      hash 0  # rjenkins1
      item osd.0 weight 1.090
      item osd.1 weight 1.090
      item osd.2 weight 1.090
      item osd.3 weight 1.090
      item osd.4 weight 1.090
      item osd.5 weight 1.090
      item osd.6 weight 1.090
      item osd.7 weight 1.090
      item osd.8 weight 1.090
      item osd.9 weight 1.090
      item osd.10 weight 1.090
      item osd.11 weight 1.090
}
host host2 {
      id -5           # do not change unnecessarily
      id -6 class hdd         # do not change unnecessarily
      # weight 13.080
      alg straw2
      hash 0  # rjenkins1
      item osd.12 weight 1.090
      item osd.13 weight 1.090
      item osd.14 weight 1.090
      item osd.15 weight 1.090
      item osd.16 weight 1.090
      item osd.18 weight 1.090
      item osd.19 weight 1.090
      item osd.17 weight 1.090
      item osd.20 weight 1.090
      item osd.21 weight 1.090
      item osd.22 weight 1.090
      item osd.23 weight 1.090
}
rack rack1 {
      id -15          # do not change unnecessarily
      id -20 class hdd        # do not change unnecessarily
      # weight 26.160
      alg straw2
      hash 0  # rjenkins1
      item host1 weight 13.080
      item host2 weight 13.080
}
. . .
root default {
      id -1          # do not change unnecessarily
      id -4 class hdd        # do not change unnecessarily
      # weight 78.480
      alg straw2
      hash 0  # rjenkins1
      item rack1 weight 26.160
      item rack2 weight 26.160
      item rack3 weight 26.160
}

# rules
rule replicated_rack {
      id 2
      type replicated
      min_size 1
      max_size 10
      step take default
      step chooseleaf firstn 0 type rack
      step emit
}
# end crush map

The utils-checkPGs.py script can read the same data from memory and construct the failure domains with OSDs. Verify the OSDs in each PG against the constructed failure domains.

Configure the Failure Domain in CRUSH Map

The Ceph ceph-osd, ceph-client and cinder charts accept configuration parameters to set the Failure Domain for CRUSH. The options available are failure_domain, failure_domain_by_hostname, failure_domain_name and crush_rule

ceph-osd specific overrides
failure_domain: Set the CRUSH bucket type for your OSD to reside in. (DEFAULT: "host")
failure_domain_by_hostname: Specify the portion of the hostname to use for your failure domain bucket name. (DEFAULT: "false")
failure_domain_name: Manually name the failure domain bucket name. This configuration option should only be used when using host based overrides. (DEFAULT: "false")

ceph-client and cinder specific overrides
crush_rule**: Set the crush rule for a pool (DEFAULT: "replicated_rule")

An example of a lab enviroment had the following paramters set for the ceph yaml override file to apply a rack level failure domain within CRUSH.

endpoints:
  identity:
    namespace: openstack
  object_store:
    namespace: ceph
  ceph_mon:
    namespace: ceph
network:
  public: 10.0.0.0/24
  cluster: 10.0.0.0/24
deployment:
  storage_secrets: true
  ceph: true
  csi_rbd_provisioner: true
  rbd_provisioner: false
  cephfs_provisioner: false
  client_secrets: false
  rgw_keystone_user_and_endpoints: false
bootstrap:
  enabled: true
conf:
  ceph:
    global:
      fsid: 6c12a986-148d-45a7-9120-0cf0522ca5e0
  rgw_ks:
    enabled: true
  pool:
    default:
      crush_rule: rack_replicated_rule
    crush:
      tunables: null
    target:
      # NOTE(portdirect): 5 nodes, with one osd per node
      osd: 18
      pg_per_osd: 100
  storage:
    osd:
      - data:
          type: block-logical
          location: /dev/vdb
        journal:
          type: block-logical
          location: /dev/vde1
      - data:
          type: block-logical
          location: /dev/vdc
        journal:
          type: block-logical
          location: /dev/vde2
      - data:
          type: block-logical
          location: /dev/vdd
        journal:
          type: block-logical
          location: /dev/vde3
  overrides:
    ceph_osd:
      hosts:
        - name: osh-1
          conf:
            storage:
              failure_domain: "rack"
              failure_domain_name: "rack1"
        - name: osh-2
          conf:
            storage:
              failure_domain: "rack"
              failure_domain_name: "rack1"
        - name: osh-3
          conf:
            storage:
              failure_domain: "rack"
              failure_domain_name: "rack2"
        - name: osh-4
          conf:
            storage:
              failure_domain: "rack"
              failure_domain_name: "rack2"
        - name: osh-5
          conf:
            storage:
              failure_domain: "rack"
              failure_domain_name: "rack3"
        - name: osh-6
          conf:
            storage:
              failure_domain: "rack"
              failure_domain_name: "rack3"

Note

Note that the cinder chart will need an override configured to ensure the cinder pools in Ceph are using the correct crush_rule.

pod:
  replicas:
    api: 2
    volume: 1
    scheduler: 1
    backup: 1
conf:
  cinder:
    DEFAULT:
      backup_driver: cinder.backup.drivers.swift
  ceph:
    pools:
      backup:
        replicated: 3
        crush_rule: rack_replicated_rule
        chunk_size: 8
      volume:
        replicated: 3
        crush_rule: rack_replicated_rule
        chunk_size: 8

The charts can be updated with these overrides pre or post deployment. If this is a post deployment change then the following steps will apply for a gate based openstack-helm deployment.

cd /opt/openstack-helm
helm upgrade --install ceph-osd ../openstack-helm-infra/ceph-osd --namespace=ceph --values=/tmp/ceph.yaml
kubectl delete jobs/ceph-rbd-pool -n ceph
helm upgrade --install ceph-client ../openstack-helm-infra/ceph-client --namespace=ceph --values=/tmp/ceph.yaml
helm delete cinder --purge
helm upgrade --install cinder ./cinder --namespace=openstack --values=/tmp/cinder.yaml

Note

There will be a brief interuption of I/O and a data movement of placement groups in Ceph while these changes are applied. The data movement operation can take several minutes to several days to complete.

The utils-checkPGs.py Script

The purpose of the utils-checkPGs.py script is to check whether a PG has OSDs allocated from the same failure domain. The violating PGs with their respective OSDs are printed to the stdout.

In this example, a pool was created with the CRUSH rule set to the host failure domain. The ceph cluster was configured with the rack buckets. The CRUSH algorithm allocated the OSDs from different hosts in each PG. The rack buckets were ignored and thus the duplicate racks which get reported by the script.

root@host5:/# /tmp/utils-checkPGs.py cmTestPool
Checking PGs in pool cmTestPool ... Failed
OSDs [44, 32, 53] in PG 20.a failed check in rack [u'rack2', u'rack2', u'rack2']
OSDs [61, 5, 12] in PG 20.19 failed check in rack [u'rack1', u'rack1', u'rack1']
OSDs [69, 9, 15] in PG 20.2a failed check in rack [u'rack1', u'rack1', u'rack1']
. . .

Note

The utils-checkPGs.py utility is executed on-demand. It is intended to be executed on one of the ceph-mon pods.

If the utils-checkPGs.py script did not find any violation, it prints Passed. In this example, the ceph cluster was configured with the rack buckets. The rbd pool was created with its CRUSH rule set to the rack. The utils-checkPGs.py script did not find duplicate racks in PGs.

root@host5:/# /tmp/utils-checkPGs.py rbd
Checking PGs in pool rbd ... Passed

Invoke the utils-checkPGs.py script with the --help option to get the script's usage.

root@host5:/# /tmp/utils-checkPGs.py --help
usage: utils-checkPGs.py [-h] PoolName [PoolName ...]

Cross-check the OSDs assigned to the Placement Groups (PGs) of a ceph pool
with the CRUSH topology.  The cross-check compares the OSDs in a PG and
verifies the OSDs reside in separate failure domains.  PGs with OSDs in
the same failure domain are flagged as violation.  The offending PGs are
printed to stdout.

This CLI is executed on-demand on a ceph-mon pod.  To invoke the CLI, you
can specify one pool or list of pools to check.  The special pool name
All (or all) checks all the pools in the ceph cluster.

positional arguments:
  PoolName    List of pools (or All) to validate the PGs and OSDs mapping

optional arguments:
  -h, --help  show this help message and exit
root@host5:/#

The source for the utils-checkPGs.py script is available at openstack-helm/ceph-mon/templates/bin/utils/_checkPGs.py.tpl.

Ceph Deployments

Through testing and verification, you derive at a CRUSH Map with the buckets that are deemed beneficial to your ceph cluster. Standardize on the verified CRUSH map to have the consistency in all the Ceph deployments across the data centers.

Mimicking the hierarchy in your CRUSH Map with the physical hardware setup should provide the needed information on the topology layout. With the racks layout, each rack can store a replica of your data.

To validate a ceph cluster with the number of replica that is based on the number of racks:

1. The number of physical racks and the number of replicas are 3, respectively. Create a ceph pool with replica set to 3 and pg_num set to (# of OSDs * 50) / 3 and round the number to the next power-of-2. For example, if the calculation is 240, round it to 256. Assuming the pool you just created had 256 PGs. In each PG, verify the OSDs are chosen from the three racks, respectively. Use the utils-checkPGs.py script to verify the OSDs in all the PGs of the pool.
2. The number of physical racks is 2 and the number of replica is 3. Create a ceph pool as described in the previous step. In the pool you created, in each PG, verify two of the OSDs are chosen from the two racks, respectively. The third OSD can come from one of the two racks but not from the same hosts as the other two OSDs.

Data Movement

Changes to the CRUSH Map always trigger data movement. It is prudent that you plan accordingly when restructuring the CRUSH Map. Once started, the CRUSH Map restructuring runs to completion and can neither be stopped nor suspended. On a busy Ceph cluster with live transactions, it is always safer to use divide-and-conquer approach to complete small chunk of works in multiple sessions.

Watch the progress of the data movement while the Ceph cluster re-balances itself.

# watch ceph status
  cluster:
    id:     07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8
    health: HEALTH_WARN
            137084/325509 objects misplaced (42.114%)
            Degraded data redundancy: 28/325509 objects degraded (0.009%), 15 pgs degraded

  services:
    mon: 5 daemons, quorum host1,host2,host3,host4,host5
    mgr: host6(active), standbys: host1
    mds: cephfs-1/1/1 up  {0=mds-ceph-mds-7cb4f57cc-prh87=up:active}, 1 up:standby
    osd: 72 osds: 72 up, 72 in; 815 remapped pgs
    rgw: 2 daemons active

  data:
    pools:   19 pools, 2920 pgs
    objects: 105k objects, 408 GB
    usage:   1609 GB used, 78819 GB / 80428 GB avail
    pgs:     28/325509 objects degraded (0.009%)
             137084/325509 objects misplaced (42.114%)
             2085 active+clean
             790  active+remapped+backfill_wait
             22   active+remapped+backfilling
             15   active+recovery_wait+degraded
             4    active+recovery_wait+remapped
             4    active+recovery_wait

  io:
    client:   11934 B/s rd, 3731 MB/s wr, 2 op/s rd, 228 kop/s wr
    recovery: 636 MB/s, 163 objects/s

At the time this ceph status command was executed, the status's output showed that the ceph cluster was going through re-balancing. Among the overall 2920 pgs, 2085 of them are in active+clean state. The remaining pgs are either being remapped or recovered. As the ceph cluster continues its re-balance, the number of pgs in active+clean increases.

# ceph status
  cluster:
    id:     07c31d0f-bcc6-4db4-aadf-2d2a0f13edb8
    health: HEALTH_OK

  services:
    mon: 5 daemons, quorum host1,host2,host3,host4,host5
    mgr: host6(active), standbys: host1
    mds: cephfs-1/1/1 up  {0=mds-ceph-mds-7cc55c9695-lj22d=up:active}, 1 up:standby
    osd: 72 osds: 72 up, 72 in
    rgw: 2 daemons active

  data:
    pools:   19 pools, 2920 pgs
    objects: 134k objects, 519 GB
    usage:   1933 GB used, 78494 GB / 80428 GB avail
    pgs:     2920 active+clean

  io:
    client:   1179 B/s rd, 971 kB/s wr, 1 op/s rd, 41 op/s wr

When the overall number of pgs is equal to the number of active+clean pgs, the health of the ceph cluster changes to HEALTH_OK (assuming there are no other warning conditions).