Re-replicated data does not seem to get uniformly redistributed after OSD failure

["Jim Schutt" <jaschut@sandia.gov>]

Hi,

I've been experimenting with failure scenarios to make sure
I understand what happens when an OSD drops out.  In particular,
I've been using "ceph osd out <n>" and watching my all my OSD
servers to see where the data from the removed OSD ends up
after recovery.  I've been doing this testing with 12 servers,
24 OSDs/server.

What I see is that all the data from the removed OSD seem to
end up distributed across the other OSDs on the host holding
the removed OSD.

It works this way if I use the default CRUSH map generated
for just host buckets and devices, i.e. a map using the
straw algorithm for the root and host buckets.  It also
works this way if I generate my own map using the uniform
algorithm for the root and host buckets.

For example, using my map based on the uniform algorithm,
after successively taking out osd.0 thru osd.9 (all on
host cs32), here's the top 24 OSD data store usage:

Host           1K-blocks      Used Available Use% Mounted on

cs39:          942180120  33379144 871114904   4% /ram/mnt/ceph/data.osd.174
cs39:          942180120  33386420 871106604   4% /ram/mnt/ceph/data.osd.186
cs43:          942180120  33484704 871008448   4% /ram/mnt/ceph/data.osd.270
cs43:          942180120  33563912 870930616   4% /ram/mnt/ceph/data.osd.267
cs38:          942180120  33637652 870856876   4% /ram/mnt/ceph/data.osd.162
cs34:          942180120  33773780 870721740   4% /ram/mnt/ceph/data.osd.67
cs37:          942180120  33834584 870660136   4% /ram/mnt/ceph/data.osd.123
cs40:          942180120  33936696 870557928   4% /ram/mnt/ceph/data.osd.203
cs38:          942180120  34212020 870283404   4% /ram/mnt/ceph/data.osd.165
cs42:          942180120  34402852 870095036   4% /ram/mnt/ceph/data.osd.259
cs32:          942180120  45969104 858567088   6% /ram/mnt/ceph/data.osd.17
cs32:          942180120  49694156 854854196   6% /ram/mnt/ceph/data.osd.16
cs32:          942180120  50182636 854370356   6% /ram/mnt/ceph/data.osd.15
cs32:          942180120  50520484 854030460   6% /ram/mnt/ceph/data.osd.12
cs32:          942180120  50669280 853882848   6% /ram/mnt/ceph/data.osd.10
cs32:          942180120  51234372 853319452   6% /ram/mnt/ceph/data.osd.14
cs32:          942180120  51277080 853276808   6% /ram/mnt/ceph/data.osd.22
cs32:          942180120  51279984 853273392   6% /ram/mnt/ceph/data.osd.13
cs32:          942180120  52364512 852192000   6% /ram/mnt/ceph/data.osd.23
cs32:          942180120  52376512 852180384   6% /ram/mnt/ceph/data.osd.21
cs32:          942180120  53026724 851531228   6% /ram/mnt/ceph/data.osd.18
cs32:          942180120  53217832 851343128   6% /ram/mnt/ceph/data.osd.20
cs32:          942180120  53723152 850838768   6% /ram/mnt/ceph/data.osd.19
cs32:          942180120  56159652 848410460   7% /ram/mnt/ceph/data.osd.11

I was thinking that CRUSH would re-replicate considering all
buckets, subject to preventing multiple replicas in the same
bucket, based on this from Sage's thesis, section 5.2.2.1:

    For failed or overloaded devices, CRUSH uniformly
    redistributes items across the storage cluster by
    restarting the recursion at the beginning of the
    select(n,t) (see Algorithm 1 line 11).

So my question is, what am I missing?  Maybe the above doesn't
mean what I think it does?  Or, is there some configuration
that I should be using but don't know about?

Thanks -- Jim