Re: Problem with query and any operation on PGs

["Łukasz Chrustek" <skidoo@tlen.pl>]

Hello,

> ./omap:
> 000011.log  CURRENT  LOCK  LOG  LOG.old  MANIFEST-000010

> after running ceph-objectstore-tool it is:

> ceph pg dump_stuck
> ok
> pg_stat state   up      up_primary      acting  acting_primary
> 1.39    active+remapped+backfilling     [11,4,39]       11      [5,39,70]       5
> 1.1a9   active+remapped+backfilling     [11,30,3]       11      [0,30,8]        0
> 1.b     active+remapped+backfilling     [11,36,94]      11      [38,97,70]      38
> 1.12f   active+remapped+backfilling     [14,11,47]      14      [14,5,69]       14
> 1.1d2   active+remapped+backfilling     [11,2,38]       11      [0,36,49]       0
> 1.133   active+remapped+backfilling     [42,11,83]      42      [42,89,21]      42
> 40.69   stale+active+undersized+degraded        [48]    48      [48]    48
> 1.9d    active+remapped+backfilling     [39,2,11]       39      [39,2,86]       39
> 1.a2    active+remapped+backfilling     [11,12,34]      11      [14,35,95]      14
> 1.10a   active+remapped+backfilling     [11,2,87]       11      [1,87,81]       1
> 1.70    active+remapped+backfilling     [14,39,11]      14      [14,39,4]       14
> 1.60    down+remapped+peering   [83,69,68]      83      [9]     9
> 1.eb    active+remapped+backfilling     [11,18,53]      11      [14,53,69]      14
> 1.8d    active+remapped+backfilling     [11,0,30]       11      [36,0,30]       36
> 1.118   active+remapped+backfilling     [34,11,12]      34      [34,20,86]      34
> 1.121   active+remapped+backfilling     [43,11,35]      43      [43,35,2]       43
> 1.177   active+remapped+backfilling     [14,1,11]       14      [14,1,38]       14
> 1.17c   active+remapped+backfilling     [5,94,11]       5       [5,94,7]        5
> 1.16d   active+remapped+backfilling     [96,11,53]      96      [96,52,9]       96
> 1.19a   active+remapped+backfilling     [11,0,14]       11      [0,17,35]       0
> 1.165   down+peering    [39,55,82]      39      [39,55,82]      39
> 1.1a    active+remapped+backfilling     [36,52,11]      36      [36,52,96]      36
> 1.e7    active+remapped+backfilling     [11,35,44]      11      [34,44,9]       34


> Is there any chance to rescue this cluster ?

I  have  now turned off all OSDs and MONs, after that I turn on two of
three MONs to make qourum. On all osds all ceph processes are off. But
ceph osd tree see old/false data: https://pastebin.com/pVGLxAPs

Why  ceph doesn't see that all osds are down ? What can him block like
this ?



-- 
Regards,
 Łukasz Chrustek